Agent-as-a-Judge: Evaluating Agents with Agents

## Abstract

Abstract Contemporary evaluation techniques are inadequate for agentic systems. These approaches either focus exclusively on final outcomes—ignoring the step-by-step nature of agentic systems, or require excessive manual labour. To address this, we introduce the Agent-as-a-Judge framework, wherein agentic systems are used to evaluate agentic systems. This is an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process. We apply the Agent-as-a-Judge to the task of code generation. To overcome issues with existing benchmarks and provide a proof-of-concept testbed for Agent-as-a-Judge, we present DevAI , a new benchmark of 55 realistic automated AI development tasks. It includes rich manual annotations, like a total of 365 hierarchical user requirements. We benchmark three of the popular agentic systems using Agent-as-a-Judge and find it dramatically outperforms LLM-as-a-Judge and is as reliable as our human evaluation baseline. Altogether, we believe that Agent-as-a-Judge marks a concrete step forward for modern agentic systems—by providing rich and reliable reward signals necessary for dynamic and scalable self-improvement.

## 1 Introduction

Recent years have seen multimodal agentic systems move from occasionally being able to solve small toy problems to being regularly deployed for challenging real-world problems (the dream of most AI research).
Yet, the current evaluation methods and the available benchmarks for agentic systems are struggling to keep up with these rapid advances, dramatically slowing true progress.

We believe that the current issue with evaluating agentic systems stems from the lack of feedback during the intermediate task-solving stages for these nontraditional systems. Agentic systems think more like humans, often act step-by-step (Wooldridge, ) and often host very human-like symbolic communications internally to solve problems (Zhuge et al., ).
And thus agentic systems should be evaluated like a human, with rich evaluative feedback which looks at the full thought and action trajectory; evaluating an agentic system in the traditional way is like evaluating a student using multiple-choice testing—a comparatively unreliable estimator (Park, ).
For example, while SWE-Bench (Yang et al., ) is widespread, its evaluation method, which relies solely on the final resolve rate for long-term automated repair tasks, does not effectively pinpoint what is happening within agentic systems that affects the resolve rate.
On the other hand, performing a better evaluation with a human is prohibitively expensive.
We instead propose that agentic systems should be used to evaluate agentic systems.
Inspired by LLM-as-a-Judge (Zheng et al., ; Fu et al., ; Chen et al., ), which uses LLMs to evaluate LLMs, we call this framework Agent-as-a-Judge, of which it is a key extension to the world of agentic systems (see Figure 1).
It not only retains the cost-effectiveness of LLM-as-a-Judge but is also equipped with agentic features, allowing it to provide rich intermediate feedback throughout the entire process, as it acts as an agentic system.
We apply the Agent-as-a-Judge systems to the problem of evaluating code generating systems—one of the areas where agentic systems have looked the most promising recently.

Figure: Figure 1: We introduce the Agent-as-a-Judge framework wherein agentic systems are used to evaluate agentic systems. We compare this to LLM-as-a-Judge, which uses LLMs to evaluate LLMs and for which Agent-as-a-Judge is a natural evolution, and Human-as-a-Judge, where skilled human labourers manually evaluate an agentic system.
Refer to caption: x1.png

In code generation, the development of benchmarks has also lagged behind the rapid advancement of agentic systems.
HumanEval (Chen et al., ), for example, focuses exclusively on algorithmic problems, while MBPP (Austin et al., ) deals with simple programming tasks.
Although they are useful for evaluating the basic skills of foundation models, neither of these two reflects the most practical challenges developers face.
As a step away from this, SWE-Bench (Jimenez et al., ) did introduce more realistic problems from GitHub, offering a fresh approach to evaluation, but still primarily focuses on automated repairs tasks development process.
Concerningly, recent research shows that large language models (LLMs) can already solve over 27% of the tasks in SWE-Bench without needing of advanced agentic systems (Xia et al., ).
Equally concerning, recent work has begun to introduce mechanisms designed specifically for the individual tasks in the SWE-Bench dataset, leading to a lack of real-world generalization and violating Goodhart’s law: “When a measure becomes a target, it ceases to be a good measure” (Goodhart, ).

To address the aforementioned issues with the current benchmarks in code generation, we introduce DevAI: the AI Developer Dataset, which contains $55$ real-world comprehensive AI app development tasks created by expert annotators.
We apply three leading open-source code-generating agentic frameworks to the tasks in DevAI: MetaGPT (Hong et al., ), GPT-Pilot (Pythagora.io, ), and OpenHands (Wang et al., ).
We evaluate their performance using human judges (a painstaking process), LLM-as-a-Judge (Zheng et al., ), and our Agent-as-a-Judge framework.

Through human evaluation, we found that GPT-Pilot and OpenHands were each able to satisfy about $29\%$ of the task requirements in DevAI, but only one full task—showing that DevAI presents a good level of challenge to current systems.
When comparing our human judges with our automatic Agent-as-a-Judge framework, we found that Agent-as-a-Judge aligns more closely with the consensus of our human judges ($90\%$) as compared to LLM-as-a-Judge ($70\%$) in all cases tested.
In addition, we find that it aligns more closely with this ensemble than the individual human evaluators do, suggesting that—not only is it suitable as a replacement for a human evaluator—but it could in fact be more useful than an average lone human evaluator.
In addition, considering the evaluation cost, Agent-as-a-Judge saves 97.72% of the time and 97.64% of the cost compared to involving three human experts.

In summary, the principal contributions of this work are:

- $\bullet$
We release the DevAI dataset, which consists of 55 comprehensive AI development tasks with accompanying tags, individual hierarchical requirements, and individual preferences.
- $\bullet$
We benchmark three top open-source code generation agentic frameworks in DevAI, providing a more comprehensive analysis than previous evaluations of them.
- $\bullet$
We introduce the general Agent-as-a-Judge concept, allowing agentic systems a fair and rich evaluation without the traditional costs associated with human involvement.
- $\bullet$
We demonstrate that an Agent-as-a-Judge outperforms an LLM-as-a-Judge and performs comparably to human evaluators in our proof-of-concept.

## 2 DevAI: A Dataset for Automated AI Development

In this section, we introduce our new DevAI benchmark. We then evaluate three state-of-the-art code-generating agentic systems on this benchmark in Section 3 and present their basic statistics.

### 2.1 Motivation

#### Background

The code generation domain is an area where agentic systems have seen significant industrial deployment over the past two years (e.g., Devin(^1^11[https://www.cognition.ai/blog/introducing-devin](https://www.cognition.ai/blog/introducing-devin)) and Cursor(^2^22[https://www.cursor.com/](https://www.cursor.com/))).
However, in code generation, there isn’t yet a benchmark that accurately reflects realistic user queries for developing complete AI systems. We believe this is because of the difficulty to evaluate such complex, real-world tasks.
For example, while many companies advertise their systems based on its performance on benchmarks such as SWE-Bench (Yang et al., ) (for automated repair) or HumanEval (Chen et al., ) (for algorithmic problems), these benchmarks cover only a small bit of an actual development process. Moreover, none of them accurately reflect the intermediate stages of development or provide sufficient reward signals for long-horizon development—similar issues are present in OpenAI’s recent MLE-Bench (Chan et al., ). A benchmark that can evaluate the entire development process—ideally in a way that can help understand the degree to which current AI methods can reduce human labour—is missing.

#### Topic

We chose automated AI development as our main topic. While AI and ML tasks are often more complex, they follow clear, standard procedures. For example, data processing typically comes first in an AI pipeline, and performance reporting goes at the end. We believe this topological nature can help better monitor the development process and provide useful signals to the agentic systems.

#### Goals

An ideal benchmark should address critical issues in automated development by focusing on three key factors. First, it should reflect practical software scenarios, where tasks are often too complex for a single LLM, requiring human or agentic systems. Second, it should emphasize the development process, not just final outcomes (e.g., pass@1 rates offer limited feedback and fail to highlight intermediate problems). Lastly, the evaluation should be computationally cost-effective and efficient, avoiding long training times or excessive manual oversight.

Figure: Figure 2: Distribution of DevAI Tasks (1) DevAI focuses on AI development tasks and so terms such as “dataset,” “model,” and “results” are particularly common in the queries. (2) The first 53 tasks in DevAI all have a one-paragraph query but of varying lengths (note that task 54 and 55 are excluded here as they are outliers, representing the longest and most complex tasks in the dataset). (3) Each task has one or more tags. The prevalence of supervised learning here reflects the fact that it dominates many machine learning applications. (4) SVM classifiers (Cortes, ) and LSTM models (Hochreiter, ) are two of the most widely used architectures—a fact reflected by DevAI.
Refer to caption: x2.png

Figure: Figure 3: A task example in DevAI. This task is adapted from a real-world demo given at [https://www.cognition.ai/blog/introducing-devin](https://www.cognition.ai/blog/introducing-devin). As this example shows, task requirements in DevAI are structured as a Directed Acyclic Graph (DAG), with nodes representing individual requirements and directed edges showing dependencies. More examples are in Appendix G.

### 2.2 The DevAI Dataset

Motivated by the ideas outlined above, we propose the DevAI dataset.
DevAI consists of a curated set of $55$ tasks, each defined by (1) a plain text user query that describes an AI development task; (2) a set of plain text requirements (for a total of $365$ requirements), each with a set of dependencies connecting them to other requirements; and (3) a set of preferences (for a total of $125$ preferences) which represent softer requirements.

DevAI is structured so that an agentic system starts by receiving a user query to begin development. The system is then evaluated on how well it meets the requirements, with preferences serving as optional, softer criteria.
An example of one of the DevAI tasks can be seen in Figure 3.

The tasks in DevAI are relatively small-scale but cover commonly used key development techniques.
As shown in Figure 2, our tasks are tagged and cover a variety of key areas in AI: supervised learning, reinforcement learning, computer vision, natural language processing, generative models, and others.
Each of the tasks is a real-world problem that could be given to a research engineer, while simultaneously being relatively inexpensive computationally to run so as to reduce the cost of evaluating a method on this benchmark.
Details of the sample collection and human labeling process for DevAI are provided in Appendix E.

The requirements belonging to each task represent a milestone in the comprehensive development process and are arranged as a directed acyclic graph (similar to the work by He et al. ()), with requirements such as visualizing results depending on correct data loading and modeling.
This allows for more comprehensive non-sparse feedback than a binary success metric.
Furthermore, the inclusion of hierarchical requirements makes simple memorization an inadequate solution strategy, as completing the entire task requires agentic capabilities rather than relying solely on symbolic memorization, as is typical in foundation models.

**Table 1: Preliminary Statistics of AI Developers. We compare three leading open-source code agents using metrics such as average cost, average time, and the average number of generated files.**
| Metric | MetaGPT (Hong et al., ) | GPT-Pilot (Pythagora.io, ) | OpenHands (Wang et al., ) |
| --- | --- | --- | --- |
| Basic Statistics |  |  |  |
| Version | Data Interpreter (Hong et al., ) | 0.2.13 | CodeAct v1.9 (Wang et al., ) |
| \hdashline(1) Average Cost | $1.19 | $3.92 | $6.38 |
| (2) Average Time | 775.29s | 1622.38s | 362.41s |
| (3) Average Input Tokens | 152863 | 606707 | 1252482 |
| (4) Average Output Tokens | 28546 | 59707 | 8457 |
| \hdashline(5) Average Saved Code Files | 0.42 | 3.84 | 2.53 |
| (6) Average Saved Code Lines | 11.15 | 273.33 | 96.56 |
| (7) Average Saved Files | 4.42 | 5.91 | 3.60 |
|  |  |  |  |

### 2.3 Preliminary Benchmark

We first conduct experiments to collect development outcomes from different frameworks, which serve as baselines in the DevAI dataset. We test three of the most popular open-source frameworks (which we will refer to as “AI developers”): MetaGPT (Hong et al., ), GPT-Pilot (Pythagora.io, ), and OpenHands (Wang et al., )—all selected for their strong community acceptance (each having over 30,000 stars on GitHub).

#### Experiment Setup

All of these three systems require a language model as a back-end engine, for which we use gpt-4o-2024-05-13, a state-of-the-art language model. These AI developers were given a time-limit of $1800$ seconds to solve each task and were forcefully halted if they exceeded this time limit (we imposed this constraint, which was visible to the AI developers, as detailed in Appendix I).
We capture the outputs generated during the automated development process, including code, files, and other artifacts. Additionally, we record key decisions and actions made by the agentic systems through some custom instrumentation code, resulting in a development trajectory for each of the agentic systems.

Analysis
The basic statistics are shown in Table 1. MetaGPT is the most cost-efficient ($1.19$ USD), while OpenHands is the most expensive ($6.38$ USD). In terms of development time, OpenHands completes tasks in an average of $362.41$s, while GPT-Pilot takes the longest at $1622.38$s. On average, a full evaluation on DevAI with one of these three took around $210.65$ USD and $14$ hours to perform.
While running, GPT-Pilot generates the most output tokens at $59707$ tokens, whereas OpenHands processed the most at $1252482$ tokens while producing the fewest at $8457$ tokens. This suggests that OpenHands’s
internal communication is more complicated but is more parsimonious in its decisions.

MetaGPT, while being the most cost-effective, generates fewer saved code files ($0.42$), suggesting it may be less inclined to save files. In contrast, GPT-Pilot generates the most saved files ($3.84$), reflecting a more prolific output. The difference in saved code lines, with GPT-Pilot saving $273.33$ lines versus MetaGPT’s $11.15$, underscores GPT-Pilot’s extensive output. Meanwhile, OpenHands, despite handling larger inputs, seems less focused on executing code to generate files, as evidenced by its lower file output ($2.53$ saved files). These statistics align with real user experiences (as discussed in Appendix F).

Evaluations
Note that the results in Table 1 are not directly indicative of performance but provide valuable insights into the practical utility of DevAI and the performance of AI developers.
The generated workspaces (generated files, code, etc.) and trajectories are utilized in subsequent experiments to perform evaluations using Human-as-a-Judge (section 3), LLM-as-a-Judge, and Agent-as-a-Judge (section 4).

## 3 Human-as-a-Judge: Manual Evaluation on DevAI

To determine the pragmatic validity of DevAI and to accurately estimate the actual code-generating abilities of current state-of-the-art agentic systems, in this section, we run and then manually evaluate the application of three AI developer baselines to DevAI.
In Section 4, we show how this evaluation can be automated.

**Table 2: Human-as-a-Judge for AI Developers. (I) and (D) represent independent performance versus performance considering task dependencies. indicates multiple experts evolved, and means the evaluations use white-box testing (allowing access to the generated workspace, human-collected trajectories, and open-source codebases). The results were derived from expert judgments and deliberations (see Appendix H).**
| Metric | MetaGPT (Hong et al., ) | GPT-Pilot (Pythagora.io, ) | OpenHands (Wang et al., ) |
| --- | --- | --- | --- |
| /  Human-as-a-Judge |  |  |  |
| (A) Requirements Met (I) | 22.13% | 44.80% | 42.89% |
| (B) Requirements Met (D) | 6.55% | 28.96% | 28.68% |
| (C) Self-Termination | 41.81% | 5.45% | 54.54% |
| (D) Task Solve Rate | 0.00% | 1.81% | 1.81% |
|  |  |  |  |

### 3.1 Benchmark Baselines by Human-as-a-Judge

Human Evaluation Setup   After obtaining the baseline executions and conducting basic statistical analysis, we have three expert human evaluators (referred to here by their anonymous names: 231a, 38bb, and cn90) review the outputs of AI developer baselines to assess whether each requirement was satisfied.
We have two rounds of human evaluations.
To capture the bias inherent in typical human evaluation
(this is desirable to capture here as it represents a likely scenario in deployment),
in the first round, our evaluators first discussed the basic standards but were given minimal instructions.
The templates the evaluators were given for the evaluation and their self-reported post-hoc descriptions of how they resolved ambiguities are reported in Figure 12 in Appendix H.

After the initial round of human evaluations (which totaled an estimated total of $58$ human hours), we asked our evaluators to discuss and reach a consensus on their assessments (which took an estimated total of $28.5$ additional human hours). This consensus, achieved after long sessions of debate, was used as the final human evaluation result for each method.

Performance Analysis
The results of this experiment are shown in Table 2.
We found that the two best-performing methods (GPT-Pilot and OpenHands) could satisfy about $29\%$ of the requirements (or around $44\%$ if prerequisites are ignored) but only on one task could they meet all the requirements.
This highlights that DevAI offers a considerable but appropriate level of challenge for current and future methods.
Moreover, the fulfillment of intermediate requirements aligns with our expectations, as discussed in Section 2, that DevAI provides richer feedback by uncovering how agentic systems falter during the process instead of just focusing on a single performance metric at the end.

### 3.2 Judging Human-as-a-Judge

Figure: Figure 4: Between the three human evaluators, a large amount of disagreement was observed in their individual evaluations—highlighting the inherent unreliability of a single human evaluation.
Refer to caption: extracted/5932469/FILES_IMAGES/disagreement_meta.png

#### Disagreement Analysis

To analyze the presence of inductive bias and the reliability of the Human-as-a-Judge paradigm here, we calculate the disagreement rate between individual evaluators (shown in Figure 4). The results indicate that the disagreement rates between pairs of evaluators range from around $10\%$ to $30\%$.

Due to the complexity of a complete AI development task, which typically involves multiple steps with varying outcomes at each step, humans can easily make errors when critical information is missed, such as environment feedback indicating small but severe coding errors or bugs. Additionally, some disagreements are not necessarily incorrect but arise from differing perspectives on how ambiguity should be resolved.

#### Error Analysis

As previously noted, the evaluators engaged in a round of debating after their initial evaluations until they reached a consensus on each requirement in each task (with the results of this consensus evaluation shown in Table 2).

In our Human-as-a-Judge pipeline, evaluators could be convinced by evidence from others and acknowledge their judgment errors, adjusting their answers accordingly. This can be used to approximate individual errors. If the consensus evaluation more accurately predicts any extant ground truth, we would expect the majority vote from the individual evaluations to more closely approximate this than any single evaluation, due to the fundamental properties of ensemble classifiers (see Hastie et al. ()).

While the consensus evaluation may not represent the absolute ground truth (we acknowledge that some quantity of error likely would still exist even after this procedure), we expect the consensus evaluation to more accurately approximate any extant ground truth (Clemen, ).
If this holds, the majority vote should align more closely with the consensus than with any individual evaluation.
As shown in Figure 5, this is the case.

As seen in the results, although significant errors occur among all evaluators, the majority vote effectively corrects most of these errors. Notably, cn9o made the most errors (for example, $23.77\%$ in evaluating GPT-Pilot). After applying the majority vote from all three evaluators, the overall error rate dropped to $6.01\%$, demonstrating the inherent benefits of majority voting.

Figure: Figure 5: Mismatch between the individual evaluations and the consensus evaluation. In particular, the majority vote classifier showed the smallest deviation from the consensus evaluation.
Refer to caption: extracted/5932469/FILES_IMAGES/human_error_rate_meta.png

#### Conclusion

Human judgment errors are inevitable. To reduce them, we suggest two methods. First, like in this work, introduce a debate round after each judgment, where individuals present evidence and either persuade others or adjust their own opinions after discussion. This is particularly important when there are only a few evaluators, as majority voting with a small group can still lead to errors (around $5\%$ compared to consensus evaluation, as shown in Figure 5).
The second approach involves assembling a larger panel of experts (more is better when their accuracy exceeds $50\%$ (Grofman et al., )), with over $5$ people recommended by Hastie and Kameda (); Larrick and Soll (), and relying on a majority vote. However, due to the high cost of engaging more experts and the fact that this is not always feasible in practice, we argue for the former.

**Table 3: AI Judges and Their Shift/Alignment with Human-as-a-Judge. We compare the results of LLM-as-a-Judge and Agent-as-a-Judge with Human-as-a-Judge. (I) represents performance on independent tasks, while (D) represents performance considering task dependencies. Note: gray-box settings use carefully manually collected trajectory data (which is nearly inaccessible in practical situations, see Appendix J). In contrast, black-box setting doesn’t need to access to such data. The red scores represent the absolute judge shift compared with Human-as-a-Judge (*e.g.*, 2.74%).**
| Metric | MetaGPT (Hong et al., ) | GPT-Pilot (Pythagora.io, ) | OpenHands (Wang et al., ) |
| --- | --- | --- | --- |
| LLM-as-a-Judge |  |  |  |
| (a) Requirements Met (I) | 19.39% (2.74%) | 12.56% (32.24%) | 11.47% (31.42%) |
| (b) Requirements Met (D) | 1.63% (4.92%) | 4.09% (24.87%) | 2.18% (26.50%) |
| (c) Task Solve Rate | 0.0% (0.0%) | 0.0% (1.81%) | 0.0% (1.81%) |
| \hdashlineAlignment Rate $\uparrow$ | 84.15% | 65.30% | 60.38% |
| Agent-as-a-Judge |  |  |  |
| (I) Requirements Met (I) | 25.40% (3.26%) | 53.00% (8.20%) | 42.62% (0.27%) |
| (II) Requirements Met (D) | 5.73% (0.81%) | 39.89% (10.93%) | 26.50% (2.17%) |
| (III) Task Solve Rate | 0.0% (0.0%) | 5.45% (3.64%) | 1.81% (0.00%) |
| \hdashlineAlignment Rate $\uparrow$ | 88.52% | 83.88% | 90.44% |
| LLM-as-a-Judge |  |  |  |
| (a) Requirements Met (I) | 28.68% (6.55%) | 38.79% (4.10%) | 43.16% (0.27%) |
| (b) Requirements Met (D) | 17.75% (11.20%) | 33.06% (4.10%) | 32.24% (3.56%) |
| (c) Task Solve Rate | 1.81% (1.81%) | 3.63% (1.82%) | 7.27% (5.46%) |
| \hdashlineAlignment Rate $\uparrow$ | 68.86% | 71.85% | 70.76% |
| Agent-as-a-Judge |  |  |  |
| (I) Requirements Met (I) | 23.49% (1.35%) | 46.44% (1.64%) | 43.44% (0.54%) |
| (II) Requirements Met (D) | 6.01% (0.54%) | 30.60% (1.64%) | 28.14% (0.53%) |
| (III) Task Solve Rate | 0.0% (0.00%) | 5.45% (3.64%) | 3.63% (1.82%) |
| \hdashlineAlignment Rate $\uparrow$ | 92.07% | 86.61% | 90.16% |
| /  Human-as-a-Judge |  |  |  |
| Alignment Rate (38bb) | 92.63% | 90.98% | 89.89% |
| Alignment Rate (cn9o) | 83.33% | 76.23% | 78.15% |
| Alignment Rate (231a) | 92.07% | 87.43% | 89.07% |
| \hdashline Average of individuals | 89.34% | 84.88% | 85.70% |
| Best of individuals | 92.63% | 90.98% | 89.89% |
| Alignment Rate (Majority Vote) | 95.08% | 93.98% | 94.26% |
|  |  |  |  |

## 4 Agent-as-a-Judge: Evaluating Agents with Agents

Human evaluation, while somewhat reliable, is time-consuming and requires substantial expertise. To address this, we propose the Agent-as-a-Judge framework.
If such an agentic system could evaluate like a human, it would reduce the need for human involvement and eliminate the trade-off between thoroughness and effort.

### 4.1 Proof-of-Concept

Based on our prior experiences with agent design and by imitating the human evaluation process, we initially designed eight modular, interacting components that form the foundation of our Proof-of-Concept for the Agent-as-a-Judge.

Figure: Figure 6: Initial diagram of Agent-as-a-Judge.
Refer to caption: x3.png

(1) The graph module constructs a graph that captures the entire structure of the project, including files, modules, and dependencies. It can also break down chunks of code into code snippets.
(2) The locate module identifies the specific folder or file referred to by a requirement.
(3) The read module goes beyond simple file parsing, supporting the reading and understanding of multimodal data across $33$ different formats, including code, images, videos and documents. This allows the agent to cross-reference various data streams and verify different kinds of requirement.
(4) The search module provides a contextual understanding of code and can quickly retrieve highly relevant code snippets, as well as the nuances behind them (e.g., hidden dependencies).
(5) The retrieve module extracts information from long texts, identifying relevant segments in trajectories.
With context from the above, (6) the ask module determines whether a given requirement is satisfied.
(7) The memory module stores historical judgment information, allowing the agent to build on past evaluations.
Finally, (8) the planning module plans the following actions, allowing the agent to strategize and sequence tasks based on the current state and the project goals.

Our initial design of the Agent-as-a-Judge, including all its components, is shown in Figure 6, and the operational process of the Agent-as-a-Judge is illustrated in Figure 9.

After conducting comprehensive ablation studies, we found that the modular combination of (1), (2), (3), (5), and (6) achieved the highest performance (see Appendix K). A sample of the dynamic evidence collected by the Agent-as-a-Judge is shown in Appendix M.
We hypothesize this is because Agent-as-a-Judge needs high-quality factual information and is sensitive to noise. For example, while our design of the planning module introduces promising decision-making for future actions, the procedure is unstable. Initially, we hoped that historical information from the memory module would help to assess current requirements. However, it proved detrimental, as any errors in previous judgments could lead to a chain of errors, negatively affecting current decisions. Besides, the current workspaces generated by developer agents, having only hundreds of lines of code, cannot fully benefit from the search module.
The details of these findings are explained in Appendix K.
Note that a perfect Agent-as-a-Judge is not the focus of this proof of concept, and thus, we leave the utilization of advanced agentic optimization methods for Agent-as-a-Judge, such as automated prompt optimization and workflow design (Zhuge et al., ; Hu et al., ), for future work.

### 4.2 Judging Agent-as-a-Judge and LLM-as-a-Judge

Judge Shift   Judge Shift measures deviation from the Human-as-a-Judge consensus results, with lower values indicating a closer alignment. As shown in table 3, Agent-as-a-Judge consistently outperforms LLM-as-a-Judge across tasks, particularly those with task dependencies. For example, in Requirement (I), Agent-as-a-Judge shows a Judge Shift as low as $0.27\%$, while LLM-as-a-Judge reaches $31.24\%$ for OpenHands. This underscores Agent-as-a-Judge’s stability and suitability for meeting task requirements. Furthermore, in the gray-box setting, both Agent-as-a-Judge and LLM-as-a-Judge show even better results than their performance in the black-box setting.

Figure: Figure 7: PR Curves comparing judge Methods.
Refer to caption: extracted/5932469/FILES_IMAGES/pr_curves_meta.png

Alignment Rate   The Alignment Rate reflects how closely the AI Judges’ evaluations align with human consensus across all $365$ requirements. It is defined as the percentage of requirement evaluations that are the same as the Human-as-a-Judge consensus evaluation.
Compared to LLM-as-a-Judge, Agent-as-a-Judge consistently achieves a higher Alignment Rate, closely matching human judgments. For example, when evaluating OpenHands, Agent-as-a-Judge reaches $92.07\%$ and $90.44\%$, surpassing LLM-as-a-Judge’s $70.76\%$ and $60.38\%$ in both gray-box and black-box settings. This shows that Agent-as-a-Judge produces more accurate and human-aligned evaluations, especially in complex scenarios.

PR Curves   Judging developer agents is a class-imbalanced task, where meeting requirements is much rarer than failing. Metrics like judge shift and alignment rate can be misleading. For example, since MetaGPT rarely meets requirements, LLM-as-a-Judge easily identifies most cases as negative (achieving $84.15\%$ in the black-box setting). PR Curves offer a clearer performance measure by balancing precision and recall. Agent-as-a-Judge even outperforms any single human evaluator on OpenHands and aligns closest with majority voting. This shows that, in some cases, Agent-as-a-Judge can nearly replace human evaluators.

### 4.3 Ablations For Agent-as-a-Judge

We conduct ablations to evaluate the impact of adding different components on Agent-as-a-Judge’s performance. The components analyzed include ask, graph, read, locate, and retrieve. The component ablation study for Agent-as-a-Judge reveals key insights into the performance gains from adding specific functionalities.

**Table 4: Component Ablation Studies for Agent-as-a-Judge. We analyze the impact of adding various components (ask, graph, read, locate, and retrieve) on the performance of Agent-as-a-Judge for judging OpenHands.**
| Metric | + ask | + graph | + read | + locate | + retrieve |
| --- | --- | --- | --- | --- | --- |
| Agent-as-a-Judge Performance |  |  |  |  |  |
| Alignment Rate | 65.03% | 75.95% | 82.24% | 90.44% | 90.16% |
|  |  |  |  |  |  |

With only the ask component, the agent achieves a $65.03\%$ alignment rate. Adding the graph component increases performance to $75.95\%$, as the agent can better understand the relationships between files.

The introduction of read further improves the alignment rate to $82.24\%$, reflecting the value of direct access to the contents of the file. Incorporating locate brings a substantial boost to $90.44\%$, as the agent can efficiently target files relevant to the requirements. However, adding retrieve does not provide a significant benefit in this case. In contrast, as shown in Table 3, the judgment of MetaGPT and GPT-Pilot indicates that retrieve is useful, as the trajectory provides additional valuable information.

### 4.4 Cost Analysis

The Human-as-a-Judge took the three evaluators a self-reported total of $86.5$ hours. With a $15$ USD minimum wage (assuming this would buy a subject expert in AI), a full evaluation under DevAI would cost around $1297.50$ USD. In comparison, Agent-as-a-Judge cost only $30.58$ USD in API calls and took only $118.43$ minutes—$2.29\%$ of the cost and $2.36\%$ of the time of Human-as-a-Judge. LLM-as-a-Judge was faster at $10.99$ minutes, but due to the absence of intelligent context selection by the Agent-as-a-Judge’s modules, it still cost $29.63$ USD.

## 5 Related Work

Agentic systems and their applications are highly active research areas with numerous recent works having a relation to this work.
This section details those works most relevant to ours.
We provide a treatment of the less relevant related works in Appendix D.

AI Developers
AI in software development is growing fast (Liu et al., ). AI-driven developers have been applied to directly imitate software companies (Hong et al., ; Qian et al., ), debug code (Yang et al., ), run data science methods (Guo et al., ; Hong et al., ; Li et al., ; Qiao et al., ), and even write academic papers (Lu et al., ).

Benchmarks for AI developments
Benchmarks like MLAgentBench (Huang et al., ), ML-Bench (Liu et al., ), SUPER (Bogin et al., ), DS-bench (Jing et al., ), and MLE-Bench (Chan et al., )
all focus on benchmarking agentic systems using AI tasks.
However, DevAI distinguishes itself from all of these by focusing on realistic user queries that target a complete development cycle.
It further includes a more comprehensive evaluation with multiple hierarchical requirements and preferences for each task.
Comparatively, MLAgentBench (Huang et al., ) for example, focuses on final performance for a limited set of well-known tasks, which risks overfitting and fails to assess a system’s generalization or adaptability.

AI Judges
Several works have looked at using AI systems as judges(^3^33Additionally, we were pleased to find that a recent industry blog ([https://www.cognition.ai/blog/evaluating-coding-agents](https://www.cognition.ai/blog/evaluating-coding-agents)), published 3 weeks before our submission, shares very similar ideas and provides further evidence that the Agent-as-a-Judge could have practical applications in agent systems.).
The work by Chan et al. (); Zhao et al. (), for example, extends LLM-as-a-Judge to have multiple LLMs in their evaluation process for conversational tasks.
Unlike Agent-as-a-Judge, they employ a trivial agentic system and apply it only to evaluate LLMs under traditional evaluation setups.
In contrast, (Lu et al., ) uses a single LLM-based evaluator but, unlike LLM-as-a-Judge, applies this to multimodal tasks rather than just for evaluating LLMs.
Less relevant are frameworks like those by Chen et al. (); Arora et al. (); Mündler et al. (), where intermediate signals are used during coding development.

## 6 Discussion and Conclusion

#### Outlook 1: Intermediate Feedback for Agentic Self-Improvement

A key power of the Agent-as-a-Judge, though not fully exploited here but nonetheless clear, is that it provides intermediate feedback that is essential for effective and efficient optimization (Zhuge et al., ; Pan et al., ).
For example, Agarwal et al. () proposes to solve the sparse reward problem in reinforcement learning, by learning auxiliary reward functions that provide intermediate feedback.
Perhaps the greatest strength of the Agent-as-a-Judge framework is that an agentic system can use it to identify and fix issues in its solutions to complex, multistage problems on the fly—something older, delayed-feedback methods did not permit.
By introducing Agent-as-a-Judge, we create the opportunity to build a process-supervised reward model (PRM) for improving agentic systems (Lightman et al., ).

#### Outlook 2: Flywheel Effect Driven by Agent-as-a-Judge

The cycle of mutual improvement between the Agent-as-a-Judge and the evaluated agents, where both evolve together through iterative feedback, presents a promising direction. We hypothesize that an agentic version of a self-play system (Zelikman et al., ; Chen et al., ; Wang et al., ), could viably emerge by using the Agent-as-a-Judge as a key mechanism. Furthermore, the ongoing interaction between the Agent-as-a-Judge and the evaluated agents has the potential to create a flywheel effect, where successive incremental improvements reinforce one another, leading to progressively greater optimization and enhanced performance over time (Wang et al., ). This iterative process may also serve as a valuable complement to LLM reasoning data, help embedding agentic capabilities into foundation models (Luo et al., ).

#### Conclusion

In this work, we introduced the Agent-as-a-Judge method to use agentic systems to evaluate agentic systems. We simultaneously released DevAI: a new benchmark that evaluates the code-generating ability of agentic systems on complete AI development tasks when used with Agent-as-a-Judge. We went on to show that Agent-as-a-Judge outperforms existing methods on this task and that it performs similarly to an ensemble of expert human evaluators. Altogether, we believe that the above opens the door for scaling up agentic far more than before.

## Acknowledgements

The authors thank Haozhe Liu, Piotr Piekos, Firas Laakom, Matteo Paltenghi for their suggestions or paper review. The research reported in this publication was supported by funding from the King Abdullah University of Science and Technology (KAUST) - Center of Excellence for Generative AI under award number 5940 and the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence.

## Appendix A Outline of this Paper

Figure: Figure 8: We Outline the Logical Flow of the Agent-as-a-Judge Framework.

## Appendix B Experiment Designs

This section outlines the experimental designs aimed at evaluating developer agents’ performance, analyzing human evaluations, and comparing AI-based judging systems. The experiments are structured across three levels, as illustrated below.

### B.1 Summary of Experiments

The experiments are categorized into three levels as follows:

- Level 1
🍂
*Experiment 1a:* Basic performance statistics for developer agents (Section 2.3)
🍂
*Experiment 1b:* Human evaluations of developer agents (Section 3.1)
- Level 2
🍂
*Experiment 2a:* Error analysis of human evaluations (Section 3.2)
- Level 3
🍂
*Experiment 3a:* AI judge baselines (Section 4.2)
🍂
*Experiment 3b:* Ablation studies for Agent-as-a-Judge (Section 4.3)

### B.2 Judges and Subjects of Evaluation

The following table summarizes the judge and the subject being evaluated in each experiment:

| Experiment | Who is the Judge? | Who is being Judged? |
| --- | --- | --- |
| Section 2.3 | *Human* | *Developer Agents* |
| \hdashlineSection 3.1 | *Human* | *Developer Agents* |
| Section 3.2 | *Human* | *Human* |
| Section 4.2 | (1) *LLM-as-a-Judge* | (1) *Developer Agents* |
|  | (2) *Agent-as-a-Judge* | (2) *Developer Agents* |
|  | (3) *Human* | (3) *LLM-as-a-Judge* |
|  | (4) *Human* | (4) *Agent-as-a-Judge* |
| \hdashlineSection 4.3 | *Human* | *Agent-as-a-Judge* |
|  |  |  |

## Appendix C Agent-as-a-Judge Pipeline

Figure: Figure 9: The pipelines of developer agents and judge agent. Some materials in this figure are from original blog ([https://www.factsmachine.ai/p/hidden-in-plain-sight](https://www.factsmachine.ai/p/hidden-in-plain-sight)).
Refer to caption: x5.png

## Appendix D Extend Related Work

Our main paper includes mostly related works of AI developers, Benchmarks for AI developments, and AI judges.
However, the following works contribute significantly to the community and also relate to this work. We record this work as additional related work.

#### LLM-based Autonomous Agents

Recent developments in LLM-based agents have expanded their capabilities beyond simple task execution to more autonomous problem-solving and decision-making. AutoGPT (Gravitas, ) and LangChain (Chase, ) provide frameworks for single-agent systems that leverage external tools for more complex tasks. Similarly, research such as CAMEL (Li et al., ), MetaGPT (Hong et al., ), NLSOM (Zhuge et al., ), AutoGen (Wu et al., ) focus on role-based multi-agent communication, improving collaboration among agents. However, the challenge of maintaining coherence in agents’ dialogue and preventing hallucination remains prominent (Du et al., ; Zhou et al., ). Besides, Agent-trust (Xie et al., ) examines if LLM agents, like GPT-4, can simulate human trust behaviors in Trust Games, showing that they can align with human behavior.

Most recently, using graphs to build agents has gained prominence. Earlier work like GPTSwarm (Zhuge et al., ) and LangGraph (langchain ai, ) proposed using nodes to represent operations and edges to represent the connections between them. In GPTSwarm, multiple agents represented as subgraphs in a graph are connected by optimizable edges, and reinforcement learning is employed to optimize the edges. Following this approach, several agent frameworks have incorporated graphs into their designs (Hong et al., ; Zhou et al., ; Qian et al., ). Additionally, various optimization methods have been developed to enhance agent performance further (Wu et al., ; Song et al., ; Hu et al., ). In practical applications, many studies focus on understanding and interacting with GUIs (Wang et al., ; Chen et al., ; Yang et al., ; Xu et al., ; Tan et al., ; Wu et al., ). For code generation agents (Jin et al., ), current research mainly emphasizes automated repair (Yang et al., ; Phan et al., ; Tao et al., ), computational modular design (Khattab et al., ; Cheng et al., ), and automated development (Tufano et al., ; Huang et al., ). Among these, open-sourced frameworks like OpenHands (Wang et al., ) have gained popularity due to their strong user experience. Moreover, scientific discovery (Jansen et al., ; Lu et al., ) and ML agents (Yang et al., ) are also receiving increased attention.

#### LLM-as-a-Judge

In the domain of AI evaluation and judgment, frameworks (Zheng et al., ; Fu et al., ) have pioneered the use of LLMs to assess conversational agents, demonstrating how LLMs can evaluate dialogue quality and consistency.
LLM-as-a-judge has also expanded into the multimodal domain, providing clear visual-language feedback (Chen et al., ; Xiong et al., ).
ICE-Score (Zhuo, ) improves on metrics like CodeBERTScore (Zhou et al., ) and G-EVAL Liu et al. () by better aligning with human preferences and functional correctness.
Expanding beyond dialogue, LLMs like CodeR (Chen et al., ) and MASAI (Arora et al., ) apply similar judging principles to the code validation process, where AI systems autonomously evaluate and verify computer programs. Our work builds on these advancements by exploring how LLMs can perform more nuanced judgment tasks, further investigating their potential in decision-making across various domains. Recent research also focuses on judging LLM-as-a-Judges (Chen et al., ; Bavaresco et al., ; Thakur et al., ; Dong et al., ; Shi et al., ; Raina et al., ).

#### Coding Benchmarks

Recent advances in code generation have led to the innovation of various benchmarks to evaluate model performance (Liu et al., ). Early benchmarks, such as MBPP (Austin et al., ), HumanEval (Chen et al., ), and MultiPL-E (Cassano et al., ), focus primarily on generating simple functions. While these benchmarks are useful for evaluating the correctness of generated code, they are limited in complexity and do not fully represent the challenges encountered in real-world software development.

As the field progressed, newer benchmarks began to focus on more complex and realistic tasks. APPS (Hendrycks et al., ),
CodeContests (Li et al., ), and LiveCodeBench (Jain et al., ) moved toward competitive programming challenges that involve advanced algorithms and data structures. These tasks are more representative of problems encountered in coding competitions and help push models toward more sophisticated problem-solving.
DS-1000 (Lai et al., ) was introduced to assess the skills of models with data science libraries, evaluating their ability to use APIs and execute complex data analysis workflows. Meanwhile, AgentBench (Liu et al., ) focuses on testing reasoning and decision-making abilities in interactive environments, highlighting differences in performance between commercial and open-source models.

To address real-world programming needs beyond code generation, specialized benchmarks have been created to evaluate tasks such as debugging, refactoring, and code navigation. BigCodeBench (Zhuo et al., ) has advanced the evaluation of LLMs by testing their ability to solve complex, real-world programming tasks beyond traditional algorithmic challenges. CANITEDIT (Cassano et al., ), DebugBench (Tian et al., ), and FixEval (Haque, ) evaluate the ability of a model to edit and improve existing code. Additionally, benchmarks such as SWE-Bench (Jimenez et al., ) focus on resolving issues in GitHub repositories, simulating practical software development scenarios. Finally, benchmarks such as RepoBench (Liu et al., ) and RepoEval (Zhang et al., ) delve into the evaluation of models in large-scale, multifile codebases.
These benchmarks measure the ability of language models to understand the structure of repositories and solve problems within more complex, collaborative environments.

There are also tasks that assess tool-use skills requiring programming knowledge, though not directly focused on programming. CRAB (Xu et al., ) sets a new standard for benchmarking autonomous agents by supporting cross-environment tasks and offering a more comprehensive evaluation framework. Similarly, WebArena-serious (Zhou et al., ; Koh et al., ) introduces a realistic benchmark for evaluating multimodal agents on complex, real-world web tasks, addressing the limitations of text-only agents by integrating both visual and textual inputs across diverse domains.

## Appendix E The Procedures of Creating DevAI Dataset

### E.1 Manually draft user queries

Given the execution cost of the developer agents, we collect small-scale AI tasks to ensure the practical applicability of our benchmark.
Since these tasks are small-scale and well studied, which are easy to overfit in terms of task performance metrics, unlike previous benchmarks (e.g.,  (Huang et al., )), we do not evaluate task performance as the development performance measure.
Instead, we prioritize the step-by-step task-solving ability, which is essential for real-world development.
Our quires are specifically designed to require the development agents to understand user intentions, solve the task in multiple steps, and adapt to unexpected step outcomes.
This approach also makes our benchmark user-friendly, transparent, and better reflects real-world deployment situations.
To enable effective evaluation, our queries present a specific development file structure for the developer agents to follow.
To ensure that the developer agents save the files to be evaluated in the workspace, we develop constraint prompts added to the query to form an extended query. The constraint prompts guide the developer agents to save and execute the source codes, which are in line with the real-world development standard. See Appendix I for our constraint prompts.

### E.2 Set Judging Criteria

To make the evaluation of the developer agent precise, we assign to each task query a list of requirements as task milestones.
The requirements are chosen so that satisfying all the requirements is a necessary condition to consider the task to be solved successively.
Since our tasks are AI-centric, our queries target essential elements of AI development, including data processing, AI method, presentation of evaluation metrics, visualization, and human-computer interaction, covering the key areas that matter most in real-world scenarios.
On the other hand, breaking down tasks into individual requirements also reflects the multi-step nature of code development.
Importantly, to avoid ambiguity, we set the requirements to be explicit, binary, and straightforward to evaluate. To include other human predispositions, we include a list of preferences per task that covers subjective, ambiguous, or non-explicitly stated characteristics.

### E.3 Building Dependency Among Requirements

To enhance the realism of our benchmark, we analyzed the dependencies among requirements. Over the past decades, methodologies such as the KDD Process (Fayyad et al., ) and CRISP-DM (Wirth and Hipp, ) have guided ML/AI development, establishing foundational frameworks that have been further refined with the rise of AutoML (He et al., ).

Inspired by these methodologies, we identified a seven-step process for automated AI development tasks. This process includes critical stages such as data preprocessing, feature engineering, model selection, and hyperparameter tuning, along with essential post-development activities like metrics recording, report generation, and the development of interactive user applications. This structured approach allows us to evaluate the agent’s ability to manage task dependencies and effectively navigate complex, real-world scenarios.

### E.4 Refine the dataset

Manual refinements were necessary to ensure the accuracy and clarity of DevAI.
We perform two rounds of comprehensive review and edits on DevAI, each round being done by a different participant.
During these manual refinements, we focus on the logical consistency of our queries and requirements, the correctness and ambiguity of the language, and the applicability of the content to the task domain.
We identified a moderate number of errors in our dataset during the review.

### E.5 Analyse the dataset

We categorized each requirement based on its focus, whether it was related to the data set, the machine learning method, visualization, metrics, HCI, or data processing. While this approach provides a useful framework, it is important to recognize that these categories might overlap or miss certain nuances. Similarly, preferences were classified by how strongly they appeared in the query text, ranging from those inferred by common sense to those explicitly stated. Although this helps to organize preferences, it is worth noting that such classifications can be subjective and may not fully capture the importance of each preference in practical applications. By structuring the requirements and preferences this way, we aim to enhance the evaluation process, though flexibility and context awareness remain crucial for truly robust assessments.

### E.6 Auxiliary Information

Some of the tasks require the download of a Kaggle  data set, where a Kaggle credential is needed.
Our constraint I requires an “is_kaggle_api_needed” tag to determine whether the credential is needed to be included in the extended query.
We further mark each task with some tags describing the AI sub-fields related to the task, including computer vision, supervised learning, reinforcement learning, natural language processing, etc., as well as two “is_training_needed” and “is_web_navigation_needed” tags as auxiliary information.
We also categorize each requirement into one of the following: (1) dataset or environment, (2) data preprocessing and postprocessing, (3) machine learning method, (4) save trained model, (4) performance metrics, (5) human computer interaction, (6) visualization, and (7) other, reflecting the nature of the requirement.

### E.7 A Json Format of Our Sample

Here, we provide a sample of the DevAI with its json format. We also provide more samples in Appendix G.

## Appendix F User experiences of code-generation agentic systems

OpenHands (Wang et al., ) offers the most refined user experience, leveraging its highly interactive frontend to enable seamless user interaction and task execution. This interface allows users to engage directly with the system, resulting in a smoother and more intuitive workflow, which drives operational efficiency.

In contrast, MetaGPT (Hong et al., ) excels in task decomposition through its use of Directed Acyclic Graphs (DAGs), a well-structured and scalable approach aligned with industry best practices in system modularization. This enhances its appeal for users focused on task clarity and modular breakdowns. However, in practical deployments, MetaGPT tends to be less aggressive in file management and preservation, potentially due to its core positioning as a data analysis tool, which does not prioritize persistent state management. Similarly, OpenDevin demonstrates a notable overconfidence in its code generation, frequently skipping the critical step of post-generation code execution, requiring users to intervene manually.

GPT-Pilot (Pythagora.io, ), praised for its detailed task delegation via over 20 specialized agents, suffers from reduced interactivity due to an overly granular division of responsibilities, resulting in a more fragmented user experience. These qualitative insights, although not fully captured by quantitative metrics, were evident through the DevAI dataset, providing key areas for improvement in user engagement and operational fluidity in future releases of these frameworks.

## Appendix G More DevAI dataset samples

Figure: Figure 10: An Example Task in DevAI: Task 13.

Figure: Figure 11: An Example Task in DevAI: Task 19.

## Appendix H Human Evaluation Procedure

We recruited three AI experts from the authors to perform human evaluation on the output of agentic code generation systems. There we present the evaluation details.

#### First round

For the first round of evaluations, our three evaluators reported spending $16.5$, $19.5$, and $22.0$ hours, respectively.
To capture the bias that a human evaluator will have, the instructions given to our experts were minimal, with them only receiving a scorecard to complete for each agentic system and each task.
Results that all evaluators agree on are considered trustworthy. The assumption here is that it is unlikely that all three evaluators make a mistake or have an effective bias in the same judgment.
The self-reported post-hoc evaluation criteria are shown in Figure 12.

#### Second round

In the second round, the evaluators present and discuss their reasons for disagreeing with judges. In doing so, human errors are likely corrected by their peers. Discussion among evaluators also helps reduce human bias by examining each other’s thought processes thoroughly. Furthermore, the consensed results are considered trustworthy given the assumption that it is unlikely that all three evaluators are convinced by the same mistake or the same cognitive bias. The three evaluators took 9.5 hours together for this second round of evaluation.

Figure: Figure 12: Each evaluator was given a full description of each task and the associated requirements and preferences in markdown format. They were then asked, for each workspace and trajectory generated by each of the agents on each of the task, whether (1) the agent successfully finished its execution cycle and (2) which of the requirements were satisfied. After the evaluation was complete, the evaluators were asked to self-report the nuances of their evaluation.

## Appendix I Suggest Constraints

Below is a sample of constraints in JSON format that describes task-specific guidelines:

To address automation and security concerns, we have written code to place the kaggle.json file into the current workspace for each baseline during each run.
However, dataset users are free to modify the solution to enable the Kaggle API for their specific developer agents.

## Appendix J Collected Trajectories

### J.1 Schema

Below is the required JSON format for a trajectory in gray-box settings (where the trajectories can serve as input for LLM-as-a-Judge and Agent-as-a-Judge).

### J.2 A Sample of Trajectory

This is sample trajectories (we select 4 steps of it) in OpenHands (Wang et al., ). All steps are recorded to enable replay of the trajectory.

## Appendix K Ablations of Agent-as-a-Judge

### K.1 Component Abalations

**Table 5: Component Ablation Studies for Agent-as-a-Judge. We analyze the impact of sequentially adding various components (ask, graph, read, locate, retrieve, planning, memory, and search) on the performance of Agent-as-a-Judge for judging OpenHands.**
| Metric | + ask | + graph | + read | + locate | + search | + retrieve | + planning | + memory |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Agent-as-a-Judge Performance |  |  |  |  |  |  |  |  |
| Alignment Rate | 65.03% | 75.95% | 82.24% | 90.44% | 86.06% | 90.16% | 88.52% | 87.97% |
|  |  |  |  |  |  |  |  |  |

Analysis
We designed 8 modular components for the Agent-as-a-Judge system. In the Table 5, components are added progressively from left to right. If the addition of a component led to a significant performance drop, we removed it from further iterations. Our experiments showed that adding the components ask, graph, read, and locate resulted in significant performance gains. However, when the search component was introduced, there was a noticeable decline in performance.

We hypothesize that the performance drop from search is due to its role in retrieving relevant code snippets (top-$3$) using BM25. The retrieval accuracy of BM25 (Robertson et al., ) might not have been high enough, potentially introducing noise. Moreover, as noted in Table 1, the DevAI tasks in our experiments did not generate a large amount of code. In fact, even when all code was fed into an LLM, the total content typically stayed within the maximum context length. Therefore, in simpler workspaces, search was less critical. However, we believe this component will become more important as the complexity of the workspace increases, making it more valuable in larger and more complex environments.

We also observed that the introduction of the planning mechanism did not bring a noticeable improvement in performance. This may be related to the nature of the Judge - it needs clean factual information. When planning is unstable, the evidence collected from different actions can become inconsistent, leading to a decline in performance. Finally, we experimented with a memory mechanism. Initially, we hypothesized that since DevAI tasks often involve interconnected requirements, memory could help track whether requirements were met. However, in practice, we saw no improvement. We suspect that the interconnected nature of the requirements may have caused biases: specifically, once a prior requirement was fulfilled, it might have overly influenced positive judgments on subsequent requirements, even if they were not fully met.

### K.2 Search Algorithms in Search Module

We initially hypothesized that the performance drop was due to the low precision of the search component, particularly with BM2.5. To explore this, we replaced BM2.5 with Sentence-BERT (Reimers, ) as a more advanced alternative and tested Fuzzy Search (Levenshtein, ) as a less precise option. However, neither improved the performance of the Agent-as-a-Judge.

**Table 6: Comparisons on Search module with different engines.**
| Search Method | Alignment Rate |
| --- | --- |
| BM2.5 | 86.06% |
| Sentence-BERT | 87.70% |
| Fuzzy Search | 85.52% |
| *without* Search Module | 90.44% |
|  |  |

hese results suggest that the performance issue is not due to BM2.5’s poor search accuracy. Instead, the workspaces generated in our DevAI tasks are too simple for the search component to have a significant impact. In simpler workspaces, direct retrieval and evaluation are sufficient. Even though Sentence-BERT performed better than the other methods, its alignment rate ($87.70\%$) still falls short of the configuration without the search component ($90.44\%$). As workspace complexity increases, the search component may become more valuable.

### K.3 Search Algorithms in Retrieve Module

In our experiments, we found that accurately locating relevant information within a trajectory is a challenging task. Although the addition of the retrieve component (gray-box) did not lead to a significant improvement in performance in this specific case, its impact has been notable in other settings, such as in GPT-Pilot. As shown in Table 3, the integration of retrieve in GPT-Pilot brought substantial gains.

**Table 7: Ablations on retrieve.**
| Method | Alignment Rate |
| --- | --- |
| Without retrieve | 83.88% |
| With retrieve (gray-box) | 86.61% |
| Trajectory Truncate (head) | 86.61% |
| Trajectory Truncate (middle) | 85.52% |
| Trajectory Truncate (tail) | 82.51% |
| Step Truncate (head) | 86.34% |
| Step Truncate (middle) | 86.61% |
| Step Truncate (tail) | 83.88% |
|  |  |

We conducted an ablation study on GPT-Pilot to optimize the retrieval of useful information at each step. Our experiments revealed that in large trajectories, truncating the final sections of the file often results in losing critical information, as the latter part of the trajectory typically contains dense information about the final development state. Truncating the beginning of the trajectory proved to be the most effective in improving the retrieval efficiency.

For individual steps, truncating the middle section worked best. This is because error messages usually appear early in the output, while the corresponding file paths and specific error locations are found towards the end. By focusing on these retrieval strategies, we can significantly enhance the performance of the retrieve component, particularly in complex scenarios like GPT-Pilot.

## Appendix L Prompt Demos of Agent-as-a-Judge

Here, we present some prompts used by the Agent-as-a-Judge system. Each of these prompt demos plays a crucial role in guiding the agent’s behavior.

### L.1 System Prompt for Agent-as-a-Judge

### L.2 System Prompt for Locate Module

### L.3 System Prompt for Retrieve Module

### L.4 Prompt for Ask Module (for requirement check)

### L.5 Prompt for Locate Module

## Appendix M Judge Evidences Collected from Agent-as-a-Judge

The Agent-as-a-Judge system dynamically collects evidence throughout the development process of the code-generating agentic system to evaluate whether each requirement is satisfied. Using this auxiliary information, the judge agent decides to focus on relevant data such as code, file structures, and trajectories. This gathered evidence supports the agent’s final judgment, ensuring an informed and accurate assessment of the project’s compliance with its requirements. A sample of the collected evidence is shown in the following Appendix M.

[Uncaptioned image]: x6.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 1
questions: 1. While the AAAJ framework achieves higher alignment rate than other baseline algorithms, chances are that AAAJ makes incorrect evaluations. Did you conduct any studies on the failure cases to see whether there are consistent patterns that may be exploited by agentic systems?
2. Can the AAAJ framework be evaluated in benchmarks other than code generation?
soundness: 2
strengths: 1. The introduction of the Agent-as-a-Judge framework addresses a critical gap in evaluating agentic systems by enabling feedback at each stage of task completion, making it especially suited for complex, multistep tasks.
2. The DevAI dataset provides a practical and comprehensive testbed for agentic systems, encompassing various AI development tasks that closely mirror real-world demands, which enhances the relevance of the evaluation.
3. The study compares Agent-as-a-Judge with both human evaluators and LLM-as-a-Judge, demonstrating superior alignment with human consensus while significantly reducing costs and evaluation time.
4. The paper includes extensive ablation studies, cost analysis, and performance metrics, providing insights into the efficacy of different components of the framework and supporting reproducibility.
summary: This paper introduces the Agent-as-a-Judge framework, an innovative approach for evaluating agentic systems by employing other agentic systems as evaluators. Building on prior methods like LLM-as-a-Judge, this framework provides intermediate feedback throughout the problem-solving process, moving beyond final outcome evaluation. The paper applies this approach to code generation tasks, introducing the DevAI dataset, consisting of 55 realistic development tasks with hierarchical solution requirements. The results show that Agent-as-a-Judge aligns closely with human evaluation and outperforms LLM-as-a-Judge in task accuracy and cost-effectiveness.
weaknesses: 1. The overall writing is inconsistent and confusing: in the abstract and introduction,  the Agent-as-a-Judge (AAAJ) framework is proposed with its novelty and effectiveness being emphasized. But in Section 2 and 3, the paper suddenly turns to the DevAI benchmark, introducing the motivation and technical details of this particular benchmark. The AAAJ benchmark is not comprehensively introduced until Section 4.  I feel like these two contributions, namely the AAAJ framework and the DevAI benchmark, should have been separated into two papers and discussed in more details respectively. Otherwise, it may be better to start from the DevAI benchmark, with agentic evaluation framework being one of its useful features.
2. Among the four principal contributions, the top two seem to be deviated from the main idea of this paper.
3. In Section 4, AAAJ is only discussed as a proof-of-concept, with little technical details. 
4. Code generation is only one of the many applications of agentic systems. There are plenty of other domains where LLM agents may help. It is therefore not enough to consider the task of code generation alone. The agentic evaluation framework also seems to be ad hoc to code generation.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - I'm not exactly sure of the meaning of independent tasks (I) and tasks considering task dependencies (D). Does R0 belong to (I) and R1, R2 belong to (D) in Figure 1? Maybe I missed something, could you please further clarify this?
- In line 486, how is Table 3 able to demonstrate the usefulness of the "retrieve" module?
- What is your backbone model for the LLM-as-a-Judge and Agent-as-a-Judge frameworks? Does using a different backbone affect your main results?

Please also pay attention to the questions and suggestions mentioned in the "Weaknesses" section.
soundness: 2
strengths: **Originality:**
The paper introduces the novel benchmark DevAI, which comprises 55 realistic AI code generation tasks with comprehensive manual annotations and hierarchical solution requirements. This benchmark addresses the gap in existing evaluation methods and is a significant contribution to the AI development community. The introduction of the Agent-as-a-Judge framework is another innovative aspect, proposing a novel method for evaluating agentic systems using other agentic systems. This extends the existing LLM-as-a-Judge framework by incorporating capabilities to provide intermediate feedback, thereby enhancing evaluation precision.

**Quality:**
The quality of the research is evident in the thorough experimental setup and robust analysis. The authors have benchmarked three leading open-source code-generating systems—MetaGPT, GPT-Pilot, and OpenDevin—using the proposed framework. They conducted experiments across various settings, including black-box and gray-box scenarios, as well as independent and task-dependent evaluations. These comprehensive experiments validate the effectiveness of the Agent-as-a-Judge framework and underscore its superiority over traditional methods. The detailed statistical analysis and alignment with human evaluators further reinforce the reliability of the proposed method.

**Clarity:**
The paper is well-structured and clearly presents its ideas and contributions. The introduction effectively outlines the motivation and objectives, while subsequent sections provide a detailed explanation of the DevAI benchmark and the Agent-as-a-Judge framework. The methodologies and experiments are described with clarity, making it easy for readers to follow and understand the research. Additionally, the inclusion of figures and tables enhances the presentation by providing visual insights into the experimental results.

**Significance:**
The significance of the paper lies in its potential to transform the evaluation of agentic systems. By introducing a novel and possibly more sound method—Agent-as-a-Judge—the paper addresses the limitations of traditional evaluation techniques that focus solely on final outcomes or require excessive manual labor. The DevAI benchmark, coupled with the Agent-as-a-Judge framework, provides a comprehensive and efficient approach for assessing the performance of agentic systems. This work lays the groundwork for more precise, intermediate evaluations, which can significantly accelerate progress in the development and deployment of sophisticated agentic systems.
summary: The paper "Agent-as-a-Judge: Evaluating Agents with Agents" addresses the inadequacy of traditional evaluation techniques for assessing agentic systems, which require more sophisticated, step-by-step feedback mechanisms. The authors propose the Agent-as-a-Judge framework, a novel approach that employs agentic systems to evaluate other agentic systems, integrating capabilities to provide intermediate feedback throughout the task-solving process for more precise evaluations.

The paper introduces a new benchmark, DevAI, to demonstrate and validate the proposed framework. DevAI comprises 55 realistic AI code generation tasks, complete with detailed manual annotations, making it ideal for agentic evaluators. The authors benchmark three leading open-source code-generating systems—MetaGPT, GPT-Pilot, and OpenDevin—using the proposed framework. Their experiments reveal that the Agent-as-a-Judge framework outperforms the LLM-as-a-Judge method and matches the reliability of a human evaluation baseline.

The primary contributions of this work include:
1. The release of the DevAI benchmark, which consists of 55 comprehensive AI development tasks with accompanying tags, hierarchical requirements, and preferences, designed to enhance the evaluation of agentic systems.
2. The introduction of the Agent-as-a-Judge framework, an innovative method for evaluating agentic systems using other agentic systems, providing rich, intermediate feedback for more accurate evaluations that align closely with human evaluators while significantly reducing time and cost.
weaknesses: **Human Evaluation Methodology :**
The use of only three human experts as judges raises concerns about the robustness and reliability of the evaluation process. Given that these experts are authors of the paper, there is a potential for bias, and the large disagreements observed among them further question the dependability of the evaluation. To improve this aspect, the study could benefit from recruiting a larger and more diverse pool of external evaluators. Even applying a subset of tasks to a larger group of judges would offer more statistically reliable results and help validate the labels in the DevAI benchmark. This would ensure that the benchmark has a more concrete "ground truth", thus making the subsequent results more robust and credible.

**Inconsistent Statements on Human Evaluation Reliability :**
There are conflicting statements regarding the reliability of human evaluations. In Section 4 (Line 366), it is mentioned that "Human evaluation, while somewhat reliable," which seems inconsistent with the prior conclusions in Section 3.4 (Line 354) "Human judgment errors are inevitable" and the observations in Lines 319-327 about the large disagreements among human evaluators. The paper would benefit from clearly articulating the relative reliability of various evaluation methods. If I understand correctly, the paper is expressing the relationship of reliability as LLM-as-a-Judge < Single-Human-as-a-Judge < Agent-as-a-Judge < Ensemble of Human Judges. The paper would benefit from stating this clearly.

**Clarity in Figures :**
Figure 2(1) appears unprofessional and is highly unreadable due to the mixed vertical and horizontal text as well as the small font size. Enhancing the readability and professional presentation of this figure is essential.  the authors might consider using a different type of diagram to display the word frequency more effectively.

**Citation Issues :**
There are several inconsistencies and inaccuracies in the citations. For instance, the citation of SWE-Bench in the introduction leads to SWE-Agent, which is incorrect. Moreover, some papers that have been accepted to conferences are cited in their arXiv versions (e.g., SWE-Bench, DSPy, HumanEval, AgentBench). Ensuring that all citations are accurate and up-to-date will enhance the paper's credibility. Each referenced work should be properly cited in its correct and most formal publication form when available.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Here are some questions:

1. Could you clarify the technical contributions of this agent-as-a-judge? How this method differ from LLM-as-a-judge? 

2. Could you clarify about the succeffiency of the tasks in DevAI? More tasks and evaluations should be evaluated. 

3. Does this method generalize to novel tasks, which is out of the considered benchmark tasks?
soundness: 3
strengths: 1. Using agent to evaluate the agent's performance is very interesting and important to advance the development of the foundation agent. This paper addresses a critical and growing challenge in AI: how to effectively evaluate increasingly complex AI agents. 

2. Thorough experimental design with multiple levels of evaluation, including comprehensive ablation studies to understand component contributions, careful analysis of human evaluation biases and errors and clear comparison between human evaluators, LLM-as-a-Judge, and Agent-as-a-Judge

3. The proposed DevAI is a good benchmark, which has carefully designed hierarchical requirements, includes explicit dependencies between requirements, incorporates both mandatory requirements and optional preferences, and focuses on realistic AI development tasks
summary: This paper introduces "Agent-as-a-Judge," a framework that uses AI agents to evaluate other AI agents' code generation capabilities, extending the existing LLM-as-a-Judge paradigm. The authors also present DevAI, a new benchmark dataset containing 55 AI development tasks with hierarchical requirements, designed to test code-generating AI systems. They evaluate three popular open-source code generation agents (MetaGPT, GPT-Pilot, and OpenDevin) using both human evaluators and their Agent-as-a-Judge system. Their results show that Agent-as-a-Judge achieves comparable performance to human evaluators (90% alignment with human consensus) while being significantly more cost-effective, reducing evaluation time by 97.72% and costs by 97.64% compared to human evaluation. The paper demonstrates that automated evaluation of AI agents is feasible and can potentially scale up the development of more sophisticated AI systems.
weaknesses: My main concerns of the paper are:

1. The limited technical contributions. This agent-as-a-judge is a natural extension of LLM-as-a-judge, and this is more like an engineering project. 

2. The proposed DevAI is small scale with only 55 tasks, which is insufficient for a comprehensive benchmark. Though low cost is a good point, this would also over-simplify the tasks with limited task complexity and diversity.

Meta-review: The paper "Agent-as-a-Judge: Evaluating Agents with Agents" introduces a novel framework, Agent-as-a-Judge (AAAJ), for evaluating agentic systems (AI systems capable of autonomous decision-making and action) by leveraging other agentic systems as evaluators. This approach extends the existing LLM-as-a-Judge paradigm by incorporating intermediate feedback mechanisms, enabling more granular and accurate evaluations. To validate this framework, the authors propose a new benchmark, DevAI, consisting of 55 realistic AI code generation tasks with detailed annotations and hierarchical requirements. The paper benchmarks three leading code-generating agentic systems (MetaGPT, GPT-Pilot, and OpenHands) using AAAJ, demonstrating its alignment with human evaluators and its superior performance compared to LLM-as-a-Judge. The authors also provide supplementary materials, including failure case analyses and experiments with different backbone models, to support their claims.


Weaknesses
- Limited Scope of Benchmark : While DevAI is a valuable benchmark, its scale (55 tasks) is relatively small compared to some existing benchmarks (e.g., MLE-Bench with 75 tasks). The authors argue that increasing the number of tasks would disproportionately increase costs, but this may limit the benchmark's generalizability to a broader range of AI development scenarios.
- Task-Specific Design : The AAAJ framework is currently tailored to code generation tasks, raising questions about its applicability to other domains. Although the authors suggest its potential for generalization, this remains a theoretical claim without concrete demonstrations across diverse tasks.
- Human Evaluation Limitations : The reliance on a small pool of human evaluators (the authors themselves) raises concerns about bias and the robustness of the ground truth labels. While the authors conducted a supplementary study with a larger group of students, the scope of this additional evaluation was limited (7 tasks), leaving room for skepticism about the reliability of the human baseline.

Overall, the paper presents a compelling contribution to the field of agentic system evaluation, with the Agent-as-a-Judge framework and the DevAI benchmark offering practical and innovative solutions. While the paper has addressed many reviewer concerns through additional experiments and clarifications, some limitations persist, particularly regarding the scope of the benchmark and the framework's generalizability. The paper would benefit from further refinement in terms of presentation and broader demonstration of the framework's applicability.

Decision: Reject