Agent-as-a-Judge: Evaluate Agents with Agents
Authors: name: Mingchen Zhuge, name: Changsheng Zhao, name: Dylan Ashley, name: Wenyi Wang, name: Dmitrii Khizbullin, name: Yunyang Xiong, name: Zechun Liu, name: Ernie Chang, name: Raghuraman Krishnamoorthi, name: Yuandong Tian, name: Yangyang Shi, name: Vikas Chandra, name: Jürgen Schmidhuber

Contemporary evaluation techniques are inadequate for agentic systems. These
approaches either focus exclusively on final outcomes -- ignoring the
step-by-step nature of agentic systems, or require excessive manual labour. To
address this, we introduce the Agent-as-a-Judge framework, wherein agentic
systems are used to evaluate agentic systems. This is an organic extension of
the LLM-as-a-Judge framework, incorporating agentic features that enable
intermediate feedback for the entire task-solving process. We apply the
Agent-as-a-Judge to the task of code generation. To overcome issues with
existing benchmarks and provide a proof-of-concept testbed for
Agent-as-a-Judge, we present DevAI, a new benchmark of 55 realistic automated
AI development tasks. It includes rich manual annotations, like a total of 365
hierarchical user requirements. We benchmark three of the popular agentic
systems using Agent-as-a-Judge and find it dramatically outperforms
LLM-as-a-Judge and is as reliable as our human evaluation baseline. Altogether,
we believe that Agent-as-a-Judge marks a concrete step forward for modern
agentic systems -- by providing rich and reliable reward signals necessary for
dynamic and scalable self-improvement.