Agent-as-a-Judge: Evaluate Agents with Agents
Authors: name: Mingchen Zhuge; name: Changsheng Zhao; name: Dylan Ashley; name: Wenyi Wang; name: Dmitrii Khizbullin; name: Yunyang Xiong; name: Zechun Liu; name: Ernie Chang; name: Raghuraman Krishnamoorthi; name: Yuandong Tian; name: Yangyang Shi; name: Vikas Chandra; name: Jürgen Schmidhuber
Published: 2024-10-14 17:57:02

Abstract: Contemporary evaluation techniques are inadequate for agentic systems. These
approaches either focus exclusively on final outcomes -- ignoring the
step-by-step nature of agentic systems, or require excessive manual labour. To
address this, we introduce the Agent-as-a-Judge framework, wherein agentic
systems are used to evaluate agentic systems. This is an organic extension of
the LLM-as-a-Judge framework, incorporating agentic features that enable
intermediate feedback for the entire task-solving process. We apply the
Agent-as-a-Judge to the task of code generation. To overcome issues with
existing benchmarks and provide a proof-of-concept testbed for
Agent-as-a-Judge, we present DevAI, a new benchmark of 55 realistic automated
AI development tasks. It includes rich manual annotations, like a total of 365
hierarchical user requirements. We benchmark three of the popular agentic
systems using Agent-as-a-Judge and find it dramatically outperforms
LLM-as-a-Judge and is as reliable as our human evaluation baseline. Altogether,
we believe that Agent-as-a-Judge marks a concrete step forward for modern
agentic systems -- by providing rich and reliable reward signals necessary for
dynamic and scalable self-improvement.