Survey on Evaluation of LLM-based Agents
Authors: name: Asaf Yehudai; name: Lilach Eden; name: Alan Li; name: Guy Uziel; name: Yilun Zhao; name: Roy Bar-Haim; name: Arman Cohan; name: Michal Shmueli-Scheuer
Published: 2025-03-20 17:59:23

Abstract: The emergence of LLM-based agents represents a paradigm shift in AI, enabling
autonomous systems to plan, reason, use tools, and maintain memory while
interacting with dynamic environments. This paper provides the first
comprehensive survey of evaluation methodologies for these increasingly capable
agents. We systematically analyze evaluation benchmarks and frameworks across
four critical dimensions: (1) fundamental agent capabilities, including
planning, tool use, self-reflection, and memory; (2) application-specific
benchmarks for web, software engineering, scientific, and conversational
agents; (3) benchmarks for generalist agents; and (4) frameworks for evaluating
agents. Our analysis reveals emerging trends, including a shift toward more
realistic, challenging evaluations with continuously updated benchmarks. We
also identify critical gaps that future research must address-particularly in
assessing cost-efficiency, safety, and robustness, and in developing
fine-grained, and scalable evaluation methods. This survey maps the rapidly
evolving landscape of agent evaluation, reveals the emerging trends in the
field, identifies current limitations, and proposes directions for future
research.