Title: AgentHard: Hardening LLM-Agent Evaluation with a Taxonomy of Artifacts and Automated Cleaning
Primary area: datasets and benchmarks
Keywords: Agent benchmark, Benchmark filtering

Abstract: Reliable evaluation of LLM‑based agents is often confounded by artifacts that misrepresent true agent capabilities by conflating model errors with benchmark flaws. To address this, we present a component-wise taxonomy of common benchmark pitfalls spanning the user, environment, evaluation, and ground truth elements of agent tasks. This analysis exposes pervasive issues such as incorrect ground-truth action sequences, ambiguous tool APIs, user simulation faults, and brittle evaluation metrics. Guided by these insights, we develop AgentBenchCleaner, an automated pipeline that filters out flawed tasks in three stages: first, rule-based detectors catch deterministic errors; second, an LLM-as-judge identifies nuanced issues; and third, a quality-filtering step enhances evaluation rigor. Applying this pipeline yields a curated benchmark suite, AgentHard-Bench, with standardized evaluation protocols and explicit quality criteria. Across diverse LLM agents, evaluations on AgentHard-Bench deliver more stable model rankings, clearer performance separations, and improved benchmark diversity relative to the original benchmarks. We will release AgentHard-Bench, along with the taxonomy and pipeline upon acceptance, to support robust, reproducible agent evaluation.