Title: AGENTBENCH: EVALUATING LLMS AS AGENTS
Benchmark: AgentBench
Summary: AgentBench presents a holistic benchmark for evaluating LLMs as agents. It is structured across three domains (code, game, and web) and aims to evaluate a wide range of abilities. 
Contribution: While it is a well-respected benchmark, it's also vague in what it actually measures.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: "core" agentic capabilities: following instructions, coding, knowledge acquisition, logical reasoning, and commonsense grounding.
Phenomenon defined by authors: No
Phenomenon definition: They broadly define agent capabilities as able to do reasoning and decision making but don't define those further.
Definition scope: Comprehensive
Definition integrity: Authors' description is unclear
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: The overall tasks are either coding, text-based games/puzzles or web browsing. Each is predominantly evaluated based on successfully solving a problem.
Task item definition: Each task has an objective/prompt, a text-based environment, and a success state. Sometimes the success state involves a "gold action sequence". 
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: The sub-benchmarks are quite heterogeneous in their realism. The coding tasks are relatively more realistic and the game tasks are quite synthetic.
Dataset size: 1014
Dataset size notes: 269
Dataset metadata available: No
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Response format detail: The interaction is extended but the output is often short. 
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Weighted Mean
Metric statistics: Aggregated scores (no additional stats)
Metric subscores: Yes
Metric subscores detail: Specific subtasks within the broader categories (e.g., "Operating System" within coding)
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Too vaguely defined phenomenon
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Author validity detail: Curiously, the authors performa "validity analysis" of the models responses but not of the actual tasks. 
Taxonomy: Agents / 
New taxonomy: General Capability / Core Agentic Capabilities