Title: AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
Benchmark: AGENTBOARD
Summary: This paper introduces AGENTBOARD, a benchmark designed to evaluate LLM agents capabilities in partially observable environments, multi-round interactions and diverse tasks through a unified evaluation framework and fine-grained metric analysis. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Agent Abilities for Real-World Problem Solving
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to perform step-by-step planning in diverse tasks and partially observable environments via multi-turn interactions.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: A task in AGENTBOARD presents an agent with a real-world scenario—such as an embodied, game, or tool-use environment—where it must perform actions, receive feedback, and plan over multiple rounds in a partially observable setting.
Task item definition: A task item includes an environment definition, a goal to achieve, step-wise observations, and defined action space
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 1013
Dataset metadata available: Yes
Dataset metadata detail: Difficulty
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: Yes
Metric subscores detail: Across each subtask dataset, difficulty and subskill
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: No
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Planning
New taxonomy: General Capability / Planning