Title: SMARTPLAY : A BENCHMARK FOR LLMS AS INTELLIGENT AGENTS
Benchmark: SMARTPLAY
Summary: This paper introduces SmartPlay, a benchmark for assessing LLMs as intelliget agents using 6 different games including Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game features a unique setting, providing up to 20 evaluation settings and infinite environment variations. Each game in SmartPlay uniquely challenges a subset of 9 important capabilities of an intelligent LLM agent, including reasoning with object dependencies, planning ahead, spatial reasoning, learning from history, and understanding randomness.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: LLMs as intelligent agents
Phenomenon defined by authors: No
Phenomenon definition: While the authors do not explicitly provide a definition for intelligent agents, they provide some key properties of agents from which the target phenomenon aims to be captured—i.e., LLM agents as systems capable of long-horizon planning, probabilistic reasoning, spatial reasoning to understand the 3D world, and learning from interactions or mistakes. This is further decomposed into 9 measurable abilities; long text understanding, reasoning, instruction following, planning, generalization, understanding the odds, learning from interactions, error/mistake handling and spatial reasoning.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: An LLM is provided with environment-specific inputs; either textual descriptions or visual descriptions (via natural language)—along with manuals containing background knowledge, rules, and examples. The LLM must then interact with the environment by selecting actions from a predefined action space to achieve task objectives across multiple trials or rollouts.
Task item definition: A task item is a description of a game, the rules, actions, environment and expected behaviour.
Task definition detail: A task item consists of a description of the game, its rules, available actions, environment state, and the expected agent behavior.
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 6 games with 20 different evaluation settings
Dataset metadata available: Yes
Dataset metadata detail: Each task item is annotated with the abilities required i.e.  long text understanding, reasoning, instruction following, planning, generalization, understanding the odds, learning from interactions, error/mistake handling and spatial reasoning.
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Extended interaction (e.g. conversation, calling an API and processing the response)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Scores are normalised relative to human performance.
Metric subscores: Yes
Metric subscores detail: Across each game in the task
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Somewhat
Author validity detail: The authors use real-world-inspired games to evaluate LLM capabilities as intelligent agents, but the extent to which success in these environments generalizes to real-world agentic behavior remains an open question.
Taxonomy: Agents / 
New taxonomy: General Capability / Core Agentic Capabilities