Title: PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change
Benchmark: PlanBench
Summary: PlanBench introduces a suite of tasks relevant to planning using similar formats to the International Planning Competition. The tasks are taken from either Blocksworld or logistics and also obfuscated to avoid reliance on common-sense knowledge.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Planning
Phenomenon defined by authors: Yes
Phenomenon definition: planning involves coming up with a course of actions (policy) which when executed would take an agent from a certain initial state to a desired world state
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The main task (planning) is given a description of a state (e.g., block configuration), rules, and a goal state, come up with a plan that transforms from state the goal state. The sub-tasks are variations of components. 
Task item definition: A specified state, actions, and goal state + a query for what the LLM should do (compe up with a plan, predict plan execution) etc.
Task definition detail: There are in total 8 different tasks with slightly different goals (e.g., direct planning, replanning, execution prediction)
Task source: Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: The task is based on real competition but which has a level of gaminess
Dataset size: 1910
Dataset metadata available: Yes
Dataset metadata detail: domain
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test
Response format: Free response (e.g. summary paragarph), Structured response (e.g. valid JSON, API call alone)
Response format detail: The plan is a fairly structured set of actions, but not quite as structured as e.g., an API
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Domain, Obfuscated (Bool)
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Planning
New taxonomy: General Capability / Planning