Title: GameTraversalBenchmark: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Maps
Benchmark: GameTraversalBenchmark (GTB)
Summary: The paper investigates the planning capabilities of LLMs by proposing GameTraversalBenchmark (GTB), a benchmark consisting of diverse 2D grid-based game maps. The paper also provide metrics to give insights towards planning abilities in LLMs.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Planning abilities of LLMs
Phenomenon defined by authors: No
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Not defined
Task definition: The task is a game based on 2D maps. They consider a generated map as one data point for the benchmark. The map’s generated objective coordinates are the points where the LLM agent needs to traverse to attain the most rewards. 
Task item definition: Each item is a 2D grid-based map if alphanumeric characters.
Task source: LLM-generated task examples (e.g. Filtered from responses to a prompt)
Dataset size: 150
Dataset metadata available: No
Sampling method: Unknown
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), The paper defines a reward score
Metric aggregation: Simple Mean
Metric statistics: simple mean and STD
Metric subscores: No
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Planning
New taxonomy: General Capability / Planning