Title: A Benchmark for Reasoning with Spatial Prepositions
Summary: The paper introduces a new benchmark designed to evaluate the inferential capabilities of statements involving spatial prepositions. Featuring original datasets in both English and Romanian, the benchmark explores the boundaries of large language models’ reasoning about spatial relations.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Spatial reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: The benchmark proposes the challenge of using spatial prepositions to refer to abstract concepts in addition to physical relationships.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Take two premises with prepositions and determine if the conclusion holds. 
Task item definition: Two premies, one conclusion, whether the conclusion holds. 
Task source: Expert-crafted task examples (e.g. hand-written examples)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 800
Dataset metadata available: Yes
Dataset metadata detail: The preposition used in a premise / conclusion in an example. 
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric subscores: Yes
Metric subscores detail: items containing different prepositions
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Reasoning / Spatial
New taxonomy: General Capability / Spatial Reasoning