Title: Do language models have coherent mental models of everyday things?
Benchmark: ParRoT
Summary: Benchmark to monitor the "mental models" of LLMs when queried about everyday physical objects. They crowdsource a dataset of 100 everyday items (e.g a flashlight) with the relationships between various parts (batteries -> are inside -> flashlight) annotated. LLMs are then asked to predict the relationship between parts. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Do language models have coherent mental models of everyday things?
Phenomenon defined by authors: Yes
Phenomenon definition: "mental models of the world, namely internal, conceptual representations of the environment which we base our decisions and actions on"

"described mental models as a ‘small-scale model’ of external reality and of its own possible actions within someone’s head."
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Task definition: Here we define our task: “Construct a parts mental model for everyday things” with the following input/output specifications:  • Input: Everyday thing, Parts list, Relation vocabulary (14 relations).  • Output: List of tuples (x, r, y) where relation r holds between parts x and y.

However, LLMs are asked an easier task: We probe them using True/False questions of type: “Judge whether this statement is true or false: In an <everyday thing>, <part1 relation part2>.” For each query q, we record an answer a ∈ {T rue, False}
Task item definition: Judge whether this statement is true or false:  In a tree, trunk is above the roots.
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 11,700
Dataset size detail: 100 everyday things, 2.2K parts and 11.7K relationships
Dataset metadata available: Yes
Dataset metadata detail: The dataset is quite rich, in that it is actually 100 fully annotated mental models of everyday things. This allows for disecting the data in many ways - by relation type, by object type, etc.
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: The dataset is quite rich, in that it is actually 100 fully annotated mental models of everyday things. This allows for disecting the data in many ways - by relation type, by object type, etc.
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Grounding / 
New taxonomy: General Capability / Grounding