Title: Can Language Models Understand Physical Concepts?
Benchmark: VEC
Summary: This paper introduces the VEC benchmark to evaluate language models' understanding of visual and embodied physical concepts. The authors also propose a knowledge distillation method that transfers embodied understanding from vision-language models to LMs, achieving significant gains without massive model scaling.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: physical concept understanding
Phenomenon defined by authors: Yes
Phenomenon definition: whether current LMs can understand basic physical concept
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Take a question and provide a short-form answer
Task item definition: Question: Is the water colder than the frying oil?
Answer: [MASK].
Task source: Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 4090
Dataset metadata available: Yes
Dataset metadata detail: Concept Category
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric aggregation: Simple Mean
Metric statistics: mean and variance
Metric subscores: Yes
Metric subscores detail: Concept Category
Metric access: Model access required (e.g. logits)
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Grounding / 
New taxonomy: General Capability / Grounding