Title: EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models
Benchmark: EmbSpatial-Bench
Summary: This paper introduces the EmbSpatial-Bench benchmark to evaluate the spatial understanding capabilities of LVLMs in embodied environments. The authors also propose EmbSpatial-SFT, an instruction-tuning dataset aimed at enhancing LVLMs' spatial reasoning from an egocentric perspective
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: spatial understanding
Phenomenon defined by authors: Yes
Phenomenon definition: understand spatial relationships between object in embodied scenarios
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Take a spatial image and multiple-choice question, provide the answer
Task item definition: <spatial image> Question: How are television and shelf positioned in relation to each other in the image? A: ..., B: ..., C: ..., D: ....
Task source: Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 3640
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: For likelihood strategy, which uses the option with the highest probability generated by
the model, the model output logits is needed
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Model access required (e.g. logits)
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: VQA / 
New taxonomy: General Capability / Grounding