Title: ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities
Benchmark: ActPlan-1K
Summary: This paper introduces ActPlan-1K for evaluating VLMs on procedural and counterfactual reasoning tasks. By combining natural language descriptions with simulated environment images, the benchmark assesses the ability of VLMs to generate coherent action plans.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: multimodal reasoning, procedural planning
Phenomenon defined by authors: Yes
Phenomenon definition: Whether VLMs can generate plausible action plans for multi-modal embodied AI task
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Given household environment E, there are manipulable objects set O. For each household activity T , an VLM agent A takes in the task description T and environment images {I1, I2, ...} as input, generates procedural plan P∗ that can accomplish the task. The household environments have multiple interior spaces therefore there are multiple images to ensure that necessary spatial information is provided.
Task item definition: NL Task description: "The task goal is to assemble gift baskets. .... <description of the environment>"
Gold Plan
Task source: Expert-crafted task examples (e.g. hand-written examples)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1000
Dataset metadata available: Yes
Dataset metadata detail: normal or counterfactual
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Planning
New taxonomy: General Capability / Planning