Title: Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Models
Benchmark: BLOCKWORLD-REPAIRS
Summary: Paper proposes a dataset and a benchmark measuring LLMs ability to respond/correct/repair ambiguous questions/requests and how they recover from them. Benchmark is built based on a simulator that simulates boxes on a table at various locations where the VLM needs to respond to questions about box positions and where they should be moved to in which such questions might be vague.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: natural language understanding
Phenomenon defined by authors: Yes
Phenomenon definition: In dialogue, the addressee may initially misunderstand the speaker and respond erroneously,
often prompting the speaker to correct the misunderstanding in the next turn with a Third
Position Repair (TPR). The ability to process
and respond appropriately to such repair sequences is thus crucial in conversational AI
systems
Definition scope: Subset
Definition integrity: Authors' description is unclear
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Task definition: Ability of the VLMs to identify the object in an image even when the quesiton is vague. In addition to that, the ability of the VLM to find the target location for which this object needs to be moved to.
Task item definition: An image and a dialogue triplets that are intrinsically connected and can only be comprehended as a whole: the initial instruction, the incorrect candidate prediction, and the repair
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt), Original benchmark modified through an agent automatically and through crowdsourcing it was filtered for quality.
Task source detail: dasdas
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: I believe the task is incomplete as "repairs" as they measure it is only done in one very specific environment and bias for the dataset (allocating boxes in an image)
Dataset size: Check table7; the total test set is 849 records
Dataset size notes: Check table7; the total test set is 1210 records
Dataset metadata available: Yes
Dataset metadata detail: human difficulty
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Short free response (e.g. single word or number)
Metric definition: IOU
Metric aggregation: Simple Mean
Metric statistics: mean, std
Metric subscores: Yes
Metric subscores detail: based on human difficulty
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: link is provided but github reads "Dataset and code coming soon! Work in progress..."
Task face validity: Only partly
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding
Other notes: There is not enough discussions on the realism of the task in capturing the phenomenon.