Title: R2H: Building Multimodal Navigation Helpers that Respond to Help Requests
Benchmark: R2H
Summary: R2H introduces benchmark that evaluates navigation‑helper agents which "see and respond". It defines two tasks: 
1. Respond to Dialog History (RDH) 
2. Respond‑during‑Interaction (RdI) 
They are built on three vision‑and‑dialog navigation corpora, and supply automatic metrics + human studies to judge how well a helper’s natural‑language responses help a (fixed) performer agent complete navigation. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Multimodal conversational navigation assistance, reasoning, natural language understanding, utilising existing dialog-based context
Phenomenon defined by authors: Yes
Phenomenon definition: The capability of multimodal conversational agents to assist task performers in completing navigation tasks by generating natural language responses to help requests, grounded in visual observations and task-specific environmental information.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: RDH: given dialog history + oracle image sequence, generate one response to the performer’s latest query. RdI: in a live loop, generate successive responses whenever the (pre‑trained) performer asks for help.
Task item definition: In RDH: dialog history, help request, image sequence → one response
In RdI: help request, current visual context → one response per turn in ongoing dialog
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: During training, the authors use GPT-3 to rewrite human responses into structured instructions via their "Parse by Step" method, not directly generating instructions from scratch.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: The task ecology is classified as "Partial real task" because the benchmark simulates a real-world human-helper navigation scenario, but conducts the evaluations inside a simulated environments using pre-trained agents or human participants interacting with a simulator.
Dataset size notes: Training set sizes: CVDN = 2 050 traj., DialFRED = 120 958, AVDN = 5 372 (all splits)., for validation numbers were not disclosed.
Dataset metadata available: Yes
Dataset metadata detail: environment type (indoor/outdoor, synthetic/photo-realistic), number of navigation trajectories, number of queries per trajectory, average human response length, average human query length, average number of images in input sequence
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragraph, executable code)
Response format detail: I think partially "targeted items" could fit in the datasample_sampling_method. However, while the authors apply scripted transformations, they don’t hand-pick examples based on task difficulty, content types, or desired distributions - which would define a "targeted" sample. They do filter by suitability, but that's already captured under “specific criteria.”
Metric definition: Exact Match (accuracy, F1, precision, recall), The primary metrics (GP, SR, SPL, PWSR) measure the task performer's navigation success when guided by the helper. Helper effectiveness is inferred from these outcomes, with SPL and PWSR combining success and path efficiency.
Metric aggregation: Simple Mean
Metric statistics: Simple means with error bars
Metric subscores: Yes
Metric subscores detail: Scores are provided for different datasets (CVDN, DialFRED, AVDN) and different splits (Seen Validation, Unseen Validation).  For RdI, results are aggregated by the number of conversation turns.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Ablations show language‑similarity metrics (BLEU/ROUGE) don’t track task success. Human study rates faithfulness/naturalness. The authors acknowledge that the complexity of the real world surpasses that of a simulated environment.
Taxonomy: Agents / 
New taxonomy: General Capability / Core Agentic Capabilities