Title: HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments
Benchmark: HAZARD Challenge
Summary: The HAZARD Challenge introduces a benchmark for evaluating embodied agents' decision-making in dynamically changing environments such as fires, floods, and wind. It supports reinforcement learning, rule-based, search-based, and LLM-driven agents, using realistic simulations built on the ThreeDWorld platform to assess object rescue performance in disaster scenarios.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Reasoning and embodied decision-making in dynamically changing environments
Phenomenon defined by authors: Yes
Phenomenon definition: The benchmark is specifically designed to assess the decision-making abilities of embodied agents in dynamic situations, characterised by unexpected events and the need to rapidly take action accordingly.  It is also described as embodied decision-making in dynamic environments by designing and implementing new capabilities for physical simulation and visual effects on top of the ThreeDWorld.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The objective for an embodied agent is to rescue a predetermined set of target objects and bring them to a given safe location such as a bag held by the agent or a shopping cart.
Task item definition: Single task item involves an embodied agent placed in a simulated environment (indoor or outdoor) facing a dynamic disaster scenario (fire, flood, or wind) with a set of target objects to rescue. The agent receives observations (RGB-D, temperature/water level, segmentation mask) and must choose actions to accomplish the rescue mission within a time limit.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: The task is set in a simulated environment with procedurally generated scenes of unexpected disasters. While it was inspired by the real-world dynamic environments and rescue scenarios, it is constructed for benchmarking purposes in that simulated context.
Dataset size: 75
Dataset size detail: The paper states 100 unique scenes per task (and there are 3 tasks) with a 3:1 train-set split, meaning approximately 75 scenes for training and 25 for testing per task (225 to train and 75 for test in total).
Dataset size notes: training: 225
Dataset metadata available: Yes
Dataset metadata detail: Object value, temperature, water level, distance, object attributes (e.g. is it waterproof, what is the ignition point)
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall), rescued value rate, averaged rescue step, averaged damaged rate
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: Scores are reported per scenario (fire, flood, wind), and with/without perception variants.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: No, link is broken
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors assess validity by:
Comparing LLM vs rule-based vs RL vs MCTS pipelines on the same benchmark
Introducing perception-noisy variants to test robustness
Analysing failure cases (such as LLM failing to account for wind dynamics or object occlusion)
Measuring differences across different LLMs

in order to show what the benchmark is measuring and the difficulty, particularly regarding perception and decision-making under uncertainty.
Taxonomy: Agents / 
New taxonomy: General Capability / Core Agentic Capabilities
Procedural notes: The link in the paper was broken, but above I provide a new working link.