HASARD: A Benchmark for Vision-Based Safe Reinforcement Learning in Embodied Agents

Authors: Tristan Tomilin, Meng Fang, Mykola Pechenizkiy
Primary area: datasets and benchmarks
Keywords: reinforcement learning, AI safety, safe RL, constrained RL, benchmark, vizdoom, 3D, difficulty levels

Abstract:
Advancing safe autonomous systems through reinforcement learning (RL) requires robust benchmarks to evaluate performance, analyze methods, and assess agent competencies. Humans primarily rely on embodied visual perception to safely navigate and interact with their surroundings, making it a valuable capability for RL agents. However, existing vision-based 3D benchmarks only consider simple navigation tasks. To address this shortcoming, we introduce **HASARD**, a suite of diverse and complex tasks to **HA**rness **SA**fe **R**L with **D**oom, requiring strategic decision-making, comprehending spatial relationships, and predicting the short-term future. HASARD features three difficulty levels and two action spaces. An empirical evaluation of popular baseline methods demonstrates the benchmark's complexity, unique challenges, and reward-cost trade-offs. Visualizing agent navigation during training with top-down heatmaps provides insight into a method's learning process. Incrementally training across difficulty levels offers an implicit learning curriculum. HASARD is the first safe RL benchmark to exclusively target egocentric vision-based learning, offering a cost-effective and insightful way to explore the potential and boundaries of current and future safe RL methods. The environments and baseline implementations are open-sourced.

Decision: accept (poster)
Reviewer scores: 8, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1729488349936

thread_type:
official_review

<thread_object>
creator: reviewer_ntcd

<content>
summary:
This paper introduces the HASARD, a benchmark designed for egocentric pixel-based safe RL in diverse and stochastic 3D environments. Unlike existing benchmarks, HASARD emphasizes spatial comprehension, short-term planning, and active prediction for high rewards while ensuring safety. It offers three difficulty levels, supporting both soft and hard safety constraints. The benchmark includes heatmaps for visual analysis, aiding in strategy development. By targeting vision-based embodied safe RL, HASARD addresses the need for benchmarks mirroring real-world complexities. The paper's contributions include the design of six novel ViZDoom environments with safety constraints, integration with Sample-Factory for rapid simulation and training. Evaluation of baseline methods within HASARD highlights challenges in balancing performance and safety under constraints.

strengths:
The paper demonstrates several notable strengths across the dimensions of originality, quality, clarity, and significance: 

  

1. **Originality**: It introduces HASARD, a benchmark specifically designed for vision-based embodied safe reinforcement learning (RL) in complex 3D environments. 

  

2. **Quality**: Comprehensive design of 6 diverse environments with 3 difficulty levels each, offering a range of challenges. 

  

3. **Clarity**: The paper is structured in a logical and coherent manner, facilitating the understanding of complex concepts. 

  

4. **Significance**: The paper Addresses an important need in safe RL research for more realistic and challenging benchmarks. It enables systematic evaluation and comparison of safe RL algorithms in vision-based 3D settings.

weaknesses:
While the paper makes valuable contributions, several areas could be improved: 

  

1. The paper refers to ViZDoom as a 3D environment, but its pixelated, less detailed graphics compared to modern 3D games challenge this characterization. 

  

2. **Narrow Range of Baselines**: Evaluations focus primarily on PPO-based algorithms. Incorporating approaches like model-based safe RL or constrained policy optimization (e.g., https://arxiv.org/abs/2210.07573) would enhance the assessment. 

  

3. **Limited Visual Input Analysis**: Though vision-based learning is emphasized, the paper lacks analysis of how visual complexity influences performance. Exploring different visual conditions (lighting, distractors) and comparing raw pixels with simplified representations would highlight the unique challenges of vision-based safe RL, especially since the visual inputs in the environment appear less realistic. 

  

4. **Action Space Limitation**: Only discrete action spaces are supported. It is unclear how continuous safe RL algorithms would be benchmarked. 

  

5. **Real-World Relevance**: The connection between the benchmark tasks and real-world safe RL challenges needs clearer articulation. Providing examples of practical applications would strengthen motivation. 

   

Addressing these points could strengthen the paper and increase the impact and utility of the HASARD benchmark for the safe RL research community.

questions:
1. Is ViZDoom truly a 3D environment, considering its graphics appear pixelated and less detailed compared to modern 3D games? 

  

2. Why are the baseline algorithms limited to PPO-based approaches? Could the paper include more diverse methods, such as model-based safe RL or constrained policy optimization (e.g., https://arxiv.org/abs/2210.07573)? 

  

3. How can continuous safe RL algorithms be benchmarked when the paper only supports discrete action spaces?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667866138

thread_type:
official_review

<thread_object>
creator: reviewer_rd3k

<content>
summary:
The paper proposes a new egocentric vision-based 3D simulated environment for benchmarking safe reinforcement learning. The benchmark is more realistic and challenging compared to common prior safe RL benchmark environments. In addition, the paper has evaluations for some safe RL algorithms on the proposed benchmark demonstrating its feasibility of use and the potential for building better approaches to perform more favorably on it.

strengths:
- The paper is well motivated and targets an important problem - that of building realistic and reliable RL benchmarks, and more specifically benchmarks for safe RL. This involves addressing challenges with the simple natural of prior benchmarks - both visually and in terms of higher dimensional action space and increased temporal horizons. 

- The proposed benchmark HASARD is built on top of an existing game engine VizDoom and is able to inherit all of its properties for re-use. The multiple levels in HASARD can be potentially helpful in evaluating different notions of safety in proposed safe RL algorithms. 

- The paper has detailed evaluations of several safe RL algorithms on HASARD indicating that the framework is feasible for training constrained RL policies. The evaluations reveal that simple algorithms based on PPO and constrained PPO can achieve non-trivial performance in the benchmark and also reasonable constraint satisfaction. It is good to see that these simple algorithms do not saturate the benchmark and there is still a lot of room for improvement.

weaknesses:
- Unfortunately, while the paper is a decent attempt at building a safe RL benchmark, I am not convinced the safe RL community will be incentivized to use it. The main reason is that the notions of constraints in this benchmark are not directly tied to the very pragmatic safety considerations that need to be tackled in the real world - ranging from control systems to robotic deployments. 

- The benchmark feels a bit incremental compared to the already existing VizDoom framework that has been around for years. The modifications for the different levels and environments in this framework do not capture the notions of open-world generalization and realism the field is headed towards in terms of evaluating RL systems. In addition, a lot of prior safe RL works have bechmakred their systems on real-world systems like robotic navigation and manipulation, and I am not convinced that a modified VizDoom framework is likely to create a reasonable impact in the community. 

- The evaluations are all with variants of PPO and no other safe RL algorithms are tested. It is unclear why this is the case, since in my understanding the benchmark should not be tied to a particular type of algorithm

questions:
Please refer to the weaknesses above:

- Unfortunately, while the paper is a decent attempt at building a safe RL benchmark, I am not convinced the safe RL community will be incentivized to use it. The main reason is that the notions of constraints in this benchmark are not directly tied to the very pragmatic safety considerations that need to be tackled in the real world - ranging from control systems to robotic deployments. Could the authors clarify how exactly they envision this benchmark to drive innovation in the safe RL community? And what sub-field of researchers would be likely to use it?

- The benchmark feels a bit incremental compared to the already existing VizDoom framework that has been around for years. Can the authors clarify if the proposed modifications are non-trivial and if they can be broadly applied to potentially other frameworks like Minecraft and other games?

- The evaluations are all with variants of PPO and no other safe RL algorithms are tested. It is unclear why this is the case, since in my understanding the benchmark should not be tied to a particular type of algorithm. Please clarify the evaluations and if there is any specific assumption on the type of safe RL algorithms that could be tested on the benchmark? 

- Can the authors make 1-1 comparisons with the proposed benchmark and the features of prior simulated and real world benchmarks that have been used by safe RL papers in  the past?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710315709

thread_type:
official_review

<thread_object>
creator: reviewer_nmgz

<content>
summary:
HASARD is a benchmark testing platform specifically designed for safe reinforcement learning, based on ViZDoom, providing a diverse range of 3D environments.

1. The tasks on this platform require agents to pursue high rewards while considering safety strategies, moving beyond simple 2D navigation to incorporate complex elements such as spatial understanding.
2. HASARD offers three difficulty levels and supports both soft and hard safety constraints, flexibly adapting to varying safety requirements.
3. The platform integrates Sample-Factory, enabling high-speed simulation that allows agents to address real-world safety challenges while reducing computational costs.
4. HASARD includes six environments based on ViZDoom and benchmarks various methods to demonstrate the limitations of existing technologies.

strengths:
1. The authors tested six baseline algorithms on HASARD and provided an analysis of the results.
2. The tasks move beyond simple 2D navigation to incorporate complex elements such as spatial understanding

weaknesses:
1. The reviewer believes that if the distinction between soft and hard constraints is merely based on whether the threshold is $0$, then other benchmarks share this characteristic, making this claim somewhat unsubstantiated.
2. Although multiple methods were tested in the current experiments, there is a lack of analysis on performance under different safety budgets. It is recommended to include experiments with varying safety thresholds to better understand the trade-off between safety and reward for each algorithm.
3. HASARD is based on the ViZDoom game engine, which, while computationally inexpensive, lacks detailed simulation of real-world physics.
4. The anonymous video link provided by the authors is inaccessible.

questions:
1. The article does not provide an in-depth analysis of performance under different safety budgets. Is there a plan to supplement the experiments with varying safety thresholds to comprehensively demonstrate the trade-offs between reward and safety for each algorithm? This would be very helpful in understanding the adaptability of different methods under various safety requirements.
2. Considering the limitations of ViZDoom in simulating real-world physics, have the authors explored other engines with superior physical simulation capabilities (e.g., Isaac Gym)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723176606

thread_type:
official_review

<thread_object>
creator: reviewer_fzxy

<content>
summary:
The paper presents HASARD, a benchmark tailored to vision-based safe reinforcement learning (RL) using egocentric, pixel-based inputs. Built on the ViZDoom platform, HASARD comprises six unique 3D environments across three difficulty levels, each designed to test safe RL in increasingly complex and dynamic scenarios. The benchmark allows for a range of agent objectives, from navigation to item collection and hazard avoidance, focusing explicitly on embodied safe RL with vision-based inputs.

strengths:
Diverse Scenarios: HASARD provides varied 3D environments with different objectives and challenges, such as item collection, navigating hazardous terrain, and avoiding neutral units. This variety enriches the learning and testing possibilities, ensuring that the benchmark assesses both task performance and safety considerations.

Structured Curriculum: By offering three difficulty levels, HASARD presents a built-in curriculum for training RL agents, allowing gradual learning in increasingly challenging conditions. This approach is effective for developing robust agents that can generalize to new, more complex scenarios.

weaknesses:
Outdated Baselines: All the baseline algorithms were published over two years ago, and the original implementations of these baselines do not support visual inputs. The lack of SOTA vision-input baselines, such as Lambda [1], Safe SLAC [2], and SafeDreamer [3], limits the benchmark’s relevance in evaluating current state-of-the-art safe RL methods.

Solvability by Existing Algorithms: Have the tasks introduced in this framework already been solved by existing algorithms? For instance, can PPO-PID successfully address these tasks? Are there settings within HASARD that current algorithms struggle to handle? By not including experiments with the latest baselines, it is unclear whether the HASARD benchmark will drive the development of new algorithms or simply reaffirm existing solutions.

Task Complexity:
What is the primary contribution of HASARD compared to existing safety benchmarks, such as Safety Gymnasium? Compared to Safety Gymnasium, HASARD primarily adds hard constraints and fast simulation. However, implementing hard constraints is relatively straightforward, merely requiring a single line of code to terminate the episode upon any unsafe action. As for fast simulation, HASARD achieves this by sacrificing simulation fidelity and simplifying the action space, which limits its meaningfulness as a contribution compared to Safety Gymnasium.

Moreover, most tasks in HASARD revolve around avoiding hazardous obstacles, which has already been extensively addressed and solved in Safety Gymnasium by existing algorithms (e.g., [1-3]). Given HASARD's simplified dynamics and action space, it would need to introduce more complex tasks than those in Safety Gymnasium to stimulate the development of new algorithms. However, I did not observe any such complexity in the task design that would distinguish it from prior benchmarks.

[1] CONSTRAINED POLICY OPTIMIZATION VIA BAYESIAN WORLD MODELS

[2] Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representation

[3] SafeDreamer: Safe Reinforcement Learning with World Models

questions:
1. Which tasks in HASARD require memory capabilities, and which involve long-horizon decision-making? It would be helpful if the authors could clarify how the benchmark challenges an agent’s memory and planning capabilities over extended time sequences.

2.  Why did you choose ViZDoom to build this benchmark? Does this platform offer specific advantages? From my perspective, it seems that ViZDoom allows only minor modifications to its existing game structure and may lack the flexibility to define more complex, varied tasks. Why not consider using a truly open-world environment, such as MineDojo [4], which enables safer RL environments with more sophisticated task definitions? A platform like MineDojo could potentially support a broader range of scenarios and facilitate more diverse task creation.

3. Additionally, I noticed that you used Omnisafe for algorithm benchmarking, but this wasn’t mentioned in the paper. I have some questions regarding one of the baselines you implemented. In the P3O algorithm code (see here:https://github.com/PKU-Alignment/omnisafe/blob/main/omnisafe/algorithms/on_policy/penalty_function/p3o.py#L82), there is a term  J_c  in the loss function that appears to be independent of the network parameters. What effect does including J_c in the loss function have? I observed in your experimental results that P3O also fails to satisfy the constraints, which may be related to the J_c term. This raises some doubts about the effectiveness of this baseline.

[4] MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a new benchmark intended to be used for evaluating safe reinforcement learning algorithms, built on the ViZDoom platform. The benchmark introduces a variety of new environments and tasks. Experiments with a few baselines illustrate the headroom on the benchmark (current methods do not saturate it, but also achieve non-trivial performance). The claimed contribution is that it is the first to exclusively target vision-based safe RL.

### Strengths
Reviewers commented on the variety of environments and challenges introduced and the utility of the structured curriculum (three difficulty levels), analysis of a variety of baselines and that the benchmark requires solving tasks beyond simple 2D navigation, that the paper targets the important / significant problem of building realistic and reliable RL and safe RL benchmarks, detailed evaluations of a variety of baselines, the originality of the contribution, and the comprehensive design

### Weaknesses
Reviewers commented that some of the baselines were outdated, potential solvability by existing algorithms, unclear contribution relative to the Safety Gymnasium benchmark (including that most of the tasks revolve around avoiding hazardous obstacles), that the baselines were narrow, there was limited visual input analysis required, the action space was limited, and that the connection between the benchmark and real-world tasks was unclear.

The authors responded to these weaknesses (see my summary of the details below), and in balance, sufficiently addressed and dispelled arguments for rejection.