AsymDreamer: Safe Reinforcement Learning From Pixels with Privileged World Models

Authors: Dongchi Huang, Kaige Zhang, Yang Li, Yi Zhan, Chunhe Xia
Primary area: reinforcement learning
Keywords: Safe Reinforcement Learing; World Model

Abstract:
Safe Reinforcement Learning from partial observations frequently struggles with rapid performance degradation and often fails to satisfy safety constraints. Upon deeper analysis, we attribute this problem to the lack of necessary information in partial observations and inadequate sample efficiency. World Models can help mitigate this issue, as they offer high sample efficiency and the capacity to memorize historical information. In this work, we introduce AsymDreamer, an approach based on the Dreamer framework that specializes in exploiting low-dimensional privileged information to build world models, thereby enhancing the prediction capability of critics. To ensure safety, we employ the Lagrangian method to incorporate safety constraints. Additionally, we formulate our approach as an Asymmetric CPOMDPs (ACPOMDPs) framework and analyze its superiority compared to the standard CPOMDP framework. Various experiments conducted on the Safety-Gymnasium benchmark demonstrate that our approach outperforms existing approaches dramatically in terms of performance and safety.

Decision: reject
Reviewer scores: 6, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730129704777

thread_type:
official_review

<thread_object>
creator: reviewer_xknh

<content>
summary:
In this work, the authors address the challenge of exploiting asymmetric inputs under the CPOMDPs framework. They propose AsymDreamer, a new algorithm that uses privileged information to build a world model for the critic. They also introduce the ACPOMDPs framework, an extension of CPOMDPs allowing asymmetric inputs for the actor and critic. Theoretically, asymmetric inputs reduce critic updates and lead to a better policy. AsymDreamer constructs two world models, one for the actor based on historical information and another for the critic with privileged information. It is integrated with the Lagrangian method and shows competitive performance on the Safety-Gymnasium benchmark and strong adaptability to complex scenarios.

strengths:
* The proposed method of combining DreamerV3 with two world models and Lagrangian methods, which is novel and well-motivated.
* The paper presents empirical results that compare AsymDreamer with many relevant model-based and model-free baselines in several tasks, achieving competitive performance on the Safety-Gymnasium benchmark.
* The paper is mostly well-written.

weaknesses:
* The work is built on top of SafeDreamer, incrementally adding another world model for critic, and differences in terms of the loss optimized are not sufficiently highlighted.
* The DreamerV3 is basically combinations of existing methods like CPOMDPs, Augmented Lagrangian, and RSSM. While this is not a problem in itself as they are common in model-based reinforcement learning, there is very little discussion on why these particular methods were chosen and what possible alternatives from the literature exist and whether they might yield better results.
* Lack of introduction to baseline algorithms. And in the PointGoal2 scenario, the performance results seem to be different from SafeDreamer reported in the paper.
* Stability verification needs to be considered. How the hyperparameters in equation (8), (9) are selected, and certain ablation experiments need to be conducted.
* Whether modeling two world models will lead to a significant increase in learning time and the effectiveness of model learning also need to be considered.
* Privileged information and partial observations should be considered in more environments. A single scenario in QuadrotorGoal1 cannot be trusted to determine whether privileged information enhances performance.

questions:
It would be great if the authors could address the weaknesses I outlined above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730601967466

thread_type:
official_review

<thread_object>
creator: reviewer_28vi

<content>
summary:
This paper focuses on the safe RL problem that struggles with performance degradation and often fails to satisfy safety constraints. They attribute this problem to the lack of necessary information in partial observations and inadequate sample efficiency. Specifically, they exploit low-dimensional privileged information to build world models, thereby enhancing the prediction capability of critics. The authors propose Asymmetric Constrained Partially Observable Markov Decision Processes, a relaxed variant of CPOMDPs. The key distinction is that ACPOMDPs assume the availability of the underlying states when computing the long-term expected values. To ensure safety, they employ the Lagrangian method to incorporate safety constraints. The experiments conducted on the SafetyGymnasium benchmark demonstrate that the proposed approach outperforms existing approaches dramatically in terms of performance and safety.

strengths:
1.	The idea of using privileged information in the Dreamer structure is interesting. Separating observation modeling and task-centric prediction modeling avoids the potential trade-off between these two tasks. This also allows the observation world model to capture more detailed observation information, thus enabling the actor model to achieve better performance with richer input features.
2.	The paper is generally well-written and well-organized. Figure 1 clearly shows the training pipeline.

weaknesses:
1.	**Justification of using privileged information**. It seems that the motivation for using privileged information is this sentence “Since training is often conducted in simulators, there is potential to leverage privileged information during training to reduce uncertainty from partial observations”. Does the proposed method have potential applications beyond simulations, such as in real-world scenarios? In addition, I am not sure if it is fair to compare with other methods that do not use privileged information.
2.	**Evaluation results**. The experimental evaluation is only conducted on 4 tasks, including one self-made task. The authors may need to include all the remaining tasks in the Safety Gymnasium. 
3.	**Missing baseline on the same benchmark** Although this paper proposes a model-based method, I think it is still meaningful to compare with some well-known methods on the same benchmark. This website contains some results that can be used as reference: https://fsrl.readthedocs.io/en/latest/tutorials/benchmark.html. 
4.	There is no clear evidence to show the benefit of using privileged information. The results in Figure 5 need more investigation and explanation. Otherwise, it is hard to summarize the main conclusion of the proposed method.

questions:
1.	I think in Figure 1, the encoder, decoder, and hidden state are different for the two world models. So, it would be clearer to use different colors of notations for them to avoid confusion.
2.	Why not use SafeDreamer as a baseline?
3.	In Figure 4, why do all baselines have constant cost? Don’t they vary across training steps? What is the target cost limit?
4.	What does the red solid line mean in Figure 4 and Figure 5?
5.	There is no explanation of the baselines. What does OSRP mean? It would be better to include simple descriptions of each baselines in the appendix.
6.	The ablation study results in Figure 5 raise a lot of questions. I think the authors also feel surprised that the privileged world model fails to train a viable cost predictor when taking privileged information as input. In addition, DreamerV3 is much better than AsymDreamer(S) in terms of the reward. This is also hard to interpret. I think the authors should do more experiments to explain these results to provide insights into the model. Otherwise, there seems no clear message that can be summarized in this paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730630835670

thread_type:
official_review

<thread_object>
creator: reviewer_sz2a

<content>
summary:
The authors propose AsymDreamer, a Dreamer-based safe reinforcement learning framework that utilizes low-dimensional privileged information to construct world models. The world model in AsymDreamer features two branches: the Privileged World Model, which takes a handcrafted low-dimensional vector as input, and the Observation World Model, which uses partially observed images (64x64 RGB). Additionally, the authors formulate their approach within the framework of Asymmetric CPOMDPs (ACPOMDPs) and integrate AsymDreamer with the Lagrangian method. Empirical results show that AsymDreamer outperforms existing safe reinforcement learning methods on the Safety-Gymnasium benchmark.

strengths:
1. The theoretical analysis of ACPOMDPs in this paper is comprehensive. The authors propose the ACPOMDPs framework and provide a detailed analysis showing that asymmetric inputs reduce the number of critic updates and lead to a more optimal policy compared to the standard CPOMDPs framework.
2. The authors introduce privileged information into the RSSM world model, enhancing the model's imaging capabilities and consequently improving the performance of the safe policy.

weaknesses:
1. The methodology in this paper is too similar to prior work[1], and lacks sufficient novelty. In my view, the only difference is the inclusion of privileged information in the modeling of the RSSM world model.
2. The authors are encouraged to include an ablation study to analyze the impact of adding privileged information to the observation input on the baseline algorithm.
3. There are some typos in the paper that need to be corrected in the next version (e.g., line 400 and figure 2).

[1] Huang, Weidong, et al. "Safe dreamerv3: Safe reinforcement learning with world models." arXiv preprint arXiv:2307.07176 (2023).

questions:
1. Please clarify the similarities and differences between the proposed method and SafeDreamer[1].
2. The reviewer is confused about what the global state in your privileged world model input is and where the privileged information (the low-dimensional vector from Appendix E) is used.

[1] Huang, Weidong, et al. "Safe dreamerv3: Safe reinforcement learning with world models." arXiv preprint arXiv:2307.07176 (2023).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730784820842

thread_type:
official_review

<thread_object>
creator: reviewer_a1zn

<content>
summary:
This paper presents AsymDreamer, an approach based on the Dreamer framework that specializes in exploiting low-dimensional privileged information to build world models, thereby enhancing the prediction capability of critics. AsymDreamer employs the Lagrangian method to incorporate safety constraints. This paper formulates the proposed approach as an Asymmetric CPOMDPs (ACPOMDPs) framework. Experiments on the Safety-Gymnasium benchmark demonstrate that AsymDreamer outperforms existing approaches in both performance and safety.

strengths:
- The paper is well-motivated and novel. It successfully extends the Dreamer framework to handle asymmetric information in RL, presenting a novel ACPOMDP framework. It shows the effectiveness of using privileged Information.
- The results from Safety-Gymnasium benchmarks show that AsymDreamer outperforms baseline models in both task performance and safety metrics, especially in complex scenarios like 3D navigation. The thorough ablation studies are also conducted.
- The paper includes rigorous theoretical analysis and compares ACPOMDP to standard CPOMDP.

weaknesses:
I am not in this field so I am unable to evaluate the signficance of the proposed method and ACPOMDP problem/analysis.

Some of my concerns include:
- Over-reliance on privileged information:  AsymDreamer’s performance relies on privileged information during training, which may not be available or hard to simualte in real-world environments. It potentially leads to bad performance or compromised safety in environments with limited or unavailable privileged information.

- Extension to real-world environments:  The experiments on Safety-Gymnasium benchmark are a bit toyish (even the most challenging 3D navigation one, although it may also be the cases for the related work). Additional testing in real-world environments would be beneficial to demonstrate the effectiveness.

questions:
I am not familar with this field at all and I cannot evaluate the novelty of this work (especially over its predecessor / related works). It is an interesting read and I didn't identify significant issues. The paper is well motivated with solid analysis and experiments. My major concern is that this paper only evaluates on toyish environments and may not generalize well to real-world scenarios.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733029185397

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response to Reviewers

comment:
We sincerely thank the reviewers for their detailed and insightful comments. Based on the feedback received, we have made the following revisions to the manuscript:
- We corrected errors throughout the manuscript and revised the methodology section to clearly highlight the differences from SafeDreamer.
- We added the SafetyRacecarGoal1 and SafetyCarGoal1 scenarios to enhance the comparison experiments.
- We conducted an ablation experiment in the SafetyRacecarGoal1 scenario to provide a clearer explanation and investigation of the role of privileged information.
- We included a comparison experiment with model-free baselines, demonstrating that AsymDreamer is significantly more effective than the baselines.
- We performed a comparative analysis of the training elapsed time for different algorithms, revealing that the inclusion of privileged information results in a 50% increase in training elapsed time.

The reviewers' comments have been instrumental in improving our paper. We kindly request that the reviewers review our responses and reconsider their scores.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper approaches the problem of safe reinforcement learning in partially-observed environments. The paper proposes a method that centers around a particularly type of constrained POMDP that enables privileged access to state observations in certain settings. Reviewers found this paper well-motivated and novel, although doubts were raised about the sufficiency of the experimental comparisons, the justification for using privileged information potentially limiting the application to simulation only, and that there's no clear evidence to show the benefit of using privileged information.

Authors responded to these weaknesses in a variety of ways. Despite encouragements to engage, all but 1 reviewer did not meaningfully engage with the author responses (the reviewer who rated this paper a 6). They were the only reviewer rated this paper above the acceptance threshold, albeit with low confidence (2).

Given the limited engagement from reviewers, the only very-mildly positive initial reception, and my own read of the paper to check for things that the reviewers missed, e.g., if the paper was highly novel in a way that reviewers misunderstood, I recommend rejection. I concur with the overall assessment that more experimental validation and written justification is needed to clarify and support the paper's claims, particularly in providing evidence of the applicability of the setting / assumptions to solve non-toy problems. I also think the paper could be improved with an overview figure that makes clear the distinction between when privileged information is available and when it is not, and ground it in an example (or examples) from the experiments.