Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents Model-based PPO-Lagrangian (MBPPO-Lagrangian) algorithm for safe RL, which reduces epistemic and aleatoric uncertainty with an ensemble of neural networks and solves the underestimation problem of cost returns in a truncated horizon with the stricter threshold using a hyperparameter. The authors compared the algorithm in Safety Gym: PointGoal1 and CarGoal1. MBPPO-Lagrangian showed higher cumulative reward and lower cumulative cost performance than baseline methods. 
strengths_and_weaknesses: The authors successfully applied a model-based approach to the PPO-Lagrangian method by reducing the uncertainty of state transition using an ensemble of neural networks. Also, they suggested a simple solution that uses a hyperparameter to fix an innate underestimation-of-cost problem in the model-based approach that assumes a truncated horizon.

However, I suggest the authors strengthen the Experiments part. The environments used in experiments are not sufficiently explained and I feel the paper needs more baseline experiments. The more detailed suggestions will be described in the next section.  

questions: <Explanation of experiments>

- Please explain the environments you used. Also, please explain why you used PointGoal1 and CarGoal1 with more hazards, not PointGoal2 and CarGoal2. 
- The names ``PointGoal`` and ``PointGoal1`` both seem to be used to name the same environment (e.g. Figure 2), so please unify the name. The same goes for the CarGoal1 environment too. 
- Figure 4 is hard to interpret. I cannot understand what you want to say from Figure 4. Is there any reference that made you decide to show the data in this format? 

<More baselines>

I think the comparison between yours and Liu et al.[1] should be done since you mentioned the common points in L264. Also, Liu’s paper dealt with the same environment as yours and they are also model-based. 

<Details>

- L163: is $\rightarrow$ in 
- I think $t+1$ should be $t+l$ in Eq. 17 and Eq. 18.

[1] Zuxin Liu, Hongyi Zhou, Baiming Chen, Sicheng Zhong, Martial Hebert, and Ding Zhao. Safe model-based reinforcement learning with robust cross-entropy method, 2020.

limitations: Yes, the authors mentioned the limitation at the end of the conclusion.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a model-based method for safe RL that learns; a dynamics model of the environment, a reward value function, a cost value function, and a policy. The method leverages the learned dynamic models to generate imaginary rollouts to learn the reward and cost value functions, which are then used by a lagrangian relaxation of PPO in the setting of constrained Markov decision processes. 

The proposed method is evaluated in the Open AI safety Gym, where it achieved better rewards than constrained model-based baselines and it also obtained better sample efficiency with lower constraints violation than other constrained model-free approaches.
strengths_and_weaknesses: Strengths:
- The presented method is simple and builds on well-established methods. The results obtained a balance of the benefits of constrained model-based and model-free baselines,  showcasing better sample efficiency with low constraint violation wrt to constrained model-free approaches and better rewards wrt model-based methods.

- The paper is clear and well written. It presents a comprehensive related work and introduces a detailed background. 

Weaknesses:
- The approach is evaluated in only two environments (PointGoal, CarGoal). Although the environments were modified to make them more challenging, the work would benefit from evaluating the method in more setups like the Doggo environment (which is partially reported in the appendix), or other variations of the Point and Car setting like the Button or Push setups from the Open AI Safety Gym.

- The method seems to struggle with longer time horizon tasks, due to the compounding errors of the learned dynamics, reward value, and cost functions. A conservative safety margin was introduced to deal especially with the underestimation of cost returns. However, such a safety margin results in a strong tradeoff wrt the reward performance. 

questions: - A static choice of the safety threshold seems very conservative. As training progresses I could imagine that the estimation of cost returns becomes less uncertain and thus relaxing the safety thresholds could allow for better performance in terms of rewards. How would you incorporate model uncertainty to set the safety threshold dynamically?

- Is there any particular reason why the method trains the value and cost function using purely imaginary rollouts? When new data is collected to learn new dynamics models, would there be any benefits of using such data to also improve the learned value and cost functions?
limitations: Yes. Limitations of the method are mentioned in the conclusions. The potential societal impact was qualified as Not Applicable by the authors.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a new algorithm for a model-based safe RL (under the framework of CMDP). The proposed variant is modification of the PPO algorithm, namely by learning a model of the environment and introducing lagrangian relaxation to make sure the policy satisfies the safety constraints.

strengths_and_weaknesses: The paper is well written and easy to understand. The main algorithm - Algorithm 1 - is also very helpful.

The motivation is sound, the results are interesting, and the algorithm itself seems easy to implement.

What I find as a satisfying sanity check is the fact that unconstrained PPO does reach the same or better performance as MBPPO-Lagrangian. I understand that as the graphs (Figure 2 and 3) use as x-axis the number of environment interactions, the model-based approach is likely to do better initially. But as we keep going, PPO really needs to get there, especially as it is unconstrained and it clearly heavily violates the constraints.


Weaknesses

> [266] CarGoal1 by increasing the number of hazards from 10 to 15.

  I am not a fan of this.. It’s fine if that’s another environment in the experiments, but the original unmodified environment should also be included.

PR > 70% is a bit of an arbitrary choice, and I don’t see this being discussed anywhere. 

Due to the truncated horizon, the introduced \beta parameter and associated Equation (26) is something that is likely to be domain-specific. I would appreciate it if the authors could run Figure1 for multiple environments to see how the effect of beta varies across domains.

All the Figures, especially Figure 1 and Figure 4  are very hard to read, please increase the font size.

questions: How was the PR > 70% choice made? Did you run ablation analysis on this parameter?

Can you please run Figure 1 for more environments?
limitations: Adressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a safe reinforcement learning method in a model-based manner. The method uses the Lagrangian relaxation of the original constrained optimization problem and then uses dual gradient descent to find the saddle point. The method also improves the sample efficiency by learning an ensemble of dynamics models. Experiment results on Safety Gym environments demonstrate its effectiveness over model-free and model-based safe RL baselines.
strengths_and_weaknesses: ### Strength
- Proposed method uses an ensemble of dynamics models to improve sample efficiency and address the problem of epistemic uncertainties.
- Experiment results show that MBPPO-Lagrangian significantly outperformstones in Safety Gym environments.
- It found that the safety criteria should be tighter in order to achieve better performance when using truncated model rollouts.

### Weakness
- It might be better if the evaluation part could be more comprehensive since only the results of two tasks are reported.
- The authors use dual gradient descent to find the local saddle point of the Lagrangian relaxation of the original constrained problem. It would be better if they can provide some mathematical analysis of the effectiveness of this approach. 
questions: - About the experiment part, why does PPO-Lagrangian perform worse than MBPPO-Lagrangian asymptomatically? Since PPO-Lagrangian uses rollouts from the real environment while MBPPO-Lagrangian learns from model rollouts, the latter can have better sample efficiency but suffer from biases. So I was wondering why MBPPO-Lagrangian can outperform PPO-Lagrangian in terms of asymptomatic performances.
limitations: NA
review_title: null

Meta-review: metareview: This paper presents Model-based PPO-Lagrangian (MBPPO-Lagrangian) algorithm for safe RL, which reduces epistemic and aleatoric uncertainty with an ensemble of neural networks. The authors experimented the proposed algorithm in safety benchmarks such as  Safety Gym: PointGoal1 and CarGoal for which MBPPO-Lagrangian showed better performances and safety guarantees than other model-free/based safe RL baseline algorithms. This paper presented a model-based safe RL algorithm that has immense applications in RL for safety-critical problems. The paper is generally well-written and intuitive, with most concepts clearly explained. The safety results demonstrated in the experiments are convincing and the algorithms are easy enough to implement for most practical applications. Therefore, the review committee has a consensus of recommending acceptance for this work to NeurIPS 22.


confidence: Certain
Decision: Accept