Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of safe reinforcement learning with hard constraints in an unknown stochastic environment. The key idea (and the contribution) is the proposed generative model based soft barrier functions that relax the hard safety constraints adapting to the unknown environment. This is further incorporated into the presented safe RL framework as a bi-level optimization process for jointly learning the optimal control policy, the soft barrier functions, and the initially unknown environment model, with avoidance of unsafe region in a probabilistic setting. Experimental results with two baseline approaches such as PPO-L and FOCOPS are presented to show the effectiveness of the proposed algorithm. 
strength_and_weaknesses: Strength:
+ The general idea of this paper is well written with clear motivation and good overview about the high level idea of the algorithm.
+ The bi-level optimization framework that jointly learns the policy and soft barrier functions is interesting.
 
Weakness:
- While the reviewer found the comparative analysis with CMDP algorithm is sound and solid, it is unclear how the algorithm could compare with other model-based safe RL algorithms, especially methods that combine RL and control theoretic approaches such as control barrier functions.
- The related work discussion needs to be improved, as some claims seem inaccurate given existing approaches that do not require explicit known model nor an initial safe policy, e.g. [Berkenkamp et al 2017].
- The proof of safety (Lemma 1) is substandard in absence of supplementary materials. For example, it is unclear how the conclusion of bounded probability for the ENTIRE trajectory could be reached with the presented proof, especially given the stochastic environment.
- The experimental results are reported with few trials and no quantitative performance in terms of mean and variances are given to justify the robustness of the performance.

clarity,_quality,_novelty_and_reproducibility: The studied problem of safe RL with bounded constraints satisfaction is an interesting topic and has gained significant attention in the learning and control community. The paper is in general well written to highlight the motivation, existing challenges, and the adapted approach of the safe RL framework with soft barrier function learning. With that being said, however, the technical details seem insufficient to justify the theoretical guarantee of the probabilistic safety and it is unclear how the algorithms would compare with those existing approaches that are not based on CMDP framework. Please see detailed comments as follows.

- Contribution and Novelty: the contribution of the proposed soft barrier functions and the bi-level optimization based safe RL framework is overall well presented, and the authors have made a good point in comparison to other CMDP based approaches. However, as mentioned it is unclear how the algorithms would compare to model-based learning with control theoretic approaches for safety considerations. Although it is claimed in the related work that “these approaches either require explicit known system models for barrier or shielding construction, or an initial safe policy…”, the following work did not require either of them:
[1] Berkenkamp, F., Turchetta, M., Schoellig, A., & Krause, A. (2017). Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems, 30.
	[2] Taylor, Andrew, Andrew Singletary, Yisong Yue, and Aaron Ames. "Learning for safety-critical control with control barrier functions." In Learning for Dynamics and Control, pp. 708-717. PMLR, 2020.

- Quality: one major concern is the technical quality in terms of proof of safety for the proposed bi-level optimization process. Given the unknown and stochastic environment, it is challenging to learn both barrier functions and the optimal policy safely without making further assumptions, e.g. bounded Lipschitz continuity, etc. With the presented analysis, it is unclear to the reviewer that (a) if the barrier functions are always learnable before violating a safety constraint, and (b) if the probability of the entire trajectory is always bounded with a satisfying probability due to unbounded stochastic noise. In the carpole example, a valid safe control policy might not exist if the cartpole is already at the boundary of a safety constraint with some initial velocity. And when the time horizon is very large, it seems intractable to bound the safety probability given the cascading effect of the step-wise unsafe probability. More discussions on this point could be helpful to make the safety conclusion more convincing. 

- The experimental results are reported with few trials and no quantitative performance in terms of mean and variances are given to justify the robustness of the performance.

summary_of_the_review: The paper presents an interesting topic of safe RL in an unknown and stochastic environment. While the general idea is clear, the reviewer feels that the technical part should be substantially improved given the insufficient details regarding safety guarantee, which is the core part of the presented approach. The experimental results also seems substandard without extensive trials under different environment settings. Authors are encouraged to address the comments in the future version of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed ways to do safe RL in an unknown stochastic environment. Specifically, they model the problem in hard constraints instead of the standard CMDP constraints and derived a novel barrier function method to incorporate the hard constraints into the RL algorithm. The authors manage to show with theoretical results that such an approach asymptotically achieves safety in policy optimization in RL, and they demonstrate the effectiveness of the proposed method on several safe RL benchmark problems for which the proposed method outperform SOTA in terms of safety guarantees and return performance.

strength_and_weaknesses: Strengths:
The barrier-based method for guaranteeing safety  is novel and is related to the work of guaranteeing safety in the form of stability in model-based RL algorithms. 
The problem of enforcing RL policy to be safe w.r.t. to hard constraints is interesting and important (in areas e.g., robotics).
The algorithms proposed is justified with theoretical studies (which utilize martingale properties of the barrier function to obtain a safety results in the asymptotic sense) and its benefits are also illustrated with some experiments over standard CMDP methods. 

Weaknesses:
It seems the safety bound only holds in asymptotic, and it is unclear whether this method ensures the property of safety during training. 
The experiments seem to be overly simple. Running this algorithm on more realistic benchmarks (e.g., Safety gym for safe RL experimentations) and doing more comparisons with other model-based safe RL methods will be better in terms of demonstrating the effectiveness of this method.
It is unclear whether the safety bounds provided in the theory can be justified by the experimental results 
The quality on the writing of the proofs is currently quite low, it requires more work to explain the notations and intuitions clearly in order to make it readable.
clarity,_quality,_novelty_and_reproducibility: Currently the paper is quite hard to follow. There are lots of notations in various parts of the proofs that have not been formally defined and well explained. Also some notations seem to be repetitive and not well defined (e.g., \tau), and for example the generative model loss is not clearly defined.  This makes checking the theoretical proofs of this work rather difficult. Several descriptions at the plots are also unclear or missing, which makes parsing the results difficult.

While some experimental setup has been discussed, it does not seem sufficient for readers to reproduce the results without the source code pf these methods. Thus this affects reproducibility.
summary_of_the_review: This paper studied an important safe RL problem and proposed a novel barrier-function based method. It also showed this method's effectiveness both theoretically (in asymptotic sense) as well as empirically on several safety RL benchmarks. However, it'd be great to verify the benefits of this method by 1) testing in more complex domains and 2) comparing with more model-based safe RL methods (e.g., Turchetta, M. et al Safe exploration in finite Markov decision processes with gaussian processes. NeurIPS 16). The writing of this paper also needs significant improvements for it to be readable/understandable by the broader ML audience. The current paper is quite hard to follow.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The focus of this work is on learning safe MDP policies, in the sense that a safety constraint holds across the entire trajectory of the policy. The distinction from the widely-studied CMDPs is that the constraint in the CMDP holds over a cost aggregated across the entire trajectory, and not at each point in the trajectory. The approach in this work is to learn a barrier function that is a supermartingale over the filtration defined by transitions, and thus with high probability is not violated during the execution. There is a (weak) theorem showing that in the asymptotic limit, the policy achieves safety. More compellingly, experiments on a few control benchmarks suggest that the method indeed avoids unsafe regions.
strength_and_weaknesses: The main strengths of the paper are (1) the approach to learning the barrier function seems to be fairly novel in this setting and (2) the experiments do seem to be pretty promising. In particular, on the technical side, I did appreciate the use of the (locally-defined) supermartingale constraint on the barrier function to obtain the safety bound.

The main weakness is that the paper is a bit of a mess -- I'll discuss this below.

An asymptotic theoretical guarantee is included, which is better than nothing, but certainly much weaker than we'd want to have if the objective was to ensure safety. Remark 1 proposes a "practical safety probability lower bound" that is used to obtain bounds reported in the experiments, that are indeed less than the empirical fraction of safe trajectories (1), but it isn't immediately obvious that the bound is valid.
clarity,_quality,_novelty_and_reproducibility: As mentioned above, the clarity is a substantial issue with this paper. For example, it seems that the notation changes at various points in the proof of the one theoretical claim of the paper (that or the objects under discussion appear without a proper introduction).  Along these lines, I am really not sure what l(tau,\hat{\tau}) means in the generative model loss function -- a likelihood is obtained for a realization against a distribution (model), but this seems to be a likelihood between two realizations. I am guessing that \hat{\tau} represents the distribution of the model, but that's just a wild guess.

Likewise, I can't make any sense of the plots in Figures 4-5---what's on the x-axis? And, what method is used to obtain the "gene" trajectory in Figure 3?

Otherwise, the quality and novelty of the work seem fairly strong, as noted above.
summary_of_the_review: The paper's presentation is rough, but the approach has some significant technical novelty and seems to be effective empirically.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper is on safe reinforcement learning (RL) with unknown stochastic environments. Under the safety definition in terms of reachability, the authors aim to enforce hard constraints rather than soft constraints that are often the main target of constrained Markov decision processes (CMDP) literature. In this paper, the authors introduce a notion of generative model based soft barrier functions to encode the hard safety constraints. The authors propose a safe RL algorithm to jointly model the environment and optimize the agent policy while satisfying the safety (reachability) constraints. Finally, this paper demonstrate the effectiveness of the proposed method in several benchmark problems.
strength_and_weaknesses: ### Strength
- Important and relatively-unexplored problem settings on safe RL with **hard constraints**.
- New algorithms based on shielding and generative models
- Good experimental results compared to CMDP methods and FOCOPS*.
- Overall, this paper is well-organized and easy to follow.

### Weakness
- Safe model-based RL
    - There are significantly few references to safe model-based RL. I know there is the following sentence, but still I consider the relevant methods should be mentioned in related work section and compared to the authors' proposed method.
    - > "However, these methods do not apply to our setting, where the environment is uncertain and stochastic without explicit dynamics, backup safe policy, or human intervention."
    - Hard constraints in safe RL has been studied in unknown (stochastic) environment. For example, Moldovan and Abbeel (2012) studied safe exploration in unknown environment in terms of ergodicity. Also, Turchetta et al. (2016) and Wachi et al. (2018) studied safe exploration problems in unknown environment while guaranteeing reachability and returnability under regularity assumptions. I think the authors should have surveyed wider safe RL methods (the three papers I mentioned are all hard safety constraint; that is, the agent is required to satisfy the safety constraint at every time step).
    - Though the above papers are indeed different from the authors' one in terms of problem formulation or assumptions, Yarden et al. (2021) can be mentioned in related work section and compared to the authors' method in the experiments. If I understand correctly, the method in Yarden et al. (2021) can be fairly compared to the authors' one, and source code is also open-sourced (https://github.com/yardenas/la-mbda),

- Experiments
    - Benchmark problems are very easy. Hopefully, Safety-Gym should be used for testing safe **deep** RL methods.
    - Computational time and sample efficiency should be also compared. I "guess" the authors' proposed method requires much more data until convergence.

### Minor comments and typos
- [Section 3.1. Equation regarding $J$] I think $t$ should be $i$ (or $i$ should be $t$).
- [Definition 2] $\tau(t)$ is not defined. Is $\tau(t) := [ s(i) ]_{i=t}^T$?

### References
- Moldovan, T., and Abbeel, P. 2012. Safe exploration in Markov decision processes. In International Conference on Machine Learning (ICML).
- Turchetta, M.; Berkenkamp, F.; and Krause, A. 2016. Safe
exploration in finite Markov decision processes with gaussian processes. In Advances In Neural Information Processing Systems, 4305–4313.
- Wachi, Akifumi, et al. "Safe exploration and optimization of constrained mdps using gaussian processes." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.
- As, Yarden, et al. "Constrained Policy Optimization via Bayesian World Models." International Conference on Learning Representations. 2021.

clarity,_quality,_novelty_and_reproducibility: ### Clarity
This paper is overall organized and easy to follow. I feel Section 3.1 is a bit messy and relatively hard to follow, but I don't have an idea for improving it.

### Quality
- As I mentioned in Weakness, the quality of related work section should be improved by surveying and referring more papers (especially on safe model-based RL). Also, in experiments section, the authors' proposed method should be compared with more baselines (e.g., Yarden et al. (2021) in more complicated environments (e.g., Safety-Gym).

### Novelty
- As far as I know, this proposed method is novel, but generative model as a form of world models has already exists; hence, several sentences should be modified.

### Reproducibility
- The source code is not attached and it is not sure whether it is open-sourced upon publications, so I need to say that reproducibility is low for now.

summary_of_the_review: I think this paper has several merits in terms of 1) important problem settings, 2) new algorithm, 3) good experimental results (easy benchmark and limited baselines), and 4) good writing.

However, I have several concerns in terms of 1) how to deal with existing safe model-based RL literature and 2) empirical experiments, as stated in Weakness.
Therefore, I recommend rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers believe the paper studies an important problem, the proposed method is novel, and the technical side of the paper is strong. However, there are concerns about 

1) The clarity of the paper. The reviewers see it as a major issue. The paper is not easy to follow and it is hard to properly verify the results. There are undefined notations and notation changes in the proofs. There is not enough information about the experimental set up to allow a reader to reproduce the results. 
2) The derived safety bound is only asymptotic, which is weak when the goal is to ensure safety. If the bound is only valid when the generative model gets (asymptotically) close to the real environment, then the title of the paper "safe RL in unknown stochastic environments" seems to be slightly misleading. 
3) The problems used in the experiments are overly simple and the results do not properly support the theoretical findings.

I believe the paper has potential but it is not ready for publication and needs to be improved. I would strongly recommend that the authors take the reviewers' comments into consideration, revise their work, improve its quality, and prepare it for future conferences. 

justification_for_why_not_higher_score: Lack of clarity in writing which makes it difficult to properly evaluate the results. Simple experiments that do not support the theoretical findings appropriately. The theoretical results are only asymptotic which is weak if the objective is to ensure safety.
Decision: Reject