Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper would like to address the problem of ``"safe exploration" with a conservative estimation of the environment. Although the problem seems reasonable, I have the following several concerns on this paper:

- Will the safety constraints be revealed to the agent? Standard RL assumes the reward is not revealed to the agent, and I feel the safety constraints cannot be revealed to the agent in prior as well.

- If the safety constraints is not revealed to the agent, then how to train the safety critic? In my opinion, when the agent collect the data that fail catastrophically, then the exploration is already not safe.

- I feel the derivation part in Section 3 is not so clear, as several notations have been introduced without explanation, though it is not hard to understand.

- I don’t understand what Lemma 1 and Theorem 1 can indicate in practice. When we estimate V_C, we already satisfies several catastrophic failures, otherwise we will know nothing about the possible failure. This can also be seen from the traditional exploration analysis. And traditional exploration analysis also says when we have sufficient number of data, the probability that we suffer from catastrophic failure will decrease. In one word, I don’t see any significant idea of ``safe exploration’’ from Lemma 1 and Theorem 1.

- Also, from the traditional exploration analysis, we have the regret lower bound, which says we must suffer from failure to get the best policy. We also have algorithms to match these regret lower bound on some simplified MDP settings, and even stronger guarantee that with high probability we will make small mistake when we have sufficient samples (see the mistake version of IPOC bound in [1]). I don’t feel the current paper present such kinds of guarantee.

A summarize of point 4 & 5: to me I don’t feel the authors well-defined the `''safe exploration''  and give a rigorous analysis on the ``the safety guarantee. For me, I prefer to define the "safe exploration" as minimizing the total failure when finding the optimal policy, but no matter how to define it,  I feel what the authors have done is not "safe".

I feel Theorem 2 is a straightforward adaption of the existing results in [2]. Though seems correct, the authors argue that Theorem 2 shows a tradeoff between safety and convergence, which I cannot agree. If we want the policy to converge to the global optimum, then we need to collect samples from the dangerous region (otherwise we cannot know if that’s dangerous and probably we can get some improvement at that region), how to keep "safe" at that time? Also, a slow convergence will need to run for a longer time and thus need more samples, which can enlarge the probability of catastrophic failure, right? 

Overall, though the empirical performance on some benchmark shows the proposed method is better than some of the previous methods, I don’t think the proposed methods really address the issue of the safe exploration. The authors does not formally define what is the safe exploration (and even little about exploration in fact), and the theoretical analysis also doesn’t address any of the safety issues. From my point of view, to achieve good performance, one must suffer from some failures, and our goal should be minimizing the number of failures, rather than minimizing the failure probability of each turn. Unfortunately, the authors does not characterize all of these issues I care about. I would like to say, considering the unsatisfactory of the problem description and theoretical analysis, I don’t think this paper is suitable for publication.

[1] Dann, Christoph, et al. "Policy certificates: Towards accountable reinforcement learning." International Conference on Machine Learning. PMLR, 2019.

[2] Agarwal, Alekh, et al. "Optimality and approximation with policy gradient methods in markov decision processes." arXiv preprint arXiv:1908.00261 (2019).

------------------
The authors answers one of the most important concerns, I have raised score to 6.
review_title: Review1

final_score: rating: 7: Good paper, accept
review_content: This paper introduces a method for performing safe exploration in RL. It addresses the problem of ensuring that partially-trained policies do not visit unsafe regions of the state space, while still being exploratory enough to collect useful training experiences. The proposed technique is based on learning conservative estimates of the probability of a catastrophic failure occurring at different states. Based on these, the authors show that it is possible to upper bound the likelihood of reaching an unsafe state at every training step, thereby guaranteeing that all safety constraints are satisfied with high probability. Importantly, the authors also show that (at least asymptotically), the method is no worse than standard unsafe reinforcement learning algorithms.

Overall, this is a well-written paper with sound mathematical arguments. The authors present a thorough review of related work, such as constrained MDPs, and argue that the proposed method requires fewer assumptions. For example, the proposed technique assumes access only to a sparse (binary) indicator of whether entering a particular state would result in catastrophic failure. The authors formally show that it is possible to upper bound the expected probability of failure during every policy update iteration (Thm1), which is a non-trivial result. All update equations are carefully derived and discussed in the appendix. The experiments are well-designed and convincing, and include evaluations of the proposed method (CSC) in 4 different domains, including an 18-DoF quadruped robot.

I believe that this paper introduces an important contribution to the RL community that is concerned with safety. It presents a principled method and introduces non-trivial bounds. In my opinion, this conference's community would benefit from having this paper accepted to its proceedings.


I have just a few questions for the authors:

1) safe exploration in RL is often expressed in terms of ensuring that, after every policy update, the new policy is no worse than the current one (or is worse by a bounded amount). In your paper, by contrast, safe exploration is related to the ability to avoid particular states. Can any of the bounds derived in this paper be used to ensure the former type of safety?

2) during policy evaluation, the proposed algorithm uses the safety critic, Q_C(s,.), to estimate how unsafe a particular state is. If an action sampled by the policy is deemed unsafe, the algorithm repeatedly re-samples new actions. Is this type of rejection sampling guaranteed to always stop? What happens if all actions available in s are unsafe? What if safe actions do exist but have zero probability under the current policy? Could the algorithm get stuck and stop exploring?

3) in Eq2, what is \hat{Beta}^pi?

4) the bound presented in Theorem 2 depends on an additional term K, which (so you argue) can be made small by picking alpha appropriately. How do you pick alpha, in practice, so that the algorithm trades-off convergence rate and safety?

5) when deriving the upper bound on V_C^pi_new (Eq26), you point out the epsilon_C cannot be computed exactly before the update, since it depends on the new optimized policy pi_new. How loose does the upper bound on V get by placing a trivial upper bound epsilon_C (epsilon_C <= 2)? Also, given that policy updates are constrained to keep the stationary distribution of successive policies similar, could this be exploited to achieve a tighter bound on epsilon_C?


review_title: This paper introduces a method for performing safe exploration in RL. It addresses the problem of ensuring that partially-trained policies do not visit unsafe regions of the state space, while still being exploratory enough to collect useful training experiences. This method makes fewer assumptions than those required by related techniques. The experiments are well-designed and convincing, and include evaluations of the proposed method (CSC) in 4 different domains.

final_score: rating: 7: Good paper, accept
review_content: (some difficulty following all of the proofs lowered the confidence of my evaluation)

## Summary
The authors lay out a new technique for safety-constrained exploration which reduces the likelihood of catastrophic failure. This is shown via extensive proofs providing theoretical guarantees on both convergence and the likelihood of failure as well as experimental results in a number of compelling tasks.

## Quality & Clarity
The authors provide regular and clear comparisons of their approach to related techniques described in other works. They provide extensive proofs and experimental results justifying their technique’s advantages.

## Originality & Significance
The work provides original techniques with both theoretical and empirical improvements over previous techniques.

## Suggestions
In the experimental section, the positive reward is clear but what is missing is how receiving negative reward for failure might compare to this approach. A natural comparison against methods that do not utilize the safety constraint features is to include a very large negative reward for failure and see if the algorithms can avoid failure via this signal instead of the separate safety signal. The authors emphasize the value of only accessing a binary signal to signify safety from the environment, so this could easily be considered a slight modification of the baseline algorithms: they induce a large negative intrinsic reward when they receive a failure signal. The trap experiment provides an especially clear setting for negative rewards: the agents could receive a lesser negative reward each time they lose a health counter before reaching catastrophic failure.

During policy evaluation, you resample actions when the likelihood of failure for a given action exceeds a certain threshold. I wonder how alternative approaches might perform, such as using the failure likelihood instead as a probability (perhaps scaled) of resampling; reweighting the action probabilities based on the failure likelihoods (maybe another action had a nearly identical expected return but a much lower chance of failure); or even utilizing the failure likelihood as an input to the policy (maybe in some scenarios the model stays as far away as possible from any risk but in others might choose to take a dangerous route that could lead to high reward). These alternatives also provide an answer to “what do you do when every action from a given state is above the threshold?” which is not specified in Section 3’s “Executing rollouts (i.e., safe exploration)” subsection nor in Algorithm 1.

Alternative approaches to the sampling procedure could potentially improve the tradeoff between safety and convergence. Where other methods might waste time exploring dangerous parts of the space, your method might be able to focus more time on the healthier parts of the states and converge to an effective policy sooner. At least, this may be true after the initial period of learning an accurate safety classifier, which as you point out takes some time for the agent to figure out. An experiment that could add to the paper would be to pretrain a safety estimator and then restart training with a newly initialized policy--how does this affect the convergence with different thresholds?

Finally, I would be interested to see how this work could be extended with non-binary safety constraints. The trap task provides a clear example of incremental danger (the agent is being “injured” before being finally destroyed) and being able to effectively utilize earlier hints signalling impending catastrophic failure would be a valuable skill.
review_title: Theoretical and practical advances in safe exploration

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this submission the authors are trying to tackle the very important problem of safe RL with safety guarantees. The problem formulation is rather clear, and the paper is overall well written. The main idea is to formulate the safe RL problem as  a CMDP problem, but with worst-case bounds to ensure that the safety constrained is guaranteed throughout the learning.

The general idea is fine, albeit not new (please check/compare against recent works on robust-CMDPs), however, I do have several issues with the lack of rigor in the mathematical proofs, as well as the rather amplified statements about safety guarantees 'throughout learning'.

Indeed, the CMDP problem (or its worst-case bound) is solved using a Lagrangian formulation, which is well-know as a soft constraints formulation, i.e., you do not have guaranteed safety during learning, as claimed in the Introduction (paragraph 3), and throughout the paper. In that case, the dramatic drone motivational example used in the Introduction (paragraph 1) is and exaggeration, i.e., this method will lead to a crash too.

Besides, mathematically, many of your variables are not defined, not even in the appendix, e.g., in equation (2) $\hat{B}$ is not defined; $\alpha$ which seems a key tuning parameter in Theorem 1 is not defined, etc.

Most importantly, the authors keep referring to the paper about CPO (Achiam et al 17) to support their proofs and technical derivations, whereas that paper is about continuous constraints costs. This paper on the other hand, is clearly using discontinuous constraints, not even $C^{0}$, and thus one cannot just use parts of the results in (Achiam et al 17) without further carful examination of the technical challenges introduced with a discontinuous cost function, e.g., you are using some Taylor developments throughout, while these only make sense for analytic functions, etc.

Furthermore, the probability bounds proposed in Theorem 1 and 2 seem to be rater weak bounds, since $\xi$ is bounded by a term inversely proportional to the confidence parameter $\omega$, which means the probability of being safe is high when the safety bound is loose, i.e., $\chi+\xi$, for $\xi\rightarrow +\infty$ . Similarly, for tight safety bound, i.e., $\chi+\xi$, for $\xi\rightarrow 0$, the safety probability drops to zero. The authors seem to minimize this point by the 'proper tuning' of $\alpha$ which remains a mysterious parameter (even after checking the proofs in Appendix).

This is all to say that the authors have to tone down their statements about guaranteed provable safety bound, quite a bit.

Finally, the numerical simulations are interesting, but only confirm my point about the fact that the obtained safety is asymptotic only, i.e., in steady state, and absolutely not during learning. This is clear from the plots in Figure 3 (bottom) where we see that in average the proposed algorithm necessitates more than 1000 iteration before reaching a truly safe behavior. One can also note that the CPO algorithm behaves almost similarly to the proposed algorithm, when tested on the car navigation example, and the laikago robot (Fig. 3, bottom number 3 and 4). One also wonders why in the first set of tests, in Fig. 3- top number 2, one can reach a performance cost of 20 with the proposed method, while it seems to plato at 10 in the second set of experiments, in Fig. 4-top number 2. Another point that is worth clarifying is that the tests in Fig. 4-top number 1, we see that the performance cost reaches 10 for the proposed method with large safety constraint bound (0.2), which is intuitively an almost unconstrained case. However, the unconstrained algorithm 'Base' in the first tests, Fig. 3-top number 1, the Base algorithm does not achieve a similar performance, could it be better tuned in that case ?

In summary, I found the paper well written, but it needs to be carefully revised for technical rigor, and toned down in terms of what is really achieved here (maybe somehow safer CMDP algorithm but definitely not safe during exploration).
review_title: The paper is generally well written, but lacks of mathematical rigor and makes unverified/ exaggerated claims.

Meta-review: Summary: 
This paper introduces a different, interesting definition of safety in RL. The paper does a nice job of showing success with empirical results and providing bounds. I think it provides a nice contribution to the field.

Discussion:
The reviewers agree this paper should be accepted. The initial points brought up against the paper have been successfully addressed or mitigated. 
Decision: Accept (Poster)