Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors of this study provide a risk-preventive training strategy for safe RL that trains a statistical contrastive classifier to forecast the likelihood that a state action combination would result in unsafe states. They can gather risk prevention trajectories and remodel the reward function with risk penalties based on the expected risk probability to promote safe RL policies. In robotic simulation environments, the authors run experiments. The outcomes demonstrate that the suggested methodology outperforms traditional model-free safe RL approaches and is on par with cutting-edge model-based methods in terms of performance.
strength_and_weaknesses: **Strengths:**
- The motivation is clear and promising.
- The prior art is discussed concisely.

**Weaknesses:**
- I couldn't understand some statements in the paper as they seem unjustified. For instance, the classifier $F_{\theta}$ is trained on randomly selected state-action pairs that are treated as "safe" and some pairs that are "unsafe", which are selected in the previous exploration. According to this, the Bayes classifier for such a problem is given by $p(y= 1|s, a)$, which is defined as:
$p(y = 1|s, a) = \frac{p(s, a|y=1)p(y=1)}{p(s, a)} = \frac{p(s,a|y=1)p(y=1)}{p(s, a|y=1)p(y=1) + p(s,a|y=0)p(y=0)}$,

where $p(s,a|y=0)$ is the probability over random state-action pairs in the dataset, corresponding to the noise distribution for contrastive learning. If we consider that $F_{\theta}$ is an expressive class containing $p(y=1|s,a)$, then we asymptotically have $F_{\theta} \rightarrow p(y=1|s,a)$ on $(s,a)$ pairs that lie in the distribution on which the classifier is trained. The paper, however, equates $p(y=1|s,a)=\frac{F_{\theta}(s,a)}{1 - F_{\theta}(s,a)}$, which is left unjustified and I couldn't understand it. Due to such an unjustified claim, Lemma 1 and Theorem 1 become inconsistent to me.

- Why didn't the authors consider giving a continuous "safeness" value between 0 and 1? In my opinion, using a binary representation for the safety of a state-action pair is taking the easy way out. Considering a continuous value, on the other hand, would increase the novelty.
- The comparative evaluations are insufficient as the authors only consider Ant, HalfCheetah, Humanoid, and  Hopper. While these can be considered challenging tasks, I expect the results for BipedalWalker (from BOX2D), Walker2d, and Swimmer. Although the latter has a low state-action dimensionality, it is a very challenging task in which many SOTA methods exhibit poor performance. 
- Although it is indicated for the HalfCheetah environment that the robot violates the safety constraint when it falls over, I don't believe it is a credible judgment. Because the Cheetah environment is "stable", a trajectory doesn't terminate unless a pre-specified number of time steps is reached, i.e., 1000. While it'd good to examine the proposed method's performance in a stable environment, the authors don't provide such a discussion. 
- The authors should assess a method's performance on a distinct evaluation environment per a pre-specified number of time steps, as it is a common practice in deep RL literature. For example, every 1000 steps, the learning is stopped, and the agent is evaluated in a distinct environment where no exploration or learning is performed for 5-10 episodes. However, the authors used the training rewards as far as I understood. 
- The number of random seeds used to produce the reported results is highly insufficient. At least ten random seeds should be used [per the deep RL benchmarking standards](https://ojs.aaai.org/index.php/AAAI/article/view/11694).

**Detailed Comments:**
- An RL agent should be "an RL.." not "a RL".
- Replace the reward function in the MDP definition with $\mathcal{R}$ instead of $r$ as it can confuse readers.
- Punctuation should be put after each equation as they are used in conjunction with the text.
- It looks ugly when a reference is used, such as: "Similar to (Hans et al., 2008)...". Either refer to the natbib package or use it as: "Similar to the work (Hans et al., 2008)...".
clarity,_quality,_novelty_and_reproducibility: **Clarity:** The authors are clear throughout the paper in expressing their motivation and describing their methodology (except for unjustified claims and math in the paper). 

**Quality:** The study is a bit behind the quality standard for a venue like ICLR due to my concerns with the credibility of the experiments. See the _Weaknesses_ section.

**Novelty:** In my opinion, the proposed method can be regarded as novel.

**Reproducibility:** I don't believe the reported results are reproducible as the authors did not provide a source code, although they extensively explain the implementation and experimental setup. In fact, the number of seeds used to evaluate the methods' performance is not sufficient.
summary_of_the_review: While the authors appear clear in their claims and methodology, I don't believe the paper can be accepted to a venue like ICLR. My main concerns with this study are:
1) The simple binary assumption used to define the cost function $c(s, a)$ and unjustified claims.
2) The credibility of the experiments:
- An insufficient number of seeds.
- Insufficient environments.
- Reported returns are training rewards, as far as I understood.
3) Reproducibility as no code or any anonymous repository, which does not reveal the authors' identity, of course, is provided.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies safety in reinforcement learning. The goal is to maximize return while keeping the cost of visiting unsafe state-action pairs below a threshold. The paper proposes learning a classifier $F_\theta(s, a)$ that predicts whether a given state action $(s, a)$ will lead to eventually visiting an unsafe state-action pair. This is then used to update the policy and the $F_\theta$ is then relearned on the fly.  
strength_and_weaknesses: Strength:

1. Paper is on safe RL which is an important topic

Weakness:

This paper suffers from three concerns.

1. *Broad Concerns:* The paper makes assumptions that will not apply to applications that the paper uses for motivation in the beginning. To begin with, the paper assumes a deterministic setting that holds in very simple cases. Secondly, a real robotic experiment can immediately fail if the robot hits a wall and breaks down. Therefore, encountering unsafe states even once may not even be an option in such cases. Lastly, $p(y = 1\mid, s, a)$ is a non-stationary distribution that evolves as the policy evolves. This seems undesirable from a learning point of view, as the model has to constantly adjust to the changes in the policy. As the policy optimizes for state-action pairs that are safe/unsafe based on value of $p(y=1 \mid s, a)$ and not directly the value of $c$, therefore, the policy may start taking certain actions or avoid actions, in a given state, as the training progresses. A more desirable strategy would be that $(s, a)$ pairs that are marked as unsafe are never revisited. 

2. *Technical Concerns:* Several statements in this paper seem either wrong or unjustified to me. It is my understanding that the classifier $F_\theta(s, a)$ is trained on random $(s, a)$ pairs that are treated as safe $(y=0)$, and a set of unsafe pairs $(y=1)$ that were selected during the previous exploration. If yes, then the Bayes classifier for this problem will be given by $p(y=1 \mid s, a)$ where

$p(y=1 \mid s, a) = \frac{p(s, a \mid y=1) p(y=1)}{p(s, a)} = \frac{p(s, a \mid y=1) p(y=1)}{ p(s, a \mid y=1) p(y=1) + p(s, a \mid y=0) p(y=0)}$.

Here $p(s, a \mid y=0)$ denotes the probability over random state-action pairs in the dataset, as that is the noise distribution for contrastive learning. Now if $F_\theta$ is an expressive class that contains $p(y=1 \mid s, a)$ then we have asymptotically $F_\theta(s, a) \rightarrow p(y = 1 \mid s, a)$ on $(s, a)$ pairs that lies in support of the distribution on which it was trained.  

The paper equates $p(y =1 \mid s, a) = \frac{F_\theta(s, a)}{1 - F_\theta(s, a)}$ which does not make sense to me.  Note that in the above version, no correction is required and all probability values are in [0, 1]. 

Because of this error, or confusion, I am unable to fully check Lemma 1 and Theorem 1. 

In another example, assumption 1 seems unclear to me. Does the increase have to be strict? If not, this already rules out deterministic policies since a deterministic policy in a deterministic setting (which is what this paper assumes) will result in a straight path and if this policy visits an unsafe trajectory, then $p(y=1 \mid s, a) = 1$ for all state-action pairs along the way.  This assumption needs to be made clearer to understand it clearly, and an example can help. 

3. *Writing Concerns:* The paper is not well-written. I give numerous examples below in the clarity section.
clarity,_quality,_novelty_and_reproducibility: Clarity/Writing Quality

1. The first point at the end of the introduction, is not a contribution but a description of what is being done. 
2. Definition of $\tau$ uses $|\tau|$ which leads to a circular definition. You can define $|\tau|$ as the number of actions in the trajectory in a second line.
3. Definition 1 is also partially an assumption since it is assumed that $c$ is binary-valued. 
4. Definition 2 is likewise not a definition. It is an algorithmic choice. The true safety is decided by $c$ and not $\eta$ and $p(y=1 \mid s, a)$.
5. Unsafe state set $S_U$ is actually a state-action set
6. What is $\pi$ on line 8? Does it mean $\pi_\phi$
7. What do lines 14 and line 15 do in the Algorithm? Are they being used in optimizing $F_\theta$. 
8. $F_\theta$ is the learned model and not the Bayes' classifier. It could be good to separate these two notions. 

Suggestions:
1. Consider using $\pi^\star$ as opposed to $\pi^*$ 

Questions:
1. Can definition 1 be generalized to handle stochastic transitions?
summary_of_the_review: Because of concerns with methodology, writing, and technical statements, I am recommending rejection. My score can change greatly if the authors clarify any misunderstanding. In particular, I would like to know if technical statements concerning the relation between the Bayes' classifier $p(y=1 \mid s, a)$ and the learned classifier $F_\theta$ is true.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a contrastive-learning inspired objective for shaping the reward function for RL agents to prevent encountering predicted, future “unsafe” states while training. They validate their claims on standard Mujoco environments.
strength_and_weaknesses: For full disclosure, I have already reviewed a previously submitted (and rejected) version of this paper. My review is based on criticisms/strengths from the old review and old meta-review, along with the changes that have been made to the paper in this new submission. Overall, this paper is much improved from the original version I reviewed.

## Strengths

**Clarity:** The paper is generally clear and readable.

**Mathematical Rigorousness:** 

- The contrastive classifier $F_\theta$’s intuition isn’t described very well, but other than that Section 4.1 describing the objective is intuitive and grounded
- The proof of Theorem 1 is well-presented and presents a mostly-grounded way of selecting the dual variable $\lambda$ rather than just treating it as a hyperparameter like in other works.

**Experiments/Results:**

- Experiments and ablations are sensible. The results presented hold up the claim that this works as well as SMBPO while being a model-free method, which is valuable due to fewer assumptions made by model-free algorithms about the environment.

## Weaknesses

**Clarity:** 

- Figure 1 is not very useful for demonstrating how the method works. There’s no indication of some risk prediction threshold ($\eta$) nor anything to indicate that actions are actually modified based on this threshold ($a_t, a_t’$ aren’t differentiated).
- The contrastive classifier $F_\theta$’s intuition isn’t described very well, and Eq. 2 just appears without explaining the terms. I understand that this derivation of $F_\theta$ makes Eq 3 convenient (and maybe the authors figured out Eq 3 before figuring out Eq 2), but this could be made more clear.

**Experiments:** 

- Assumption 1 is presented without empirical evidence it exists. Can the authors experiment and demonstrate how true Assumption 1 is?
- It’s hard to say that RPT+DA is better, as the authors claim, due to the overlapping standard deviations.

**Method and Framing:** 

- Intro: “benefitting from the generalizability of risk prediction, the proposed approach can avoid safety violations much earlier in the training process” → It’s not clear intuitively how this risk prediction differs from the aforementioned prior work at this point in the intro, and therefore how it is more generalizable than prior work.
- Method: “With safety constraints and risk predictions, the proposed approach (to be
elaborated below) is expected to incorporate the strengths of both categories of safe RL techniques.” → by this point, it’s still not clear why this is better. **The authors should make more clear, in the intro and method introduction, that their contrastive method is able to reason about *future risk*, while many current methods either only implicitly do (through reward modifications and TD-learning) or do not at all (binary classification with model-based RL).** This would make the paper's contributions clearer, and make it better in my opinion.
clarity,_quality,_novelty_and_reproducibility: The paper is pretty clear and easy to read.

The paper's objective and Theorem 1 is original/novel.

The paper is of above average quality, but figure 1 should be improved.

The paper seems reproducible from the paper descriptions alone.
summary_of_the_review: I recommend a weak accept. While I still have some problems with this paper, especially regarding how it's framed, I think the paper is much improved over the previous submission that I reviewed and makes valuable contributions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a method of safe reinforcement learning (RL) that learns a classifier for distinguishing between safe state-action pairs from unsafe state-action pairs.  A novelty is in contrastive learning of the classifier to enable stable learning.  The effectiveness of the proposed approach is well supported by the experiments with the safety gym environments, which constitute the major strength of the paper.

The major weakness is in the derivation or the theoretical support of the proposed approach.  While some of the the initial confusions among the reviewers about the definitions and the use of F_\theta are resolved through the discussion with authors and among reviewers, there remains questions and potential improves that should be made around the expositions or definitions of F_\theta or p(y=1|s,a).


*The following is a personal opinion of the area chair:

Consider the behavior policy that eventually reaches an unsafe state almost surely.  The behavior policy considered in Section 5.1 is such a policy ("RL agent reaches the end of the trajectory once it encounters the safety violation").  Then p( y=1 | s, a ) = 1 for any (s, a). Then, F_theta(s,a)=1, Assumption 1 cannot hold, etc.  Despite this, in the feedback from the authors, Assumption 1 is shown to be empirically satisfied to some extent in one environment, and overall the proposed approach is working fine (surprisingly given the inconsistency in the theory). Perhaps p( y=1 | s, a ) is representing something different, but it is unclear what it is.  So, the approach appears to be working, but its derivation or its theoretical support are confusing.  Due to these inconsistencies in theory, it is unclear why and when the proposed approach works.
justification_for_why_not_higher_score: There are inconsistencies in the derivation or theoretical support of the proposed method, and it is unclear why and when the proposed approach works.
Decision: Reject