Safe Reinforcement Learning with Contrastive Risk Prediction
Authors: Hanping Zhang
Yuhong Guo
Venue: Submitted to ICLR 2023 (2023)
Keywords: safe reinforcement learning
contrastive risk prediction

Abstract: As safety violations can lead to severe consequences in real-world applications, the increasing deployment of Reinforcement Learning (RL) in safety-critical domains such as robotics has propelled the study of safe exploration for reinforcement learning (safe RL). In this work, we propose a risk preventive training method for safe RL, which learns a statistical contrastive classifier to predict the probability of a state-action pair leading to unsafe states. Based on the predicted risk probabilities, we can collect risk preventive trajectories and reshape the reward function with risk penalties to induce safe RL policies. We conduct experiments in robotic simulation environments. The results show the proposed approach has comparable performance with the state-of-the-art model-based methods and outperforms conventional model-free safe RL approaches.


Under review as a conference paper at ICLR 2023
SAFE REINFORCEMENT LEARNING WITH CONTRASTIVE RISK PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
As safety violations can lead to severe consequences in real-world applications, the increasing deployment of Reinforcement Learning (RL) in safety-critical do- mains such as robotics has propelled the study of safe exploration for reinforce- ment learning (safe RL). In this work, we propose risk preventive training method for safe RL, which learns statistical contrastive classifier to predict the probability of state-action pair leading to unsafe states. Based on the predicted risk probabilities, we can collect risk preventive trajectories and reshape the re- ward function with risk penalties to induce safe RL policies. We conduct exper- iments in robotic simulation environments. The results show the proposed ap- proach has comparable performance with the state-of-the-art model-based meth- ods and outperforms conventional model-free safe RL approaches.
INTRODUCTION
Reinforcement Learning (RL) offers great set of technical tools for many real-world decision making systems, such as robotics, that require an agent to automatically learn behavior policies through interactions with the environments (Kober et al. 2013). Conversely, the applications of RL in real-world domains also pose important new challenges for RL research. In particular, many real- world robotic environments and tasks, such human-related robotic environments (Brunke al.) 2021). helicopter manipulation (Martín H & Lope |2099 Koopejan & Whiteson, 2011). autonomous vehicle (Wen et al., (2020), and aerial delivery (Faust et al.] (017). have very low tolerance for violations of safety constraints, as such violation can cause severe consequences. This raises substantial demand for safe reinforcement learning techniques.
Safe exploration for RL (safe RL) investigates RL methodologies with critical safety considerations, and has received increased attention from the RL research community. In safe RL, in addition to the reward function (Sutton & Barto 2018), an RL agent often deploys cost function to maximize the discounted cumulative reward while satisfying the cost constraint (Mihatsch Neuneier 2002 Hans et al. 2008 Ma al. 2022). A comprehensive survey of safe RL categorizes the safe RL techniques into two classes: modification of the optimality criterion and modification of the explo- ration process (Gareia & Fernández 2015). For modification of the optimality criterion, previous works mostly focus on the modification of the reward. Many works (Ray et al. 2019 Shen et al., 2022 Tessler et al.] 20188] Hu et al. 2020 Thomas al. 2021 Zhang et al. 2020) pursue such modifications by shaping the reward function with penalizations induced from different forms of cost constraints. For modification of the exploration process, safe RL approaches focus on training RL agents on modified trajectory data. For example, some works deploy backup policies to recover from safety violations to safer trajectory data that satisfy the safety constraint (Thananjeyan et al.,
2021 Bastani et al. 2021 Achiam et al. 2017b.
In this paper, we propose novel risk preventive training (RPT) method to tackle the safe RL prob- lem. The key idea is to learn a contrastive classification model to predict the risk-the probability of a state-action pair leading to unsafe states, which can then be deployed to modify both the explo- ration process and the optimality criterion. In terms of exploration process modification, we collect trajectory data in a risk preventive manner based on the predicted probability of risk. A trajectory is terminated if the next state falls into an unsafe region that has above-threshold risk values. Regarding optimality criterion modification, we reshape the reward function by penalizing it with the predicted risk for each state-action pair. Benefiting from the generalizability of risk prediction, the proposed

---

Under review as a conference paper at ICLR 2023
approach can avoid safety constraint violations much early in the training phase and induce safe RL policies, while previous works focus on backup policy and violate more safety constraints by interacting with the environment in the unsafe regions. We conduct experiments using four robotic simulation environments on MuloCo (Todorov et al. 2012). Our model-free approach produces comparable performance with a state-of-the-art model-based safe RL method SMBPO (Thomas iet al. 2021) and greatly outperforms other model-free safe RL methods. The main contributions of the proposed work can be summarized as follows:
This is the first work that introduces contrastive classifier to perform risk prediction and conduct safe RL exploration.
With risk prediction probabilities, the proposed approach is able perform both explo- ration process modification through risk preventive trajectory collection and optimality cri- terion modification through reward reshaping.
As model-free method, the proposed approach achieves comparable performance to the state-of-the-art model-based safe RL method and outperforms other model-free methods in robotic simulation environments.
RELATED WORKS
Many methods have been developed in the literature for safe RL. Altman (1999) first introduced the Constrained Markov Decision Process (CMDP) to formally define the problem of safe exploration in reinforcement learning. Mihattsh & Neuneier (2002) introduced the definition of risk for safe RL and intended to find a risk-avoiding policy based on risk-sensitive controls, Hans et al. (2008)
further differentiated the states as "safe" and "unsafe" states based on numan-designn an RL agent is considered to be not safe if it reaches "unsafe" states. Garcia & Fernández) (2015) presented comprehensive survey on safe RL, which categorizes safe RL methods into two classes: modification of the optimality criterion and modification of the exploration process.
criteria, while
Modification of the optimality criterion. Since the optimization conventional criterion (long- term cumulative reward) does not ensure the avoidance of safety violations, previous works have studied the modification of the optimality objective, based on different notions risk (Howard & Matheson 1972 Sato al. 2001), probabilities of visiting risky states (Geibel & Wysotzki (005)). etc. |Achiam et al. (2017) proposed a Constrained Policy Optimization (CPO) to update the safe policy by optimizing the primal-dual problem trust regions. Recently, reward shaping (Dorigo & Colombetti 1994 Randløv& & Alstrom 1998) techniques have been brought into the safe exploration RL. et al. (2018) applied the reward shaping technique in safe RL to penalize the normal training policy, which is known as Reward Constrained Policy Optimization (RCPO). Zhhng et al. (2020) developed a reward shaping approach built upon Probabilistic Ensembles with Trajectory Sampling (PETS) (Chua al. 2018) that maximizes the average return of predicted horizons. pre- trains an predictor the unsafe state in an offline sandbox environment and penalizes the reward of PETS during the adaptation in online environments. A similar work in (Thomas al. 2021) reshapes reward functions using a model-based predictor. It regards unsafe states as absorbing states and trains the RI agent with penalized reward to avoid the visited unsafe states.
Modification of the exploration process. Some previous works have attempted to optimize the safe RL policy by interacting with the environment with adjusted exploration processes. For exam- ple, Driessens & Džeroski (2004); Martín H & Lope (2009); Soog et al. (2012) provided guidance to the exploration process based on prior knowledge on the environment. Similarly, Abbeel et al.) (2010): (Tang)[Tang&aa et al. (210)) restricted the exploration process learning based on demonstration data. al.) (202T): |Bastani et al. (202T) focused on using backup policies of the safe regions, aiming to avoid safety violations. If the agent takes potentially dangerous action, the task policy will be replaced with a guaranteed safe backup policy. Ma et al. (2022) proposed a model-based conservative and adaptive penalty approach explore safely by modifying the penalty adaptively in the training process.
More recently, (Thananjeyan
Safe RL is important for application environments with limited cost for trial-and-error, such the human-related robotic environments. where violations of safety concerns may lead to catastrophic failures (Brunke et al. 2021). Todorov al. (202)) developed robotic simulation environment
2

---

Under review as a conference paper at ICLR 2023
named MuJoCo, which promotes the study of the RL applications in robotic environments, Thomas et al. (2021) further modified the MuJoCo environment to define safety violations for robotic simu- lations. Such environments provided suitable test bed for safe RL methods.
3 PRELIMINARY
Reinforcement learning (RL) has been broadly used to train robotic agents by maximizing the dis- counted cumulative rewards. The representation mulated Markov Decision Process (MDP) M (S. AT.,,,RR,,
reinforcement learning problem can be for-
(Sutton & Barto 2018 where S is the state space for all observations, A the action space for available actions, T S x A + S is the transition dynamics, R S x A - [ tion, and 7 E 10, 1) is the discount factor. An agent can start from a random initial state 8o to take actions and interact with the MDP environment by receiving rewards for each action and mov- ing to new states. Such interactions can produce a transition (St, at Tt, s(+1) ) at each time-step t with S;-1 = ((st ar) and Tt c(st, az). while sequence of transitions comprise a trajectory = (So, 4o. ro: 81, 11,1. 1pr1+)), where [r] + 1 denotes the length of trajectory r-i.e., the number transitions. The goal of RL. is to learn an optimal policy n² S -> that can maximize the expected discounted cumulative reward (return): πt* = arg maxx (T( = ETwD, EE/ (rr)]
is the reward func-
3.1 SAFE EXPLORATION FOR REINFORCEMEN LEARNING
Safe exploration for Reinforcement Learning (safe RL) studies RL with critical safety considera- tions. For a safe RL environment, in addition to the reward function, cost function can also exist to reflect the risky status of each exploration step. The process of safe RL can be formulated as a 7, c, d), which introduces extra cost function and cost threshold d into MDP. An exploration trajectory un- der CMDP can be written as T = (Su. ao To; Co $1, Q1 11, C1, ())[(+), where the transition at time-step t is (st: at, St+1, Tt, ct), with cost value o induced from the cost function C = c(s,, at) CMDP monitors the safe exploration process by requiring the cumulative cost Je(T)) does not eeeeeeeepeeeee
Constrained Markov Decision Process (CMDP) (Altman 1999), M (S, A. T, R,
ceed the cost threshold d, where (T(TT) can be defined as the expected total cost of the exploration, Je( E,.D, E'z q]] (Ray et al. 2019). Safe RL hence aims to learn optimal policy n' that can maximize the expected discounted cumulative reward subjecting to a cost constraint, as follows:
Ei s.t. Jo(T) EI-D,
|r] C
arg max (T) BYND,
d.
(I)
4 METHOD
Robot operations typically have low tolerance for risky/unsafe states and actions, since a robot could be severely damaged in real-world environments when the safety constraint being violated. Similar to the work in (Hans et al. 2008), we adopt strict setting in this work for the safety constraint such that any "unsafe" state can cause violation of the safety constraint and the RL agent will terminate an exploration trajectory when encountering an "unsafe" state. We have the following definition:
Definition 1. For a state s and an action a, the value of the cost function c(s, a) can either be or 1. When c(s, a) O, the induced state T(s, a) is defined as safe state; when c(s.a) 1. the induced state Txx, a) is defined as an unsafe state, which triggers the violation of safety constraint and hence causes the termination of the trajectory.
Based on this definition, the cost threshold d in Eq. ( should be set strictly to 0. The agent is expected to learn safe policy π that can operate with successful trajectories containing only safe states. Towards this goal, we propose novel risk prediction method for safe RL. The proposed method deploys a contrastive classifier to predict the probability of a state-action pair leading to un- safe states, which can be trained during the exploration process of RL and generalized to previously unseen states. With risk prediction probabilities, a more informative cumulative cost Je(T) can be formed to prevent unsafe trajectories and reshape the reward in each transition of trajectory to in- duce safe RL policies. Previous safe RL methods in the literature can typically be categories into two classes: modification of the optimality criterion and modification of the exploration process (Garcıa
3

---

Under review as a conference paper at ICLR 2023
Safe
Unsafe
SU
a
p(y=1|s, a)Pn
p(y=1|s, a)sn
Figure 1: An overview the risk preventive process. The blue area denotes the safe region of the trajectory. The dot 8y unsafe state. The red area around the unsafe state Sy denotes the unsafe region, bounded by the value of risk prediction p = l[s, a). The RL agent explores risk preventive trajectories by avoiding entering the unsafe region in the training process.
& Fernández, 2013). With safery constraints and risk predictions, the proposed approach (to be elaborated below) expected incorporate the strengths of both categories of safe RL techniques.
4.1 RISK PREDICTION WITH CONTRASTIVE CLASSIFICATION
Although an RL agent would inevitably encounter unsafe states during the initial stage of the ex- ploration process an unknown environment, we aim to quickly learn from the unsafe experience through statistical learning and generalize the recognition of unsafe trajectories to prevent risk for future exploration. Specifically, we aim to compute the probability of a state-action pair leading to unsafe states, i.e., p(y = 1/ 0e), where E {0, denotes a random variable that indicates whether (5e, dr) leads to an unsafe state 8u E Sy. The set of unsafe states, Su, can either pre- given or collected during initial exploration. However, directly training binary classifier to make such predictions is impractical as is difficult to judge whether a state-action pair is safe-i.e never leading to unsafe states. For this purpose, propose to train contrastive classifier Fø(st, at) with model parameter to discriminate positive state-action pair (st, (z) in a trajectory that leads to unsafe states (unsafe trajectory) and random state-action pair from the overall distribution of any trajectory. Such contrastive form of learning can conveniently avoid the identification problem of absolute negative (safe) state-action pairs.
Let p(st, at ly 1) denote the presence probability of state-action pair (st, 0e) in a trajectory that leads unsafe states, and p(y 1) denote the distribution probability of unsafe trajectory in the environment. The contrastive classifier Fo 0z) is defined as follows:
p(se, aely 1)p(y= =
(8 a.) p(s:, avly = T)l(y = 1) a()
(2)
where ply = 1) is used as weight for the positive samples and weight is given to the contrastively- negative samples. This binary classifier identifies the state-action pairs in unsafe trajectories trastively from general pairs in the overall distribution.
From the definition of Fo(se, az) in Eq. 2). can easily derive the probability of interest, p(y 1 1|8 a),, by Bayes' theorem, as follows:
p 1|8,, 0x) p(s., a,ly 1)p(y= =
p( 44)
1- Fo(se, ax)
Although the normal output range for the probabilistic classifier Folst, az) should be [0, 1], this could lead to unbounded p(u = 1|,,, a,) E 00, xo) through Eq, F. Hence we propose to rescale the output of classifier Fø(se, at) to the range of 0..)
We optimize the contrastive classifier's parameter 0 using maximum likelihood estimation (MLE). The ligelikeli objective function can be written as:
L(0) p(y 1)(Ep((..ab==))lle Fo(st, at)l + (E(s (log(1 - Fe
FF(ss,, ae))]
4.2 RISK PREVENTIVE TRAJECTORY
a trajectory terminates when the RL agent encounters an unsafe state and triggers safety constraint violation. It is however desirable to minimize the number of such safety
Based on Definition M
4

---

Under review as a conference paper at ICLR 2023
violations even during the policy training process and learn good policy in safe regions. The risk prediction classifier we proposed above provides a convenient tool for this purpose by predicting the probability of state-action pair leading to unsafe states, p\y 1|s,,a;) Based on this risk prediction, we have the following definition for unsafe regions:
Definition 2. A state-action pair (st,0) falls into an unsafe region if the probability of (8t, ae) leading to unsafe states is greater than a threshold de ply = 11s,, > 1/, where vj (0,
With this definition, RL agent can pursue risk preventive trajectories avoid safety violations by staying away from unsafe regions. Specifically, we can terminate a trajectory before violating the safety constraint by judging the risk probability of p'y = 1| az). The process is illustrated in Figure T
Without doubt, the threshold 1) is key for determining the length T = |+| of early stopped risk preventive trajectory f. To approximate a derivable relation between and T, we make the following assumption and lemma:
{5o, dg, To: Co. $1. SHH that leads to an unsafe state sH Su, the risk prediction probability p(y = 1/s 0z) increases linearly along the time steps.
Assumption 1. For a trajectory
Lemma 1. Assume that Assumption 7 holds. Let H E N denote the length of an unsafe trajectory {Su, ag. To, Co, $1, 01, TT, "1.......... { that terminates at an unsafe state Sn E Sp. The number of transition steps, T, along this trajectory the unsafe region determined by n in Definition 2 can
be approximated as: T p- H. where po = p(y = 1|50, an).
Proof. According to assumption [ the probability for (s,, a,) leading unsafe states p(y: 1 St, at) increases linearly along time-steps. Let Pa denote the probability of starting from the initial state spz Po p(v. 1|sa ag). For probability threshold 7) (0, for unsafe region identifi- cation in Definition 2. the ratio between the number of environment transition steps T to the unsafe region and the unsafe trajectory length H will be approximately (due to integer requirements over T') equal to the ratio between the probability differences of r Po and 1 Po. That is, T Po.
Hence can be approximated as: T
Po
This Lemma clearly indicates that a larger value will allow more effective explorations with longer trajectories, but also tighten the unsafe region and increase the possibility of safety violiations.
4.3 RISK PREVENTIVE REWARD SHAPING
With Definition [ the safe RL formulation in Eq. 11 can hardly induce a safe policy since there are intermediate costs before encountering unsafe state. With the risk prediction classifier proposed above, we can rectify this drawback by defining the cumulative cost func- tion Je(T) using the risk prediction probabilities, p(y 1|s,, at), over all encountered state- action pairs. Specifically, we adopt a reward-like discounted cumulative cost as follows: Je(T) =
E,-D, El = 1|8,, a:)]. which uses the predicted risk at each time-step as the estimate cost. Moreover, instead of solving safe RL as a constrained discounted cumulative reward maxi- mization problem, we propose using Lagrangian relaxation (Bertsekas) 1997) to convert the con- strained maximization CMDP problem in Eq. (1) to an unconstrained optimization problem, which is equivalent to shaping the reward function with risk penalties:
min max
(J)(T)) X(J((T) d))
(5)
X
min max X
[Jr(T) - XJe(T)]
(6)
|r| min max BAND. |IIhh 7 Xp = 1|s,, xp(y=110,,, 1/2, X20
The Lagrangian dual variable À controls the degree of reward shaping with the predicted risk value.
Theorem 1. To prevent the RL agent falling into known unsafe states. the penalty factor (i.e., the dual variable) X for the shaped reward it rt Xp(y: 11st, 0t) should have the following
5

---

Under review as a conference paper at ICLR 2023
lower bound, where H. and n are same in Lemma
7
(1-7) (1-7"))
(T Tmin)
A>
m-m H
1- H
1/7
Proof. For trajectory with length H that leads to unsafe state Sи Sv. the largest penalized return the unsafe trajectory should be smaller than the lowest possible unpenalized return from any safe trajectory with the same length:
70(r (' saac
H-1 Σ (= 7'r
sup
< inf
(9)
where - ppy = 1ss, Gt). With this requirement, the RL agent can learn a policy to explore safe states and prevent the agent from entering unsafe states Sy through unsafe regions. As the reward function is bounded within [Tmin, mmazls we can further simplify the inequality in Eq. (9) by seeking the supremum of its left-hand side (L.HS) and the infimum of its right-hand side (RHS).
we can split an unsafe trajectory T with 2 and Lemma sub-trajectory within the safe region, T length H into two sub-trajectories:
Based on Definition
(so, 00, TO, Co,".' -1, 1T- 1,, 1,,)) where Pr. < and sub-trajectory 72 (ST,UT; (T; IT, CT, ... 88) after entering the unsafe region determined by Pt > 1). With Assump-
tion I. we have Pt > 7 for all state-action pairs in the sub-trajectory Tg. Then the LHS and RHS of Eq. (9) can be upper bounded and lower bounded respectively as follows:
LHS S
Σ
t= ? Tmma
H-1
2-1 - 7'X-0+
Σ E 7m)))
(10)
RHS N Σ t=
Tmin.
(11)
To ensure the satisfaction of the requirement in Eq.
we then enforce the follows:
Σ 1= Tmmax 72.0 -
H-1
E E }((n)
E Tmmn
11
(12)
7 Umaz
An Am-7I(L - (1 (1 Mmin
(13)
1-7
1
7 ) Tmin)
(14)
m7 (1 -T
With Assumption T and Lemma we can estimate the length T for the safe sub-trajectory as T P HJ. With this estimation, the lower bound for in Eq. can be derived.
4.4 RISK PREVENTIVE TRAINING ALGORITHM
Our overall risk preventive RL training procedure is presented in Algorithm I which simultane- ously trains the risk prediction classifier and performs reinforcement learning with risk preventive trajectory exploration (line 12-13) and risk preventive reward shaping (line 10).
5
EXPERIMENT
5.1
EXPERIMENTAL SETTINGS
Following the experimental setting in (Thomas al.) 2021 we adopted four robotics simulation environments, Ant, Cheetah, Hopper, and Humanoid, using the MuJoCo simulator (Todorov et 2012). For Ant and Hopper, a robot violates the safety constraint when it falls over. For Cheetah, robot violates the safety constraint when its head flips the ground, which is modified from the HalfCheetah environment with extra safety constraint (Thomas et al.) 2021). For Humanoid, the human-lit like robot violates the safety constraint when the head of the robot falls to the ground. The RL agent reaches the end of the trajectory once it encounters the safety violation.
Experimental Environments

---

Under review as a conference paper at ICLR 2023
Algorithm Risk Preventive Training
Input Initial policy *g. classifier Fe, trajectory set = 0. set of unsafe state-action pairs Su,
threshold 7) penalty factor X, set of unsafe trajectory length H. 0
Output Trained policy Te
1:
for k == 1,2 K do
2:
for t = 1,2, ..., Tmax do
3:
Sample transition (st, Tt, Ct, St41) from the environment with policy Tt.
4:
if c > 0 then
5:
Add risky state-action (st, az) into the unsafe state set Sy.
6:
Add length to H. Increase À if the lower bound increases with = L and Eq. Stop trajectory and break.
7:
8:
Sample next action ar+1 based on policy T and next state S1+17 a++1 (-(1))441)
9:
Calculate p: and Pt+1 using Eq, 3
Penalize reward r, with p:: r Tt Xp;
11:
Add transition to the trajectory set: D D U (st, at S1+1) if Pi+1 17 then
13:
Stop trajectory and break.
14:
Sample risky state-action pairs from Su
15:
Sample transitions from history: (st, dz, fr, S141)) ~ D
16:
Update classifier Fag by maximizing the likelihood (()) in Eq
17:
Update policy T with the shaped rewards J+(TT in Eq
We compare our proposed Risk Preventive Training (RPT) approach with three state-of-the-ar safe RL methods: SMBPO (Thomas et al. 2021), RCPO TTssler et al. 2018), and LR (Ray et al. 2019).
Comparison Methods
Safe Model-Based Policy Optimization (SMBPO): This is model-based method that uses an ensemble of Gaussian dynamics based transition models. Based on the transition models, it penalizes unsafe trajectories and avoid unsafe states under certain assumptions.
Reward Constrained Policy Optimization (RCPO): This is policy gradient method based on penalized reward under safety constraints.
Lagrangian relaxation (LR): It uses Lagrangian relaxation for the safety constrained RL.
In addition, we have also further extended the proposed RPT with simple data augmentation, pro- ducing data augmented method, RPT+DA, for comparison. Specifically, we perform data aug- mentation only for the data sampled from the set of risky states Sy in line 14 of Algorithm [] The MuJoCo environments are built upon robotic simulations, and they take parameters of different parts of the robot as state observations, including coordinates, orientations, angles, velocities, and angular velocities. Each element of the state observation falls into the range of (-oc, oc). Hence, for each sampled risky state-action pair (St, 44), we propose to produce an augmented state & by adding a random Gaussian noise sampled from the standard normal distribution (N 1) to each entry of the observed data We can repeat this process to generate multiple (e.g., n) augmented states for each 8t. In our experiments, we used n 3. Together with a,, each & can be used to form an additional risky state-action pair (3,, ar) for training the contrastive classifier. The hypothesis is that without any prior information about the environment, the training of the proposed contrastive classifier highly depends the data collected during the agent's 's interactions with the environment, especially the limited number of observed unsafe states. By using data augmentation, we expect to improve the sample efficiency of the approach while reducing the possible safety violations during the exploration process.
Implementation Details Implementations for LR and RCPO algorithms are adapted from the recovery RL paper (Thananjeyan et al.) 2021). For fair comparisons, following the original setting of LR and RCPO, we disable the recovery policy of the recovery RL framework, which collects offline data to pretrain the agent. For MBPO, we adopted the original implementation from the MBPO paper (Thomas et al. (02T). Both SMBPO and RCPO are built on top of the Soft-Actor- Critic (SAC) RL. method (Harmoja et al. 2018). Hence in the experiments also implemented

---

Under review as a conference paper at ICLR 2023
Leamingg Return Violations
Ratlo between Naturn and Videeiom
0.002
0.000
RCPO
SHIPP
-0.002
For each method, each plot presents the undiscounted return the total number of violations. Results are collected for three runs with random seeds. The curve shows the mean of the return over the three runs, while the shadow shows the standard deviation. Right: The plot reports the ratio of the normalized return over the number of violations vs. time-steps. The curve for each method is calculated by averaging across all four environments (Ant, Cheetah, Hopper, Humanoid).
Figure 2: Left:
the proposed RPT approach (also RPT+DA) on top of SAC, although RPT general safe RL methodology. We used = 0.9 for risk preventive trajectory exploration, and collected the mean episode return and violation for 10 time-steps.
5.2 EXPERIMENTAL RESULTS
We compared all the five methods by running each method three times in each of the four MuJoCo environments. The performance of each method is evaluated by presenting the corresponding return vs. the total number of violations obtained the training process. The results for all the methods (LR, RCPO, SMBPO, RPT. and RPT+DA) are presented on the left side of Figure 2 one plot for each robotic simulation environment. The curve for each method shows the learning ability of the RL. agent with limited safety violations. From the plots, we can see RPT, RPT+DA and SMBPO achieve large returns with a small number of violations on all the four robotic tasks, and largely outperform the other two methods, RCPO and LR, which have much smaller returns even with large numbers of safety violations. The proposed model-free RPT produces slightly inferior performance than model- based SMBPO on Ant and Cheetah, where method requires more examples of unsafe states to yield good performance at the initial training stage. Nevertheless, RPT outperforms SMBPO on both Hopper and Humanoid with smaller number of safety violations. As model-free safe RL method, RPT produces an overall comparable performance with the model-based method SMBPO. With data augmentation, RPT+DA further improves the performance of RPT all the four environments.
To provide a direct and illustrative comparison about the overall performance of the comparison methods across all the four environments, we need take the average of the results on all four environments using a measure that is independent of any specific environment's return scale. We propose to use the measure that divides the ratio between the return and the number of violations by the maximum trajectory return- -i... the ratio between the trajectory normalized return and the number of violations, and report the average performance of each method under this measure across all four environments. The results are presented on the right plot of Figure We can see that RPT produces comparable or even better performance on certain region time-steps than SMBPO, and greatly outperforms RCPO and LR. With data augmentation, RPT+DA produces notable improve- ments over RPT, especially in the early training phase. The average result curve RPT+DA is in general above the result curve of the model-based method RCPO across the considered environment time-steps. All these results demonstrate that the proposed RPT approach and its augmented version produce the state-of-the- the- performance for safe RL robotic environments.
8

---

Under review as a conference paper at ICLR 2023
2305
2500
2000
1506
1000
-00
Taral aid amor
Figure 3: The plot shows the undiscounted return vs. total number of violations, where the purple curve represents RPT without risk preventive reward shaping (RPT-W/-RSS and the brown curve represents RPT without risk preventive trajectory exploration (RPT-w/-EXP), Results are collected by the average of three runs. The curve shows mean of the return for the runs, while shadow shows the standard deviation.
5.3 ABLATION STUDY
We conducted an ablation study to examine the effectiveness of two main components of the pro- posed RPT approach, the exploration modification component- Risk Preventive Trajectory Explo- ration and the reward modification component-RRin Preventive Reward Shaping, on two environ- ments, Ant and Hopper. Specifically, we additionally evaluate the performance of two RPT variants: (1) RPT-w/ORRS, which drops the reward shaping from the full approach RPT by setting À = O; and (2) RPT-w/O-EXP, which drops the risk preventive trajectory exploration component from RPT by removing the risk-prediction based trajectory stopping step line 12-13 of the training Algorithm I Hence, the risk prevention function of RPT-w/O-RS will totally depend on the risk preventive trajec- tory exploration component, and the risk prevention function of RPT-w/o-EXP will depend on the risk preventive reward shaping component.
We compared these two variants with the full approach RPT and two other model-free safe RI. methods, LR and RCPO. The comparison results are reported in Figure 3 We can see that by drop- ping one component, both RPT-w/o-RS (purple curve) and the RPT-w/OEXX (brown curve) have dramatic performance degradation comparing with the full approach RPT. This verifies the nontriv- ial contribution of both components (Risk Preventive Trajectory Exploration and Risk Preventive Reward Shaping) to the proposed approach RPT for safe RL. Meanwhile, the variant RPT-w/o-RS largely outperforms RPT-w/o-EXP and the other two base safe RL methods, LR and RCPO, in terms of the average results, but also demonstrates very large variations over the results of multiple runs, which are particularly notable in the Ant environment. This suggests though the risk preventive tra- jectory exploration component is very useful for safe RL, it is better deployed under controlled exploration policy under the risk penalized reward. The variant RPT-w/o-EXP produces similar per- formance as RCPO in the Ant environment, and only slightly better performance than LR on both Ant and Hopper. The results suggest that without the guidance of safe exploration, the effective- ness of reward shaping itself is relatively limited for preventing risks. Overall, the results of this ablation study validate the design of the proposed RPT approach in integrating the strengths of both exploration process modification and reward shaping for safe RL.
CONCLUSION
Inspired by the increasing demands for safe exploration of Reinforcement Learning, we proposed a novel mode-free risk preventive training method to perform safe RL by learning a statistical con- trastive classifier to predict the probability of state-action pair leading to unsafe states. Based on risk prediction, we can collect risk preventive trajectories that terminate early without triggering safety constraint violations. Moreover, the predicted risk probabilities are also used as penalties to perform reward shaping for learning safe RL policies, with the goal of maximizing the expected return while minimizing the number of safety violations. We compared the proposed approach with few sfate-of-the-ar safe RL methods using four robotic simulation environments. The proposed approach demonstrates comparable performance with the state-of- ahr model-basee method and outperforms the model-free safe RI. methods.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Pieter Abbeel, Adam Coates, and Andrew Y Ng. Autonomous helicopter aerobatics through ap- prenticeship learning. The International Journal of Robotics Research, 29(13):1608-15 2010.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning (ICML), 2017.
Eitan Altman. Constrained Markov decision processes: stochastic modeling. Routledge, 1999.
Osbert Bastani, Shuo Li, and Anton Xu. Safe reinforcement learning via statistical model predictive shielding, In Robotics: Science and Systems, 2021.
Dimiuri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3):
334-334, 1997.
Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela Schoellig. Safe learning in robotics: From learning-basee control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5, 2021.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems (NeurIPS), 2018.
Marco Dorigo and Marco Colombetti. Robot shaping: Developing autonomous agents through learning. Artificial intelligence, 71(2):321-370, 1994.
Kurt Driessens and Saão Džeroski. Integrating guidance into relational reinforcement learning. Machine Learning. 57(3):271-304 2004.
Aleksandra Faust, Ivana Palunko, Patricio Cruz, Rafael Fierro, and Lydia Tapia. Automated aerial suspended cargo delivery through reinforcement learning. Artificial Intelligence, 247:381-398, 2017.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research (JMLR), 16(1):1437-1480, 2015.
Peter Geibel and Fritz Wysotzki. Risk-sensitive reinforcement learning applied control under constraints. Journal of Artificial Intelligence Research, 24:81-108, 2005.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning (ICMI), 2018
Alexander Hans, Daniel SchneegaB, Anton Maximilian Schäfer, and Steffen Udluft. Safe explo- ration for reinforcement learning. In ESANN, 2008
Ronald A Howard and James E Matheson. Risk-sensitive markov decision processes. Management science, 18(7):356-36,, 1972.
Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A approach of reward shaping. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in roboties: A survey. The International Journal of Robotics Research, 32(11): 2013.
Rogier Koppejan and Shimon Whiteson. Neuroevolutionary reinforcement learning for generalized control of simulated helicopters. Evolutionary intelligence, 4(4):219-241, 201
Yecheng Jason Ma, Andrew Shen, Osbert Bastani, and Jayaraman Dinesh. Conservative and adap- tive penalty for model- based safe reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), 2022.
10

---

Under review as a conference paper at ICLR 2023
José Antonio Martin H and Javier de Lope. Learning autonomous helicopter flight with evolutionary reinforcement learning. In International Conference on Computer Aided Systems Theory, pp. 75- 82. Springer, 2009.
Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning, 49 (2):267-290, 2002.
Jette Randløv and Preben Alstrom. Learning to drive a bicycle using reinforcement learning and shaping. In International Conference on Machine Learning (ICML). 1998.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01700, 7:1, 2019.
Makoto Sato, Hajime Kimura, and Shibenobu Kobayashi. Td algorithm for the variance of return and mean-variance reinforcement learning. Transactions of the Japanese Society for Artificial Intelligence, 36(3):353-362 2001.
Li Shen, Long Yang. Shixiang Chen, Yuan, Xueqian Wang, Dacheng Tao, et Penalized proximal policy optimization for safe reinforcement learning. arXiv preprint arXiv:2205. 1814, 2022.
Yong Song, Yi-bin Li, Cai-hong Li, and Gui-fang Zhang. An efficient initialization approach of q-learning for mobile robots. International Journal of Control, Automation and Systems, 10(1): 166-172, 2012.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Jie Tang, Arjun Singh, Nimbus Goehausen, and Pieter Abbeel. Parameterized maneuver learning for autonomous helicopter flight. IEEE international conference on robotics and automation (ICRA), 2010.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint / /074, 2018.
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, (3):49154492) 2021.
Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforcement learning by imagining the near future. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In JEEE/RSJ international conference on intelligent robots and syslems, pp. 5026-5033. IEEE, 2012.
Lu Wen, Jingliang Duan, Shengbo Eben Li, Shaobing Xu, and Huei Peng. Safe reinforcement learning for autonomous vehicles through parallel constrained policy optimization. In IEEE In- ternational Conference on Intelligent Transportation Systems (ITSC), pp. 1-7. IEEE, 2020.
Jesse Zhang, Brian Cheung, Chelsea Finn, Sergey Levine, and Dinesh Jayaraman. Cautious adapta- tion for reinforcement learning in safety-critical settings. In International Conference on Machine Learning (ICML), 2020.
11


Reviews:
[{"review_id": "rHCAZCm4-_", "reviewer": "Reviewer_mSBr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a contrastive-learning inspired objective for shaping the reward function for RL agents to prevent encountering predicted, future “unsafe” states while training. They validate their claims on standard Mujoco environments.", "summary_of_the_review": "I recommend a weak accept. While I still have some problems with this paper, especially regarding how it's framed, I think the paper is much improved over the previous submission that I reviewed and makes valuable contributions.", "strength_and_weaknesses": "For full disclosure, I have already reviewed a previously submitted (and rejected) version of this paper. My review is based on criticisms/strengths from the old review and old meta-review, along with the changes that have been made to the paper in this new submission. Overall, this paper is much improved from the original version I reviewed.\n\n## Strengths\n\n**Clarity:** The paper is generally clear and readable.\n\n**Mathematical Rigorousness:** \n\n- The contrastive classifier $F_\\theta$’s intuition isn’t described very well, but other than that Section 4.1 describing the objective is intuitive and grounded\n- The proof of Theorem 1 is well-presented and presents a mostly-grounded way of selecting the dual variable $\\lambda$ rather than just treating it as a hyperparameter like in other works.\n\n**Experiments/Results:**\n\n- Experiments and ablations are sensible. The results presented hold up the claim that this works as well as SMBPO while being a model-free method, which is valuable due to fewer assumptions made by model-free algorithms about the environment.\n\n## Weaknesses\n\n**Clarity:** \n\n- Figure 1 is not very useful for demonstrating how the method works. There’s no indication of some risk prediction threshold ($\\eta$) nor anything to indicate that actions are actually modified based on this threshold ($a_t, a_t’$ aren’t differentiated).\n- The contrastive classifier $F_\\theta$’s intuition isn’t described very well, and Eq. 2 just appears without explaining the terms. I understand that this derivation of $F_\\theta$ makes Eq 3 convenient (and maybe the authors figured out Eq 3 before figuring out Eq 2), but this could be made more clear.\n\n**Experiments:** \n\n- Assumption 1 is presented without empirical evidence it exists. Can the authors experiment and demonstrate how true Assumption 1 is?\n- It’s hard to say that RPT+DA is better, as the authors claim, due to the overlapping standard deviations.\n\n**Method and Framing:** \n\n- Intro: “benefitting from the generalizability of risk prediction, the proposed approach can avoid safety violations much earlier in the training process” → It’s not clear intuitively how this risk prediction differs from the aforementioned prior work at this point in the intro, and therefore how it is more generalizable than prior work.\n- Method: “With safety constraints and risk predictions, the proposed approach (to be\nelaborated below) is expected to incorporate the strengths of both categories of safe RL techniques.” → by this point, it’s still not clear why this is better. **The authors should make more clear, in the intro and method introduction, that their contrastive method is able to reason about *future risk*, while many current methods either only implicitly do (through reward modifications and TD-learning) or do not at all (binary classification with model-based RL).** This would make the paper's contributions clearer, and make it better in my opinion.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ja2h0ASNOB2", "reviewer": "Reviewer_aGao", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies safety in reinforcement learning. The goal is to maximize return while keeping the cost of visiting unsafe state-action pairs below a threshold. The paper proposes learning a classifier $F_\\theta(s, a)$ that predicts whether a given state action $(s, a)$ will lead to eventually visiting an unsafe state-action pair. This is then used to update the policy and the $F_\\theta$ is then relearned on the fly.  ", "summary_of_the_review": "Because of concerns with methodology, writing, and technical statements, I am recommending rejection. My score can change greatly if the authors clarify any misunderstanding. In particular, I would like to know if technical statements concerning the relation between the Bayes' classifier $p(y=1 \\mid s, a)$ and the learned classifier $F_\\theta$ is true. ", "strength_and_weaknesses": "Strength:\n\n1. Paper is on safe RL which is an important topic\n\nWeakness:\n\nThis paper suffers from three concerns.\n\n1. *Broad Concerns:* The paper makes assumptions that will not apply to applications that the paper uses for motivation in the beginning. To begin with, the paper assumes a deterministic setting that holds in very simple cases. Secondly, a real robotic experiment can immediately fail if the robot hits a wall and breaks down. Therefore, encountering unsafe states even once may not even be an option in such cases. Lastly, $p(y = 1\\mid, s, a)$ is a non-stationary distribution that evolves as the policy evolves. This seems undesirable from a learning point of view, as the model has to constantly adjust to the changes in the policy. As the policy optimizes for state-action pairs that are safe/unsafe based on value of $p(y=1 \\mid s, a)$ and not directly the value of $c$, therefore, the policy may start taking certain actions or avoid actions, in a given state, as the training progresses. A more desirable strategy would be that $(s, a)$ pairs that are marked as unsafe are never revisited. \n\n2. *Technical Concerns:* Several statements in this paper seem either wrong or unjustified to me. It is my understanding that the classifier $F_\\theta(s, a)$ is trained on random $(s, a)$ pairs that are treated as safe $(y=0)$, and a set of unsafe pairs $(y=1)$ that were selected during the previous exploration. If yes, then the Bayes classifier for this problem will be given by $p(y=1 \\mid s, a)$ where\n\n$p(y=1 \\mid s, a) = \\frac{p(s, a \\mid y=1) p(y=1)}{p(s, a)} = \\frac{p(s, a \\mid y=1) p(y=1)}{ p(s, a \\mid y=1) p(y=1) + p(s, a \\mid y=0) p(y=0)}$.\n\nHere $p(s, a \\mid y=0)$ denotes the probability over random state-action pairs in the dataset, as that is the noise distribution for contrastive learning. Now if $F_\\theta$ is an expressive class that contains $p(y=1 \\mid s, a)$ then we have asymptotically $F_\\theta(s, a) \\rightarrow p(y = 1 \\mid s, a)$ on $(s, a)$ pairs that lies in support of the distribution on which it was trained.  \n\nThe paper equates $p(y =1 \\mid s, a) = \\frac{F_\\theta(s, a)}{1 - F_\\theta(s, a)}$ which does not make sense to me.  Note that in the above version, no correction is required and all probability values are in [0, 1]. \n\nBecause of this error, or confusion, I am unable to fully check Lemma 1 and Theorem 1. \n\nIn another example, assumption 1 seems unclear to me. Does the increase have to be strict? If not, this already rules out deterministic policies since a deterministic policy in a deterministic setting (which is what this paper assumes) will result in a straight path and if this policy visits an unsafe trajectory, then $p(y=1 \\mid s, a) = 1$ for all state-action pairs along the way.  This assumption needs to be made clearer to understand it clearly, and an example can help. \n\n3. *Writing Concerns:* The paper is not well-written. I give numerous examples below in the clarity section.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "beKD-ozaYqb", "reviewer": "Reviewer_jp1h", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors of this study provide a risk-preventive training strategy for safe RL that trains a statistical contrastive classifier to forecast the likelihood that a state action combination would result in unsafe states. They can gather risk prevention trajectories and remodel the reward function with risk penalties based on the expected risk probability to promote safe RL policies. In robotic simulation environments, the authors run experiments. The outcomes demonstrate that the suggested methodology outperforms traditional model-free safe RL approaches and is on par with cutting-edge model-based methods in terms of performance.", "summary_of_the_review": "While the authors appear clear in their claims and methodology, I don't believe the paper can be accepted to a venue like ICLR. My main concerns with this study are:\n1) The simple binary assumption used to define the cost function $c(s, a)$ and unjustified claims.\n2) The credibility of the experiments:\n- An insufficient number of seeds.\n- Insufficient environments.\n- Reported returns are training rewards, as far as I understood.\n3) Reproducibility as no code or any anonymous repository, which does not reveal the authors' identity, of course, is provided.", "strength_and_weaknesses": "**Strengths:**\n- The motivation is clear and promising.\n- The prior art is discussed concisely.\n\n**Weaknesses:**\n- I couldn't understand some statements in the paper as they seem unjustified. For instance, the classifier $F_{\\theta}$ is trained on randomly selected state-action pairs that are treated as \"safe\" and some pairs that are \"unsafe\", which are selected in the previous exploration. According to this, the Bayes classifier for such a problem is given by $p(y= 1|s, a)$, which is defined as:\n$p(y = 1|s, a) = \\frac{p(s, a|y=1)p(y=1)}{p(s, a)} = \\frac{p(s,a|y=1)p(y=1)}{p(s, a|y=1)p(y=1) + p(s,a|y=0)p(y=0)}$,\n\nwhere $p(s,a|y=0)$ is the probability over random state-action pairs in the dataset, corresponding to the noise distribution for contrastive learning. If we consider that $F_{\\theta}$ is an expressive class containing $p(y=1|s,a)$, then we asymptotically have $F_{\\theta} \\rightarrow p(y=1|s,a)$ on $(s,a)$ pairs that lie in the distribution on which the classifier is trained. The paper, however, equates $p(y=1|s,a)=\\frac{F_{\\theta}(s,a)}{1 - F_{\\theta}(s,a)}$, which is left unjustified and I couldn't understand it. Due to such an unjustified claim, Lemma 1 and Theorem 1 become inconsistent to me.\n\n- Why didn't the authors consider giving a continuous \"safeness\" value between 0 and 1? In my opinion, using a binary representation for the safety of a state-action pair is taking the easy way out. Considering a continuous value, on the other hand, would increase the novelty.\n- The comparative evaluations are insufficient as the authors only consider Ant, HalfCheetah, Humanoid, and  Hopper. While these can be considered challenging tasks, I expect the results for BipedalWalker (from BOX2D), Walker2d, and Swimmer. Although the latter has a low state-action dimensionality, it is a very challenging task in which many SOTA methods exhibit poor performance. \n- Although it is indicated for the HalfCheetah environment that the robot violates the safety constraint when it falls over, I don't believe it is a credible judgment. Because the Cheetah environment is \"stable\", a trajectory doesn't terminate unless a pre-specified number of time steps is reached, i.e., 1000. While it'd good to examine the proposed method's performance in a stable environment, the authors don't provide such a discussion. \n- The authors should assess a method's performance on a distinct evaluation environment per a pre-specified number of time steps, as it is a common practice in deep RL literature. For example, every 1000 steps, the learning is stopped, and the agent is evaluated in a distinct environment where no exploration or learning is performed for 5-10 episodes. However, the authors used the training rewards as far as I understood. \n- The number of random seeds used to produce the reported results is highly insufficient. At least ten random seeds should be used [per the deep RL benchmarking standards](https://ojs.aaai.org/index.php/AAAI/article/view/11694).\n\n**Detailed Comments:**\n- An RL agent should be \"an RL..\" not \"a RL\".\n- Replace the reward function in the MDP definition with $\\mathcal{R}$ instead of $r$ as it can confuse readers.\n- Punctuation should be put after each equation as they are used in conjunction with the text.\n- It looks ugly when a reference is used, such as: \"Similar to (Hans et al., 2008)...\". Either refer to the natbib package or use it as: \"Similar to the work (Hans et al., 2008)...\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject