Learning Safe Policies with Cost-sensitive Advantage Estimation
Authors: Bingyi Kang
Shie Mannor
Jiashi Feng
Venue:  (2021)
Keywords: Safe Reinforcement Learning

Abstract: Reinforcement Learning (RL) with safety guarantee is critical for agents performing tasks in risky environments. Recent safe RL algorithms, developed based on Constrained Markov Decision Process (CMDP), mostly take the safety requirement as additional constraints when learning to maximize the return. However, they usually make unnecessary compromises in return for safety and only learn sub-optimal policies, due to the inability of differentiating safe and unsafe state-actions with high rewards. To address this, we propose Cost-sensitive Advantage Estimation (CSAE), which is simple to deploy for policy optimization and effective for guiding the agents to avoid unsafe state-actions by penalizing their advantage value properly. Moreover, for stronger safety guarantees, we develop a Worst-case Constrained Markov Decision Process (WCMDP) method to augment CMDP by constraining the worst-case safety cost instead of the average one. With CSAE and WCMDP, we develop new safe RL algorithms with theoretical justifications on their benefits for safety and performance of the obtained policies. Extensive experiments clearly demonstrate the superiority of our algorithms in learning safer and better agents under multiple settings. 

Under review as a conference paper at ICLR 2021
LEARNING SAFE POLICIES WITH COST-SENSITIVE ADVANTAGE ESTIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement Learning (RL) with safety guarantee is critical for agents perform- ing tasks in risky environments. Recent safe RL algorithms, developed based on Constrained Markov Decision Process (CMDP), mostly take the safety require- ment as additional constraints when learning to maximize the return. However, they usually make unnecessary compromises in return for safety and only learn sub-optimal policies, due to the inability of differentiating safe and unsafe state- actions with high rewards. To address this, we propose Cost-sensitive Advantage Estimation (CSAE), which is simple to deploy for policy optimization and ef- fective for guiding the agents to avoid unsafe state-actions by penalizing their advantage value properly. Moreover, for stronger safety guarantees, we develop a Worst-case Constrained Markov Decision Process (WCMDP) method to augment CMDP by constraining the worst-case safety cost instead of the average one. With CSAE and WCMDP, we develop new safe RL algorithms with theoretical justi- fications on their benefits for safety and performance of the obtained policies. Extensive experiments clearly demonstrate the superiority of our algorithms learning safer and better agents under multiple settings.
INTRODUCTION
In recent years, Reinforcement Learning (RL) has achieved remarkable success in learning skillful AI agents in various applications ranging from robot locomotion (Schulman et al., 2015a; Duan et al., 2016; Schulman et al., 2015c), video games (Mnih et al., 2015) and the game of Go (Silver al., 2016; 2017). These agents are either trained in simulation or risk-free environments, and the deployed RL algorithms can focus maximizing the cumulative return by exploring the envi- ronment arbitrarily. However, this is barely workable for real-world RL problems where the safety of the agent is important. For example, a navigating robot cannot take the action of crashing into front obstacle even if the potential return on reaching the target faster is higher. Actually, in reality, some states or actions might be unsafe and harmful to the system, and the agent should learn avoid them in deployment when performing certain tasks. Conventional RL algorithms do not particularly consider such safety-constrained environments, which limits their practical application.
Recently, Safe Reinforcement Learning (Garcıa & Fernánde7, 2015; Mihatsch & Neuneier, 2002; Altman, 1999) has been proposed and drawn increasing attention. Existing safe RL algorithms generally fall into two categories based on whether or not the agents are required to always stay safe during learning and exploration. The algorithms with exploration safety (Dalal al., 2018; Pecka & Svoboda, 2014) insist that safety constraints never be violated even during learning, and thus they usually require certain prior knowledge of the environment to be available, egg., in the form of human demonstrations. Comparatively, deployment safety (Achiam et al., 2017; Chow et al., 2018) RI. algorithms train the agents from interaction with the environment and allow safety constraints violation during learning some extent. This is reasonable since whether state is safe will not be clear until the agent visits that state. Since human demonstrations are too difficult or expensive to collect in some cases and may not cover the whole state space, we foeus deployment safety in this work.
RL problems with deployment safety are typically formulated Constrained Markov Decision Pro- cess (CMDP) (Altman, 1999) that extends MDP by requiring the agent to satisfy cumulative cost constraints in expectation in the meanwhile of maximizing the expected return. Leveraging the

---

Under review as a conference paper at ICLR 2021
success of recent deep learning powered policy optimization methods (Schulman et al., 2015b), Constrained Policy Optimization (CPO) (Achiam et al., 2017) makes the first attempt on high- dimensional control tasks in continuous CMDPs. However, CPO only considers the total cost of a trajectory of a sequence of state-action pairs during policy optimization. It does not differenti- ate the safe state-action pairs from the unsafe ones in the trajectories. Due such incapability of exploiting the intrinsic structure of environments and trajectories, CPO sacrifices too much on the expected return for learning the safety policy.
In this work, we propose Cost-sensitive Advantage Estimation (CSAE) which generalizes the con- ventional advantage estimation for safe RL problems by differentiating safe and unsafe states, based on the cost information returned by the environment during training. CSAE depresses the advan- tage value of unsafe state-action pairs but controls effects upon their adjacent safe state-actions in the trajectories. Thus, the learned policy can maximally gain rewards from the safe states. Based on CSAE, we develop a new safe RL. algorithm with proved monotonic policy performance im- provement in terms of both safety and return from safe states, showing superiority over other safe RL algorithms. Moreover, to further enhance the agent's 's ability of enforcing safety constraints, we propose Worst-case Constrained Markov Decision Process (WCMDP), constraining the cumulative cost in worst cases through the Conditional Value-at-Rist. (Tamar et al., 2015), instead of that in expectation. This augmentation makes the learned policy not only safer but also better, both experimentally and theoretically.
extension of CMDP by
With CSAE and WCMDP, we develop new safe RL algorithm by relating them to trust region methods. We conduct extensive experiments to evaluate our algorithm on several constrained robot locomotion tasks based on Mujoco (Todorov al., 2012), and compare with well-established baselines. The results demonstrate that the agent trained by our algorithm can collect a higher reward, while satisfying the safety constraints with less cost.
2 RELATED WORK
Safe Reinforcement Learning has drawn growing attention. There are various definitions of 'safety' in RL (Garcia & Fernández, 2015; Pecka Svoboda, 2014), e.g., the variance of return (Heger, 1994; Gaskett, 2003), fatal transitions (Hans et al., 2008) and unknown states (Garcıa et al., 2013). In this paper, we focus on the RL problems with trajectory-basee safety cost, under the constrained MDP (CMDP) framework. Through Lagrangian method, Geibel & Wysotzki (2005) propose to convert CMDP into an unconstrained problem to maximize the expected return with a cost penalty. Though such problem can be casily solved with well-designed RL algorithms, e.g. (Schulman et al., 2015b; 2017), the trade-off between return and cost is manually balanced with a fixed Lagrange multiplier, which cannot guarantee safety through learning. To address this, inspired by trust region methods (Schulman et al., 2015b), Constrained Policy Optimization (Achiam et al., 2017) (CPO) establishes linear approximation the safety constraint and solves the corresponding optimization problem in the dual form. Compared with previous CMDP algorithms, CPO scales well to high- dimensional continuous state-action spaces. However, CPO does not distinguish the safe states from the unsafe ones in the training process, limiting its performance in the return.
Besides developing various optimization algorithms, some recent works also explore other ap- proaches to enhance the safety constraints, e.g., adopting the Conditional Value-at-Risk (CVaR) of the cumulative cost as the safety constraint (Tamar et al., 2015). Along this direction, Tamar et al. (2015) develop a gradient estimator through sampling to optimize CVaR with gradient de- scent. Prashanth (2014) further applies this estimator to CVaR-Constrained MDP to solve the stochastic shortest path (SSP) problem.
Our work considers a similar framework to CPO (Achiam et al., 2017), but it treats states differently by extending Generalized Advantage Estimation (Schulman et al., 2015c) to be senstvese Our proposed CSAE can boost the policy performance in terms of the return while ensuring the safety property. Moreover, our algorithm with WCMDP is safer than CPO in terms of constraint violation ratio during learning.
There are also some CMD- based algorithms for safe RL that are not in the scope of this work. In (Dalal et al., 2018), a linear safety-signal model is built to estimate per-step cost from state-action pairs and rectify the action into safe one. However, this method requires a pre-collected dataset
2

---

Under review as a conference paper at ICLR 2021
to fit the linear cost estimation model, which limits its application. Similarly, Cheng al. (2019) augment the model-free controller to enforce safety per step by designing a modle- baste controller with control barrier functions (CBFs). Some works introduce Lyapunov functions to build safe RL algorithms. For example, Berkenkamp et al. (2017) apply I. Lyapunov functions for safely recovering from exploratory actions, while Chow al. (2018) construct Lyapunov functions that explicitly model constraints.
PRELIMINARIES
A standard Markov Decision Process (MDP) (Sutton al., 1998) is defined with tuple (S,A, A, P. R,Y; 1), where S and A denote the set of states and actions respectively, P S x A - [0, 1] the transition dynamics modeling the probability of transferring from state 8 to s' after taking action R(s, a, 8') represents the reward function during this transition, 7 E [0, is the discount factor and μ S + [0, 1] denotes the starting state distribution.
An MDP agent is usually equipped with policy {a]s), which denotes the probability distribu- tion over actions a given a state 8. The performance of a policy π is measured with the expected discounted total reward J(x) Errw,,srryl EE= x RRss ae, St+1)], where 7 (So. dg $1,...) a trajectory generated by following policy x. RL algorithms for MDPs try to find the pol- icy x' that achieves the highest reward, i.e., x²
arg max, J(r). They commonly TE (E=o 7 R(s,, at 84,1 = s]. the action value function Qr(s, a) = Ersz EE 7 R(s ae 51+1)|5y = 8, 0g = al and the advantage function A (s, a) = (s, - (s). The discounted future state distribution will also be useful, which is defined as (s) = 7) 7 P(s, 2' s|x).
the value function (*(s)
Constrained Markov Decision Process (CMDP) (Altman, 1999) extends MDP environments with safety cost that could harm the agent when undesired actions are taken. As various safety costs may exist in single CMDP, we relate them with cost functions (Ci(s, a, ..),,...,C, Cm Cm a, s')), each of which denotes the cost an agent receives for each transition (s, a's') (similar to reward functions). Let CT(r) Ei 7 (5t,, 0t, St+1) denote the cumulative cost along trajectory generated from policy т. We consider a trajectory-basee cost constraint in CMDP, which limits the cumulative cost in expectation Jc. E,,, (T)] with value d,. Then safe RL aims to learn the policy π under CMDP by solving the following problem,
arg max ((T), s.t. Jc. = ( S d, 1 1, 1,...,...,...
Safe RL algorithms search for the policy x' that achieves the maximal cumulative reward and mean- while does not violate the imposed safety constraints on the costs. In the following, analogous to the definition of value functions (i.e., Vn, Qr and A₁), we use VC. QC: and A denote the cost-value functions w.r.. cost function C,.
METHOD
In this section, we develop a policy gradient based algorithm for solving the safe Reinforcement Learning problem in Equation 1. We will first derive novel cost-sensitive advantage estimation its learned policy in terms of
method and present thcoretical guarantees on the performance rewards from safe states. Then, we further develop a worst-case constrained MDP to augment the safety guarantee for learning policies. Finally, we present our safe RL algorithm in details.
4.1 COST-SENSITIVE ADVANTAGE ESTIMATION
Conventional policy optimization methods (either for RL or for Safe RL) usually model the policy with a parametric function approximator (e.g., neural networks). and directly optimize the expected return J(T)), where denotes the policy parameterized with 0. The gradient estimator y for policy gradient methods (Schulman al., 2015b;e) generally takes the following form:
E
(ss, au VoTo((a(ss)
(2)
3

---

Under review as a conference paper at ICLR 2021
where (st, ae) is responsible for guiding the policy updating direction and popular choice for ((s. ae) is Generalized Advantage Estimator (GAE) (Schulman et al., 2015c) which substantially reduces the variance of policy gradient estimate. The formulation for GAE' is given by
A GAE
(7)))
(3)
where À E [0 is hyper-paramente parameter. When À 0, it reduces to one-step TD error estimator; when 1, it reduces to the empirical return estimator.
Cost-sensitive Advantage Estimation Existing safe RL algorithms directly deploy these estima- tors without adaptation to the specific feature safe RL problems and fail to consider the safety requirement within the gradient estimation. For example, CPO (Achiam et al., 2017) uses environ- ment reward to estimate the advantage function for policy optimization, without considering that some high-reward states may also be unsafe.
In safe RL, an unsafe state with high reward would bias policy update towards favoring such a state and wrongly encourage the agent to violate cost constraints, if directly applying the GAE estimator. A natural solution is to penalize the reward for unsafe states. However, it is difficult to adjust the penalty appropriately. Specifically, over-penalization would suppress visiting the nearby safe states with high reward as their ((s,, qz) will be negatively affected during bootstraping. On the other hand, the unsafe state cannot be avoided when the penalty is too small.
Since & can be considered as an estimate of the advantage value of taking action at step t, the policy gradient estimator 9 points to the direction of increasing (a, (8)) only if the advantage of a is greater than zero. Therefore, to guarantee that agents can gain rewards mainly from safe states, we propose to generalize GAE for safe RL by zeroing the TD error 5 of unsafe states to avoid the agents from further exploring these regions. This is given by
A CSAE(T.))
(4)
1=0
where a is binary variable denoting whether transition (st, ae, st+1) is safe (a, = 1) or not (a, 0). Following standard assumption safe RL (Achiam et al., 2017). given the returned cost from the environment in the training phase, a can be obtained by binarizing the cost value C(st, Gt, 8t+1), i.e., 0r = (C(s, ae, 8(+1) > 0]. With this new advantage estimation, the policy gradient estimator for safe RL is given by
oc
CSAE
E
Σ
VOTo(aalss)) VoTTCas)
A
CSAE(SAA)
which is compatible with any policy gradient based methods.
CSAE and Reward Reshaping The above CSAE is equivalent to a moderate reward reshaping to penalize the reward for unsafe states. More specifically, it replaces the reward value for an unsafe state with the expected one-step reward an agent can receive at this state:
[ R(ss at, ar, 8141), En [R(s s'),,
if or
R(st, att 51+1)
(5)
if aa
Using this reshaped reward function induces the above CSAE advantage estimator. To see this, we use rt and F, substitute R(s,, ae, St+1) ) and R(s,, at, St+1), respectively, the following and drop subscript ı from the value function for notation simplicity Following standard definition, at time
step a k-step advantage estimation ( using the value function V and our revised reward signal F can be expressed as
A V(s)) + 7F+1 + + - + + ((++k)
(6)
I We use A GAA((TA) denote A AGAE(s (st,
Note that the reward revision mechanism in Equation 5 is only used for advantage estimation. For fitting the value function during leaminng we still the original reward function R(s,a, 8').
4

---

Under review as a conference paper at ICLR 2021
By substituting one-step TD error ó, and reward function (Equation 5) into Equation 6, the above advantage can be rewritten as
1 Σ
(7)
I=0
See the appendix for the complete proof. Analogous GAE, CSAE can be obtained by
taking the ssponentiallywweihhue average of above k-step advantage: A (CSAE(TAA) = (1
A)
(")) This provides another perspective, from reward reshap-
ing, to interpret the proposed CSAE. As policy optimization methods will automatically force agents to find high-reward regions in the state space, using the averaged reward can prevent unsafe yet high- reward states from attracting the agent during learning.
From the reward reshaping perspective, another possible approach deal with the cost is to include the cost &a in the reward by reshaping Tr to RG = T₁ + À x cr. But it is difficult to properly choose the trade-off parameter À due to: if has fixed value, it is not easy balance rr and ct as their best trade-off varies across environments, method is free of hyperparameter tuning and easy deploy. 2) À is treated as the dual variable for safety hard constraints and updated in similar way as PDO, the performance is worse than our method, due to the optimization difficulties, as justified in our experiments.
verified by Tessler et al. (2018). In contrast, our proposed
Worst-Case Constraints As discussed in Sec. 3. in CMDP, the trajectory-based safety cost for cost function C, is computed and constrained in expectation, i.e., Jc, (T) Bx D t= D ('Ca(st, az St+1)] S dz. However, this will certainly lead the agent to violate the constraints frequently during learning. To further enhance safety, we instead consider the worst cases and constrain the cost from the trajectories incurring largest cost.
We propose the Worst-case Constrained MDP (WCMDP). an MDP with a constraint on the CVaR of cost values (Tamar et al., 2015; Prashanth, 2014) in safe RL. It tries to find policy that maximizes the cumulative return, while ensuring the conditional expectation of other cost functions given some confidence level 9, to be bounded. Formally, for cost function C; and given 3 E (0, 1), the worst case constraint is given by
Σ CCis (osss,,,,,se)) 81:1))
(T) RT-A,,
where Ax.9 3 is the set of top 3 worst trajectories with the largest costs. We found the performance is robust the value of B and empirically set 3 0.1. Accordingly, the safety constraint related cost function C2 expressed as Jé, (T) S dj.
4.2 SAFE RL ALGORITHM WITH CSAE
Different from general RL problems, for safe RL, it is critical to ensure that the agent mostly gains reward from safe states and transitions. Thus, we are concerned with the following cost-sensitive return developed from the reshaped rewards in Equation 5 in safe RL:
20 R(s at St41)
( ETTTI
where R(s,, at, 8141) 0r R(s,, az, 8') + 0z) B. [R(s,, a, 8'). Different from the ventional return that accumulates the rewards from both safe and unsafe states, the above reshaped return characterizes how much the agent can gain reward from safe state-actions. In this section, we demonstrate adopting the proposed CSAE in policy optimization would naturally optimize Saafc- To this end, we establish the following theoretical result that gives performance guarantees for the policies in terms of the cost-sensitive return Jaate(T).
Theorem For any policies n',, x with x² +maxs [Bawor (A (SSAE(T.)) (s, a)}} the following bound holds:
2
Junte - -sato(TT)
(T) 2 1-
ExndT axx
Ar
(s,
1-7 Drv (rv(T(|
(10)
5

---

Under review as a conference paper at ICLR 2021
Here Dry denotes the total variance divergence, which is defined as Drv (rv(pllg) E lp: qu for discrete probability distributions and q. Due to space limit, we defer all the proofs to the appendix.
The above result bounds the difference of two policies in terms of the cost-sensitive return via the CSAE. Leveraging such a result, our safe RL algorithm updates the policy by
T = arg max E Eendtt a~n TT ().A) () a)] DrD (T [+]s]
(ASAAE(>
(11)
s.t. Já. = (Cr <
In particular, from Equation 10, for appropriate coefficients vas the above update ensures mono- tonically non-decreasing return from safe states. Details of the practical implementation of this algorithm are provided in the appendix.
EXPERIMENTS
As this work targets at obtaining safer and better policies, through experiments we aim to investigate: 1) whether our designed CSAE is effective for guiding the policy optimization algorithm to achieve higher cumulative reward while satisfying safety constraints; 2) whether the new policy search al- gorithm induced from WCMDP can guarantee stronger safety without sacrificing the performance; and 3) whether our method is able adjust the advantage value of each transition properly to better guide policy optimization. Therefore, we evaluate our methods on multiple high-dimensional con- trol problems that mainly include two different tasks. 1) Circle (Schulman et al., 2015b) where the agent is required to walk in circle to achieve the highest cumulative reward, but the safe region restricted to lie in the middle of two vertical lines. 2) Gather where several apples are randomly placed in both safe and unsafe regions, and an agent should collect as many apples as possible from the safe regions and avoid entering the unsafe regions. In our experiments, the reward for collecting one apple is 10, and the cost is I for each time the agent walks into an unsafe region. See Fig. 3 for an example of the gather environment, For the circle enviroument, we use three different robot agents in Mujoco (Todorov et al., 2012), i.e., point mass, ant and humanoid. For the gather environment, we conduct experiments with point mass and ant.
We use CSAE (Sec. 4.2) denote the safe policy search algorithm equipped with our proposed cost-sensitive advantage estimation, and ESAE-WC to denote the algorithm that further includes worst-case constraints. We compare these two methods with three well-established baselines. TRPO (Schulman et al., 2015b); the most widely used policy optimization method; CPO (Achiam al., 2017): the state-of-the-art safe RL algorithm for large-scale CMDP: PDO: primal-dual optimization based safe RL algorithm (Achiam et al., 2017). For all the experiments, we use a multi-layer perceptron with two hidden layers of (64, 32) units as the policy network. Our imple- mentation is based on rllab (Duan et al., 2016) and the Github repository?. The hyper-parameters for the environments and algorithms are given in the supplementary material.
Results The learning curves for all the methods and environments are plotted and compared in Fig. The first row is the cumulative reward. As we are dealing with environments with safety cost, we only accumulate the rewards collected through safe transitions as an optimal safe RL algorithm should be able to acquire rewards from safe states and avoid high-reward unsafe states. We also visualize the full returns in Fig. 1 (second row) for completeness. From the results, one observe that our CSAE surpasses CPO throughout all the environments. This demonstrates the effectiveness of CSAE for learning safe agents with higher rewards. Furthermore, with the help of worst-case constraints, CSAE- WC performs the best in terms of rewards from safe states for PointCircle and PointGather or comparably well for AntCircle, HumanCircle and AntGather, outperforming CPO. The second and third rows in Fig. plot the cumulative cost and ratio of the safe trajectories' in all the trajectories at each sampling. Specifically, safe ratio of means all the collected trajectories are safe. From the results, the cost value of TRPO agents explodes as the training proceeds, while all the other three methods converges. Among them, CSAE achieves comparable cost value as CPO and higher safe ratio. CSAE-WC surpasses the other methods it not only satisfies the constraint with less cost but also achieves highest safe ratio (nearly 1). These results clearly show that our method is effective at both enforcing safety and collecting more rewards, or it is safer and better.
Ptttps: : //gi thub. cww/ jachiam/ cpo/ *One trajectory is counted as safe if its cumulative cost is smaller or equal to the constraint value d.
6

---

Under review as a conference paper at ICLR 2021
PaintCircle
AntCircle
HumanCircle
PointGather
AntGather
1803
200
1040
150
1900
100
2000
Figure 1: Learning curve comparison between our methods (CSAE and CSAE- and the state-of- the-arts (TRPO, PDO, CPO) for five safe RL problems. First row: safe cumulative reward. Second row: total cumulative reward. Third row: cumulative cost. Fourth row: ratio of safe trajectories. axes denote the training iteration. (Best viewed in color). Each curve is obtained by averaging over five random runs. The standard deviation of different runs is visualized with the shade.
TRPO
PDO
CPO
CSAE
CSAE-WC
Figure 2: Agents trained in PointCircle. The grey circle denotes the path with highest reward. The two red dotted lines are the boundaries and the agent constrained to run between them. Lines with different colors starting from the center are agent trajectories learned with different random seeds.
Visualization To intuitively justify our method indeed learns agents that take safer and better ac- tions, visualize agent trajectories for the circle task (Fig. 2) and the gather task (Fig. 3). Fig. shows TRPO agent follows the circle specified by the reward function without considering con- straints. The other safe RL agents can learn to obey the constraints to some extent. However, they do not perform well as they usually get stuck in a corner (e.g., for PDO and CPO). Our CSAE-WC agents, however, can walk along the ares and safe boundaries. Similar observations can be made in AntGather, where TRPO agent inevitably violates the constraint and rushes into unsafe regions (i.e., the red squares). The other agents learn to avoid such cost but sacrifice the rewards. However, CSAE and CSAE-WC can work better to collect more rewards than others. In summary, both visu- alizations in Fig. and Fig. 3 demonstrate the effectiveness of our method for learning better agents that generate more reasonable and safer trajectories.
Analysis We here investigate how our proposed CSAE helps the training process and the resulted agents. We use PointCircle as the environment to conduct the following analysis. First, justify
7

---

Under review as a conference paper at ICLR 2021
PDO
CSAE
CSAE-WC
Figure 3: Agents trained AntGather. The green circles denote the randomly placed apples to collect and red-colored squares the unsafe regions. The blue lines trajectories of an agent trying to explore the environment to collect apples.
PointCircle
CPO
CSAE-WC
66-
Mean
Zero
100
500
Iteration
100
(a)
(b)
Figure 4: (a) Comparison of average return PointCircle for different reward modifications. "Mean" is our method in Eqn. equation 5 and "Zero" reshapes reward into Tz = Or X rz. (b) Advantage value visualization. Each colored dot represents transition in the trajectories, whose intensity denotes relative value of the corresponding advantage. (Best viewed in color).
the method of replacing the reward with the expected one-step reward (Bquation 5) for unsafe states. We compare it with simple reward reshaping method that zeros the reward of unsafe transitions and plot their learning curves (of average return) in Fig. 4a. The results show that our method (denoted by "Mean" in Fig. 4a) performs much better. This indicates that our method can overcome the shortcomings of penalizing the reward of unsafe transitions not properly.
Second, is important for safe RL algorithms help the agent distinguish high-reward but unsafe states from the safe ones. To investigate the differences of safe RL algorithms (PDO, CPO and our CSAE-WC) in this ability, we sample 300 trajectories (100 from each method). For different algo- rithms, we use their deployed reward and value functions to estimate the advantage value for each transition in these trajectories. The advantage values are visualized Fig. 4b, where more reddish means higher relative advantage value and bluish means lower value. From such visualization, one can observe that these three methods can recognize high-reward and safe state-actions by assign- ing higher advantage values, shown in the left- and right-top in Fig. 4b. However, algorithm CSAE-WC prefers these safe and high-reward regions more with higher advantage val- ues. Importantly, shown in the right-bottom (unsafe but high-reward regions), state-actions within such regions much lower advantage. In contrast, PDO and CPO even assign above-the-avee aveerage advantages to them. This result clearly demonstrates the superior and desired ability of our method to distinguish unsafe states from the safe ones for policy learning.
method gives
6 CONCLUSION
In this paper we consider Safe Reinforcement Learning and propose novel CSAE method to appro- priately estimate the advantage value for policy optimization under risky environments. Compared to conventional advantage estimation. CSAE eliminates the negative effect of high-reward but un- safe state-actions by depressing their advantages. To further enforee safety constraints, we augment the CMDP with the worst-case cost constraint and proposed WCMDP. We theoretically analyze their performance and safety benefits. We then develop new safe RI algorithm which shown effective for learning safer and better agents in multiple large-scale continuous control environments.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Gearning-Volume 70, pp. 22-31. JMLR. org, 2017.
Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. In Advances in neural information processing systems, pp. 908-918, 2017.
Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforce- ment learning through barrier functions for safety-critical continuous control tasks. arXiv preprint /00.00/900,0899., 2019.
Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for evar optimization in mdps. In Ad- vances neural information processing systems, pp. 3509-3517, 2014.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Iyapunov- based approach to safe reinforcement learning. In Advances in Neural Information Processing Systems, pp. 8092-8101, 2018.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint
2018.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning. volume 48 of Proceedings of Machine Learning Research, pp. 1329-1338, New York, New York, USA, 20-22 Jun 2016. PMLR. URL http://proceedingg www.s/rs.....eeeeeeeeeeeeeeeeeee pwwps///////ppeepppp /4////an1666666 html.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Javier Gareia, Daniel Acera, and Fernando Fernández. Safe reinforcement learning through proba- bilistic policy reuse. RLDM 2013, pp. 14, 2013.
Chris Gaskett. Reinforcement learning under circumstances beyond its control. 2003.
Peter Geibel and Fritz. Wysotzki. Risk-sensitive reinforcement learning applied to control under constraints. Journal of Artificial Intelligence Research, 24:81-108, 2005.
Alexander Hans, Daniel SchneegaB, Anton Maximilian Schäfer, and Steffen Udluft. Safe explo- ration for reinforcement learning. In ESANN, pp. 143-148, 2008.
Matthias Heger. Consideration of risk in reinforcement learning. In Machine Learning Proceedings 1994, pp. 105-111. Elsevier, 1994.
Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning, 49 (2-3):267-290, 2002.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529-555 February 2015. ISSN 00280836. URL http://dx. doi. org/10../00. 10t8/naturee1e.eem
Martin Pecka and Tomas Svoboda. Safe exploration techniques for reinforcement learning-an an overview. In International Workshop Modelling and Simulation for Autonomous Systems, pp. 357-375. Springer, 2014.
LA Prashanth. Policy gradients for constrrntr mdps. In International Conference on Algo- rithmic Learning Theory, pp. 155-169. Springer, 2014.

---

Under review as a conference paper at ICLR 2021
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning. volume 37 of Proceedings Machine Learning Research, pp- 1889-1897, Lille, France, 07-09 Jul 2015a. PMLR. URL http: http::://pcoceenngs.... mmrmnnsss..r.....m /tt.aiiinnsmmr..eees///// v37/schulman15.ng htmml.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015b.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint Stfi::1506.02433 2015c.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbremner, Ilya Sutskever, Timothy P. Lilli- crap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489 2016. doi: 10. 1038//a/ee 16961. URL htttp::///ot http://ooooooo////////o 1rg/10. htt.://...1///ntuuuuua 1838/nature16961
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550:354-, October 2017. URL http://dx. doi org/ 10. 1038 1100////ntrrr nature2277
Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998.
Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the evar via sampling. In Tiwent-Ninth AAAI Conference Artificial Intelligence, 2015.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805. 11074, 2018.
Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control.
In 2012 IEFF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
10

---

Under review as a conference paper at ICLR 2021
7 APPENDIX
7.1 POLICY OPTIMIZATION WITH WORST-CASE CONSTRAINTS
Since solving the exactly optimal policy is intractable for large-scale problems, policy gradient based methods represent the policy with 6-parameterized model, and try search for the best policy within the parameter space lIg, i.e., x = arg maX (TT)). Similarly, in the optimization prob- constrained policy optimization, we additionally require the policy
lem induced from the worst satisfy a set of safety constraints 11g. In other words, we are optimizing the policy achieve highest cumulative reward over the intersection of Hl and IT which is formulated as
[TT,, s.t. (x)( S d,,
max
m72
#EIly
Compared to CMDP objective in Equation 1, proposed method requires the worst 3-quantile trajectories (instead of the average cost) to still satisfy the safety constraints. This will yield safer policy as proved later. Before presenting our algorithm in full details, the following result is given that is useful for connecting the worst case safety cost of two different policies and their difference.
Theorem 2.
Let P3 denote the state transition probability P of B-worst case trajectories. For any policies n and n², define EC; = mmax; (Ea- s' [Ci(s, a, a, (s'))))s)> Vc. (s)]l. Let d' denote the discounted future state distribution for the B-worst trajectories. Then the following bound holds:
J8 (T') S JE,(T)
27 (C) + 1-> Drv (T (T]])))
(12)
The above gives an upper bound on the worst case cost for policy n' Explicitly constraining such an upper bound during the policy learning would enforce less cost constraint violation. Compared with the risk-sensitive CVaR models (Chow & Ghavamzadeh, 2014), this work is among the first to introduce such worst-case constraints into safe RL problems. Besides, it is also the first to present theoretical analysis the expected worst-case cost of two policies, which is of independent interest.
We now show how to develop a practical algorithm for safe RL based on WCMDP and CSAE. Inspired by trust region methods (Schulman et al., 2015b), by replacing Jc, with Jé and applying Theorem 2, we reformulate the update in Equation into
Th+1 = arg max E Bewdr aa ICSAB (s, a)]
s.t. Jg + 1-
sTatTe amst
[AC S di
(13)
DKL( [Tx)
m,
which is guaranteed to produce policies with monotonically non-decraasing returns from safe state- actions. Meanwhile, the policies will satisfy the original safety cost constraints.
7.2 ALGORITHM DETAILS
To efficiently solve Equation 13, we linearize the objective and cost constraint around т and expand the trust region constraint second order, similar (Schulman al., 2015b; Achiam al., 2017). We use 0ş to denote parameters of policy πg. Denote the gradient of objective and constraint for J as g and b,, respectively, and the Hessian of the KL-divergence as H. The approximation to Équation 13 is given by
OK+1 = -rag T (0 0s)
s.t.
6. (0- 0x) - - (mx) d, 5 0, 1,..., 1
(14)
2 0x) H 8x) < 8.
11

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Worst-case Constrained Policy Optimization
Input:
Initial policy E llg, tolerance and confidence level B
for = 0, 1, dol.s do
Sample trajectories D. {T), ne:
Form sample estimates g. HI, with D;
If the primal problem in Equation 14 is feasible then
Solve dual problem in Equation to get X", 1²
Compute updated policy parameters 6* with Equation 16.
else
Compute recovery policy parameters O with
= 0k
11-16 H -b.
end if
Obtain 0k+1 by backtracking line search to enforce satisfaction of sample estimates of con- straints in Equation 13. end for
As H is always positive semi-definite, the above optimization problem be efficiently solved in its dual form when the gradient and b; are appropriately estimated. Here, g can be easily obtained by taking derivative of the objective after replacing GAE with our proposed CSAE, For estimating the gradient b; of the CVaR constraint et al. (2015):
Jg. we adopt the likelihood estimate proposed by Tamar
VoJg
EyWA, Sumij
(J& (60) (J,, Vo log (als))
Here VaRa(JC, (so)) is empirically estimated from the batch of sampled trajectories used for each update. Then we use the same algorithm as CPO (Achiam et al., 2017) to learn the policy.
We now derive the full algorithm solve the optimization problem in Equation 14, which can also be found CPO Achiam al. (2017). Let c denotes J (Tx) d,. B [bi...., 6m] and
[cI...., cml", we can express the dual to Equation 14 as follows:
1 (9" H - 2r v + v2 c-
A6
max A20,00
(15)
2X
2
r 9 B B. Solving this problem much easier than the primal problem
especially when the number of constraints is low. Let X", ./* denote solution the dual problem, the solution to the primal is given by
1 8" = 0y + X
(g-
(16)
We now ready to present the full algorithm in Algorithm 1.
7.3 EXPERIMENTAL PARAMETERS
For circle tasks, the cost function is given by
C(s, 1[ > xmml
(17)
where x is the horizontal position of the agent after this transition, Fim is a Slyper-parameter speci- fying the location of the two vertical lines defining the safe regions.
For all the experiments, set the discount factor to be 0.995, and the KL step size for trust region to be 0.01. The other parameters for environment and algorithms in our experiments are listed the following table.
12

---

Under review as a conference paper at ICLR 2021
PointCircle
AntCircle
HumannoidCircle
PointGather
AntGather
Circle Radius
10
Plim
4
1.5
2
Batch Size
50k
100k
100
50k
100k
Trajetory Length
50
1000
1000
30
500
Constraint Value
5
10
10
1
B
0.5
0.1
0.1
0.1
0.1
Table 1: flyper-parancen settings.
7.4 PROOF OF K-STEP ADVANTAGE
We have V(s²) = En,s+ Irx + (s Rearranging it gives Eu,, [r,] = V(s;)
GlBaa (V(++1)), which actually provides unbiased estimator f. for the expected one-step ward Bai [ra], as given by
P = V(+) - (8
(18)
Rewrite F3 as F, = a,T, + - (1-a,) fr then we have
A,a0 - (8t) + + + TF++1 + ... F14h-1 + + V((+kk)
=V((8)) + art + 700+11+41
7 a1tk 17ttk-1 + (8(44)
+ - (1)- + 7(1 - a++1) + h - at+k-1)
= -V((s) + or + 11 +... 7 -
04 + + (8(++*)
+ - (1-ax)[V(ss) -7V(5++1)
+ +1 11(1-a++100 (V -
+..
+ - -00---1
= -v(s + ar/re + +V((s(+1) - V(s))) + (V(s), - - [V(84+1] +
+ 70tt1/1e41 + (V((422 V((++1) ++VV))++([+[++11 (8(422))
+
+ 10- + || ( -V((e++)
((
044102+1:
where 8,_1 = Te4 + (8(4411)) - (88+1).
7.5 PROOF OF THEOREM
Lemma 1, Achiam et al. (2017) For any function f:S-I + R and any policy x,
(1 Eo (s)) + Ex-ar
[xf((')] Ba [f() = 0.
(19)
are
s'mP
Combining this with Equation 9, we obtain the following, for any function f and any policy .:
J = Ea-u((())) + Rena
(F),, + f())]
(20)
In particular, we choose the function f(s) to be value function V*()). Thus, we have
ssafs = (s)]
+
E
Exda
+V* V*
1
13

---

Under review as a conference paper at ICLR 2021
Lemma 2. For any function: f f S E and any policies π and n', define
<'(ass)
1) +
Lxns(T²) = Band" aut d'mP
(21)
x{aas)
and eg. max, |aawa' (788, a, 8)) + 70((8) - [()]]. Then the following bounds hold:
jealo ( joale (Tt) 2
(m,((T²)) - -2e DTv (dFl())))))))
7
(22)
joufo jsnfo $ 1-7 (x,,(x²)) + 2 DI DIv(d"lld"))
where Drv is the total variational divergence.
Proof. The proof be established by following the one for Lemma 2 in Achiam et al. (2017) where we substitute Ju in Equation 20.
Lemma 3. Achiam et al. (2017) The divergence between discounted furture state visitation distri- butions, ||d= d= [|,, bounded by an average divergence of the policies x' and
ldd -d- S 1
1 Bond (Drv(''|
27
(23)
where (rv(3'|[T)[s) (1/2) (a|ls) (alss)-<aala (a|s))
Now, with Lemma 2 and Lemma 3, we are ready prove Theorem follows.
Proof. By choosing f(x:) = (s:) to be the safety value function in Lemmas 2 and, we have
f E. serud"
[Fx + 7VT(((+1)))) V (s:)] E
serdd
[Fr + VV (8t+1) V*()))]
amaa' 1
am- S1:1NP
E srrd
I + 7F4+1 + + (82-2) V - (s))) - E
sove
Fe + TT41 +72V" (*()))
P
[ire + --
Fz4k-1 + (5(+k)) - (st)
an avd
E sinud
Fi + + 2 F VV - (s:)))
a
11-1npP
[A[]
aand [A]] St+10p
s arot'' P
Thus, computing the exponentially average of y with À the weighting cofficient gives:
[A(SAB] ( E send
E serdd ann
[ACSAE]
E servd [AÇSAE]
(24)
awon' S1+1
The last inequality comes from the fact that E stndd [ACSAE]
S 0. Then applying Lemma 3 gives
a~x
the result.
7.6 PROOF OF THEOREM
Define be the 3-worst-case distribution over the trajectories, i.e., 1/8 is among
the top 33 most costly trajectories; and {(T) = otherwise. Denote = o o to the weighted probability distribution and d3 to be the discounted future state distribution for the 3-worst cases.
14

---

Under review as a conference paper at ICLR 2021
Then the expected cost over the 3- 3-worst-case trajectories can be expressed compactly as:
[g())= =
E swdz [C(s,a,8))]
(25)
and 6 PrPp
We also have the following identity:
-
= = (1-7)u
(26)
With the above relation, we can obtain the following lemma.
Lemma 4, For any function f -> R and any policy n,
(1 2 [s()]] E snd3 [x](6')] Eswdy Esd))][(s)] = O.
(27)
P
Combining with Equation 25, we have
1
Tt( = [s))(f(s)] + E sn ((s,,,,s)++fff f(()).
(28)
sNPP
Choosing the cost value function V8 as f gives:
(x( (VEE-pIV8 1-7
E swdg +VVE()) V8
(29)
s'mPs
Following the proof for Theorem 1, we obtain Theorem 2.
7.7 EXPERIMENTS ON WCMDP
To further study how our two contributions (CSAE and WCMDP) contribute the final algorithm, we perform ablation study where the safe algorithm does not dampen the advantage function but respects the worst-case constraints, which is referred as WC in the following. We compare WC with CSAE, CSAE-WC and the other baseline methods in Fig. 5. Compared to CSAE, though WC is able to give better safety guarantee. it actually produces inferior performance, especially on PointCircle and AntGather. Besides, CSAE also demonstrates faster convergence speed than WC. By involving them in the same algorithm, CSAE WC is able to combine their strengths and overcome their weaknesses, thus results in superior return performance and safety guarantee.
15

---

Under review as a conference paper at ICLR 2021
PointCircle
AntCircle
HumanCircle
PointGather
AntGather
CRAP
2000
100
CAAT
Figure 5: Learning curve comparison between our methods (CSAE, WC and CSAE-WC) and the state-of-thee-ats (TRPO, PDO, CPO) for five safe RL problems. First row: safe cumulative reward. Second row: total cumulative reward. Third row: cumulative cost. Fourth row: ratio of safe trajec- tories.
16


Reviews:
[{"review_id": "mx832ge-iGy", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n#### Summary:\n\nIn this work, the authors extend the CPO (Achiam et al) with a Cost-Sensitive Advantage estimation technique that eliminates the negative effect of unsafe high-reward state-action transitions. The authors then modify the CMDP objective with a CVAR based objective function and provide empirical results on the Mujoco based safety environments.   \n\n\n#### Strengths:\n\n- The idea of incorporating the notion of including safe/unsafe states to build a new advantage estimation is an interesting one, and I believe the link to Cost-Sensitive Advantage Estimation and reward shaping is novel.\n\n- The visualizations of the trajectory of the final policy returned by the algorithm were insightful to see. \n\n#### Weakness:\n\n- **Unclear setting:** The authors start with the motivation with the setting of the Constrained MDPs, but through-out the draft the discussion is carried in terms of safe and unsafe states. Note that, while the safe-state formulation can be defined using the CMDP framework, the opposite is not always true.  \n\n\n- **Technical flaws:** The method relies on an important assumption that is not discussed in enough detail. The paper assumes that the agent has knowledge about what states are safe and which states are unsafe ($\\alpha_t$ variable in Eq 4). Note that, this is an important distinction, as for CMDPs defining what states are unsafe (or what states to avoid) is a big challenge in itself (Altman, 1999). \n For the purpose of this work, the authors use a hard-coded safety distinction ($\\apha_t = \\mathbb{1}[C(s_t,a_t,s_{t+1})] > 0 ) and though this works for the examples chosen in this work (the examples based on CPO) there is no reason this will be true in general. In most cases, an optimal policy of CMDP is the stochastic policy, and directly avoiding some states is not always possible, and usually, the safe or unsafe states for CMDP are a function of the policy. \n \n This is also related to the author's claim that their proposed method is \"free of hyper-parameter tuning and easy to deploy\". In this case, the authors have already abstracted the difficult aspect of finding safe and unsafe states by including that information in the $\\alpha$ variable, and the authors assume that they always have access to this variable. As I mentioned above this is not always true. Furthermore, if the agent already knows safe and unsafe state distinction, there are much simpler and computationally efficient methods (compared to CPO) are available that can be deployed, for instance, (Dalal et al, 2018) that the authors mention in the manuscript.   \n  \n- **Incremental nature of work**: The proposed method relies heavily on the CPO (Achiam et al), both for the theoretical and empirical results and baselines. Combined with the above flaw of hard-coding the unsafe states, it is not clear what is the novelty of this work. The Worst-Case formulation uses a CVAR based objective instead of expectation based. While that is an interesting formulation, there are no comments made on how that changes the quality of the solution compared to the regular objective. Are the optimal policy in regular CMDP and CVAR-based objective different? How tough the problem becomes? \n  \n- **Reproducibility**: There is no mention of code release for empirically intense work. \n\n- **Baselines:** The authors mention that CPO is the SOTA, but that is not true. From (Ray et al, 2019), we know that Lagrange based approaches can actually perform better than CPO when trained properly. \n \n#### References:\n\n- Ray, Alex, Joshua Achiam, and Dario Amodei. \"Benchmarking safe exploration in deep reinforcement learning.\" arXiv preprint arXiv:1910.01708 (2019).", "extra_scores": {}, "extra_text": {}}, {"review_id": "l0cvciu9KgB", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nIn this paper, the authors proposed a new constrained policy optimization algorithm and a worst-case version of the constrained MDP framework. Tho proposed constrained policy optimization algorithm is based on CPO, and a novel advantage function (CSAE) based on the concept of a \"safe\" state. Experiments in control simulation tasks are provided.\n\nIn general, the main novel contribution of this work is the cost-sensitive advantage estimation (CSAE), which is later justified by the empirical study. However, I think the problem setting needs further clarification, and the new advantage estimation and reward shaping are not justified enough.  \n\nPros:\n1. The connection between the CSAE with a new form of reward is interesting. I think it provides more intuition or a different understanding of the advantage function.\n\n2. The CVaR version of CMDP is also a novel and interesting setting.\n\nCons:\n1. It is not clear enough what is the problem formulation that this work (or CSAE algorithm) aims to solve. Before section 4 it seems that the goal is to maximize the reward while keeping the *trajectory-level* cost under a budget. However, later the CSAE algorithm leverages the concept of \"safety\" of states. While the trajectory-level cost may or may not be decomposed into some state-level binary constrained, it seems either the algorithm design needs more justification or the problem setting is misleading. \nFurthermore, I think the choice of $\\alpha_t$ in the algorithm needs to be discussed extensively: instead of saying \"e.g. C_t > 0\", it needs to discuss how to choose the threshold, how the state-level constrained relate to the constraints on the discounted sum, etc. \n\n2. Since the new advantage function is linked to the new reward form in Eq (5), it needs more discussion about why this form of reward is a natural one. It isn't obvious why using the average reward for the \"unsafe\" state-action pairs can penalize them.\n\n3. The paper proposed two contributions but I did not follow how they are connected to each other. WCMDP can be applied to any constrained policy optimization algorithm. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "bqp0udk1jrg", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose two contributions:\n1 - A Cost-Sensitive Advantage Estimation procedure called CSAE which is intended to differentiate more clearly safe and unsafe state-actions pairs with high rewards;\n2 - A beta-quantile worst-case formulation of the Constrained MDP called WCMDP, and a variant of CPO  to solve this problem instead of its usual expectation formulation.\nI liked the second part which prevents constraint violation with high confidence but I found the part on CSAE hard to follow.\nThe notion of safety being defined here in term of trajectory budget, not in term of state, I had difficulties to understand the notion of \"safe-states\" or \"safe state action pairs\"  when opposed to the notion of \"safe-policies\" as defined for instance in (Achiam et al. 2017). This notion of safe-sates seems however to be a key element for the definition of the CSAE estimate which is a variant of the \"Generalized Advantage Estimation\" (Schulman et al. 2016) where the TD error is nullified on \"unsafe states\".\nIs it because the considered policies are stationary ?\nProbably for the same reason I did not understand the claim of the authors on page 2 that \"CPO sacrifices too much on the expected return\".  \nThe proposed algorithm is detailed in the Appendix, its main difference with the CPO algorithm is the formulation of the primal-dual problem of equation (14) which rely on a beta-worst case discounted cost (and CSAE) instead of the expected discounted cost.\nTheorem 1 gives a TRPO-like lower bound similar to the ones in  (Schulman et al. 2016).\nThe experiments in section 5 are performed on the same environments as (Achiam et al. 2017), allowing for a comparison of the two methods and underlining clearly the improvement of CSAE-WC against CPO and other baselines. The improvement from CPO to CSAE seems less clear to me.\nWhat would be the performance of a simple beta-worst case version of CPO without the CSAE machinery ?\n\nMinor remarks:\nA few typos did add to my confusion when delving into the paper, for instance on equation (2) and on page 4 when introducing the gradient CSAE the authors forgot the log in the policy gradient formula, a strange mistake that is hopefully not committed in the appendix.\nThe TD error delta_t should be defined before  equation (3) on page 4.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NUJ7CxAwUy", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors' rebuttal has addressed some of my confusion regarding the paper, which is greatly appreciated. The additional baseline of early termination would still be interesting to have, though I agree it's not critical for the presented line of work. In general, I think the work is interesting and will keep my current score (6).\n\n=================================\n\nThe paper introduces a learning algorithm for training a control policy to complete certain tasks while satisfying some safety constraints. The main ideas in this work are to first use a cost-sensitive advantage function, where the advantage values for the unsafe states are set to zero. Second, a more conservative estimation of the safety cost is proposed to further improve the safety of the robot during the learning process. The authors demonstrated that with the proposed modified reward function, the algorithm would obtain a controller that completes the task while being safe. The proposed algorithm is evaluated on a set of simulated control problems and with both proposed components used, the algorithm achieves better performance in terms of both task completion and safety than prior methods.\n\nThe paper solves an important problem of training a performant control policy while taking the safety of the robot into consideration. The paper is well written and the experiments show good results compared to prior methods.\n\nHowever, I do have a few questions about the paper:\n1. The paper mentioned first about zeroing the advantage for the unsafe states and showed the equivalence of doing that to a shaped reward which replaces the original reward with the shaped reward. I'm wondering if this equivalence also holds for the safe states? For example, within a trajectory that ends in an unsafe state, a prior safe state's advantage will take the original reward of the unsafe state into consideration if only the unsafe state's advantage is modified, which might be different from the shaped reward version. It would be great if the authors could help me understand this part better.\n2. In the implementation of the work, is it using the modified advantage function in Eq. 4, or the shaped reward in Eq. 5? It's a bit confusing in that the title of the paper suggests the use of Eq. 4, while section 4.2 seems to be saying it's using the shaped reward? If it's using the shaped reward version, is there a reason not to use Eq. 4 directly?\n3. It seems a simple baseline to compare to is to terminate the rollout when the robot enters an unsafe state, as is done in various OpenAI Gym tasks. There seems to be some analogy in this strategy and the proposed one in that when the rollout is terminated, the policy will not have any updates regarding the unsafe states, i.e. zero gradient, and thus corresponds to zero advantage in policy gradient formulation. I do note that there are still many differences between the two methods, but I feel it's worth trying given the simplicity of the approach and how it's been effective in teaching the robots to run upright in existing problems.\n\nOverall, I think it's an interesting approach with good results.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1nJHwqksDg5", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors propose to improve a safe RL algorithm, constrained policy optimizaiton, that can learn the optimal safe policy while exploring unsafe states less often during the training process. In particular, they dampen the estimated advantage associated with unsafe states, which encourages the RL algorithm to explore safe states more often during the learning process. In addition, the authors aim to find a policy that satisfies the constraints with high probability, rather than only in expectation, by considering the worst-case constraints. The empirical results show that a safe RL algo that dampens the advantage and respects worst-case constraints are able to learn policies with large returns and avoid unsafe states.\n\nPros:\n+ The authors present a simple modification of a safe RL method to make it learn faster and more safely. The empirical results validate this claim.\n\nMajor concerns:\n\n1. The authors describe how their method replaces the reward for an unsafe state as the average reward, and claim that this is generally better than previous reward reshaping methods, e.g. setting the reward for unsafe states to zero. The authors only present a single example that compares the two approaches (Fig 4); However, it's not clear if the average reward is a wider range of scenarios. I'd like to see a more comprehensive comparison in Fig 1 to justify this claim. Also, can the authors give an explanation for why the mean is a more suitable penalization of unsafe states or is more universally applicable, compared to other reward reshaping methods?\n\n2. The authors compare the cumulative safe reward of different RL algorithms. However, the theoretical results in Theorem 1 compare the cumulative reshaped rewards. It is somewhat difficult to interpret the result in Theorem 1 since the reshaped reward uses the average reward for unsafe states, rather than setting the reward to zero. Given the similarity between there two rewards, can the authors establish a bound on the cumulative safe reward instead, as this seems to be the quantity we are truly interested in?\n\n3. In the empirical analysis, the improvement from CSAE to CSAE-WC was quite large, whereas the improvement from CPO to CSAE was smaller. To understand how much dampening the advantage improves on the worst-case constraints, can the authors do an ablation study where the safe RL algo does not dampen the advantage function but respects the worst case constraints?\n\nMinor concerns:\n\n1. Please plot the constraint threshold in Fig 1 row 2. It's hard to tell if the methods have satisfied the constraints, since there is no reference line.\n2. There is a typo when referring to the rows in Fig 1 -- see last paragraph on page 6. It should say \"third and fourth rows\", rather than \"second and third rows\".", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciate the importance of enforcing safety in RL, and the technical directions considered in the paper related to incorporating cost in advantage estimation.  However, they express several concerns about the formulation of the problem considered and the consistency of the approach, as well as the somewhat incremental contribution w.r.t. CPO.  Three reviewers recommend rejection.
Decision: Reject