Constrained Update Projection Approach to Safe Policy Optimization
Authors: Long Yang
Jiaming Ji
Juntao Dai
Linrui Zhang
Binbin Zhou
Pengfei Li
Yaodong Yang
Gang Pan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Constrained MDP
Deep Reinforcement Learning

Abstract: Safe reinforcement learning (RL) studies problems where an intelligent agent has to not only maximize reward but also avoid exploring unsafe areas. In this study, we propose CUP, a novel policy optimization method based on Constrained Update Projection framework that enjoys rigorous safety guarantee. Central to our CUP development is the newly proposed surrogate functions along with the performance bound. Compared to previous safe reinforcement learning meth- ods, CUP enjoys the benefits of 1) CUP generalizes the surrogate functions to generalized advantage estimator (GAE), leading to strong empirical performance. 2) CUP unifies performance bounds, providing a better understanding and in- terpretability for some existing algorithms; 3) CUP provides a non-convex im- plementation via only first-order optimizers, which does not require any strong approximation on the convexity of the objectives. To validate our CUP method, we compared CUP against a comprehensive list of safe RL baselines on a wide range of tasks. Experiments show the effectiveness of CUP both in terms of reward and safety constraint satisfaction. We have opened the source code of CUP at https://github.com/zmsn-2077/CUP-safe-rl.

Constrained Update Projection Approach to Safe Policy Optimization
Long Yangl.2., Jiaming J Juntao Dait, Linrui Zhang? Binbin Zhou, 3, Pengfei Li,", Yaodong
Yang2.,, Gang Pan
College of Computer Science and Technology, Zhejiang University, China
School of Artificial Intelligence, Peking University, China 3 Tsinghua Shenzhen International Graduate School, Tsinghua University, China
4 Department of Computer Science and Computing, Zhejiang University City College, China Institute for Artificial Intelligence, Peking University & BIGAI, China yanglong001 1pku. edu. cn, gpanšzju. edu. cn
Abstract
Safe reinforcement learning (RL) studies problems where intelligent agent has to not only maximize reward but also avoid exploring unsafe areas. In this study, we propose CUP, novel policy optimization method based on Constrained Update Projection framework that enjoys rigorous safety guarantee. Central to our CUP development is the newly proposed surrogate functions along with the performance bound. Compared to previous safe reinforcement learning meth- ods, CUP enjoys the benefits of 1) CUP generalizes the surrogate functions generalized advantage estimator (GAE), leading to strong empirical performance. 2) CUP unifies performance bounds, providing better understanding and in- terpretability for some existing algorithms; 3) CUP provides non-convex im- plementation via only first-order optimizers, which does not require any strong approximation on the convexity of the objectives. To validate our CUP method, we compared CUP against a comprehensive list of safe RL baselines on a wide range of tasks. Experiments show the effectiveness of CUP both in terms of reward and safety constraint satisfaction. We have opened the source code of CUP at https: //get////ttub...g wwt.coo///aaaa/ honm/mmmn--aa http:/mmmn2277777 wwt.:///m-2277//00------- 20T77/0P--a--------
Introduction
Reinforcement learning (RL) [Sutton and Barto, 1998] has achieved significant successes in many fields (e.g., [Mnih et al., 2015; Silver et al., 2017: OpenAI, 2019; Afsar et al., 2021; Yang et al., 2022]). However, most RL algorithms improve the performance under the assumption that an agent is free to explore any behaviors. In real-world applications, only considering return maximization not enough, and we also need to consider safe behaviors. For example. robot agent should avoid playing actions that irrevocably harm its hardware, and recommender system should avoid presenting offending items to users. Thus, is crucial to consider safe exploration for RL, which usually formulated as constrained Markov decision processes (CMDP) [Altman, 1999].
challenging solve CMDP since traditional approaches (e.g., Q-learning [Watkins, 1989] & policy gradient [Williams, 1992]) usually violate the safe exploration constraints, which is undesirable for safe RL. Recently, Achiam al. [2017]; Yang al. [2020]: Bharadhwaj el al. [2021] suggest use some surrogate functions to replace the objective and constraints. However, their implementations involve some convex approximations the non-convex objective and safe constraints, which leads
"L. Yang and J.Ji share equal contributions. PaP is the corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

many error sources and troubles. Concretely, Achiam er al. [2017]; Yang et al. [2020]; Bharadhwaj er al. [2021] approximate the non-convex objective (or constraints) with first-order or second Taylor expansion, but their implementations still lack theory to show the error difference between the original objective (or constraints) and its convex approximations. Besides, their approaches involve the inverse of a high-dimension inverse Fisher information matrix, which causes their algorithms require a costly computation for each update when solving high-dimensional RL problems.
To address the above problems, we propose the constrained update projection (CUP) algorithm with a theoretical safety guarantee. We derive the CUP bases on the newly proposed surrogate functions with respect to objectives and safety constraints, and provide a practical implementation of CUP that does not depend on any convex approximation to adapt high- -ighedimensioaal safe RL.
Concretely, in Section 3, Theorem 1 shows generalized difference bounds between two arbitrary policies for the objective and constraints. Those bounds provide principled approximations to the objective and constraints, which are theoretical foundations for us to use those bounds as surrogate functions to replace objective and constraints to design algorithms. Although using difference bounds as surrogate functions to replace the objective has appeared in previous works (e.g. [Schulman et al., 2015; Achiam al., 2017]), Theorem refines those bounds (or surrogate functions) at least two aspects: (i) Firstly, our rigorous theoretical analysis shows a bound with respect to generalized advantage estimator (GAE) [Schulman er al., 2016]. GAE significantly reduces variance empirically while maintaining tolerable level of bias, the proposed bound involves GAE is one of the critical steps for us to design efficient algorithms. (ii) Our new bounds unify the classic result of CPO Achiam et al. [2017], i.e., the classic performance bound of CPO is special case of our bounds. Although existing work (e.g.. Zhang et al. [2020]; Kang et al. [2021]) has applied the key idea of CPO with GAE to solve safe RL problems, their approaches are all empirical and lack a theoretical analysis. Thus, the proposed newly bound partially explains the effectiveness of the above safe RL algorithms. Finally, we should emphasize that although GAE has been empirically applied to extensive reinforcement learning tasks, this work is the first to show a rigorous theoretical analysis to extend the surrogate functions with respect to GAE.
In Section we provide the necessary details of the proposed CUP. The CUP contains two steps; first performs policy improvement, which may produce temporary policy violates the constraint. Then in the second step, CUP projects the policy back onto the safe region to reconcile the constraint violation. Theorem 2 shows the worst-case performance degradation guarantee and approximate satisfaction of safety constraints of CUP, result shows that with a relatively small parameter that controls the penalty of the distance between the old policy and current policy, CUP shares desirable toleration for both policy improvements and safety constraints. Furthermore, we provide a practical implementation of sample-based CUP. This implementation allows us to use deep neural networks to train a model, which is an efficient iteration without strongly convex approximation of the objective or constraints (e.g., [Achiam et al., 2017; Yang al., 2020]), and it optimizes the policy according to the first-order optimizer. Finally, extensive sigh-dimensional experiments on continuous control tasks show the effectiiveness of CUP where the agent satisfies safe constraints.
2 Preliminaries
Reinforcement learning (RL) [Sutton and Barto, 1998] is often formulated as a Markov decision process (MDP) [Puterman, 2014] that is a tuple M = A, r, po.7). 7). Here S is state space, A is action space. IP(s' |s, a) is probability of state transition from s to s after playing a. r(-)= S S x A- + R. and r(s"[s, (s. a) denotes the reward that the agent observes when state transition from s to s after it plays a. po(.): S + [0, 1] the initial state distribution and y E (0, 1).
stationary parameterized policy Te is a probability distribution defined on S x A, ne (als) denotes the probability of playing in state s. We use II to denote the set of all stationary policies, where Ile = {TT 0 E }. and 0 is parameter needed to be learned. Let P Xe E (sx|s be state transition probability matrix, and their components are: P PRols, [s, s'] = >.EA [((a)s)
=: P.o (s' [s), which denotes one-step state transformation probability from to s by executing me. Let T = (s,,a,, az, T121 ~ Te be a trajectory generated by ne. where so ~ po(-). a ~ (-(-)s)). $141 ~ P(-]ss,a)) at), and r1+1 = r(st41|st, a:). We use PTo (st = s' [s) to denote the probability of visiting the state after t time steps from the state by executing me. Due to the Markov property in

---

MDP, Pro (s; = s'|s) is (s, 3-th component of the matrix P i.e., Pro (st = s'ss) P [s, s' Finally, let dh (s) (1 - 7) Ei= >TPr=, (s: s|so) be the stationary state distribution of the Markov chain (starting at so) induced by policy тe. We define d (s) Beompo() ( (s)] as the discounted state visitation distribution on initial distribution po(-).
The state value function of тo is defined as Va (s) TR Ei=g 1'r2+1/80 = s), where Bmg H+] denotes conditional expectation on actions which are selected by TT. Its state-action value function is Q (s, a) E -t= =t==7"TT+1 7"T++1
3, ag a). and advantage function is Axo (s,a)
Qxo (8, a) v (s). The goal of reinforcement learning is to maximize J(TT) = LE Ee VV (s)].
2.1 Policy Gradient and Generalized Advantage Estimator (GAE)
Policy gradient [Williams, 1992: Sutton et al., 2000] is widely used to solve policy optimization, which maximizes the expected total reward by repeatedly estimating the gradient 9 (TTm) Schulman et al. [2016] summarize several different related expressions for the policy gradient:
( =
v log
(1)
where 1 be total discounted reward of the trajectory, value function, advantage function temporal difference (TD) error. As stated by Schulman et al. [2016], the choice V = A(s,, a,) yields almost the lowest possible variance, which is consistent with the theoretical analysis [Greensmith el al., 2004; Wu el al., 20. Furthermore, Schulman el al. [2016] propose generalized advantage estimator (GAE) A 2) (st, (x) to replace Vú; for any À E [0, 11.
A (A (st, > (7)) o+t::
where of ri+1 + 7V6-)) V(s,) is TD error, and V(.) is an estimator of value function. GAE is an efficient technique for data efficiency and reliable performance of reinforcement learning.
2.2 Safe Reinforcement Learning
Safe RL is often formulated as a constrained MDP (CMDP) M C [Altman, 1999], which is standard MDP . augmented with additional constraint set C. The set C = {(c,, b )m where & are cost functions: G : S A R, and limits are 6,, = 1, m. The cost-return defined as: J (Too) = Em EE== ('c,(st, at)], then we define the feasible policy set Il as: The goal of CMDP is to search the optimal policy m.:
He = n {To E IIg and J (TT(TT) < b.}
= arg max J(TT). JeElle
Furthermore, we define value functions, action-value functions, and advantage functions for the auxiliary costs in analogy V= Qup. and A Amo with e replacing respectively, denote them as Ve. Q% and A For example, Ve. (s) = Ex, - 7 Ca(e,, az)[ s). Without loss of generality, we will restrict our discussion to the case of one constraint with cost function e and upper bound 6. Finally, we extend the GAE with respect to auxiliary cost function c:
A (A
(st, ax) = (G>))
(4)
E=
where of = = + *C((444) C(s)) is TD error, and C(.) is an estimator of cost function c.
3 Generalized Policy Performance Difference Bounds
In this section, we show some generalized policy optimization performance bounds for J(ro) and (T(T).. The proposed bounds provide some new surrogate functions with respect to the objective and cost function, which are theoretical foundations for us to design efficient algorithms to improve policy
3

---

performance and satisfy constraints. Before we present the new performance difference bounds, let us revisit classic performance difference from Kakade and Langford [2002].
M(mo) J (TTgg) (1 ())a)) A (5, (s,a)]
(5)
(5) shows difference between two arbitrary policies тo and Th" with different parameters 0 and 0 According to (5), rewrite the policy optimization (3) follows
arg max Be-aly ())s) AA (s, (s,a)] ToElle
(6)
However, Eq.(5) (6) is very intractable for sampling- based policy optimization since it requires the data comes from the (unknown) policy To that needed to be Iearned. In this section, we provide a bound refines the result (5), which provide the sights for surrogate functions solve problem (3).
3.1 Some Additional Notations
use bold lowercase letter to denote a vector, e.g., a (01, 02, ... ,,n), and its i-th element a[i] =: ag Let ( S - RR be function defined on S, o% = r(sc+118e. uz) + 75p(8++1) (s.)) TT error with respect to yp(-). For two arbitrary policies me and Tgy, we denote 8% (x.)(() as the expectation TD error, and define A (TO. , as the difference between (s) and 8% (s)):
[ (o(ar((s) - 88
8 (s)
(Ta, To', [68 (TT
8)=
8 ao (11s)
P ao (11s)
To (8c)
a (-1sa)
a a o¹ (-lee) (-lee)
*14 -P( ((((s2;as)
Furthermore, we introduce two vectors 8 8%.... A (TTo To E R |s and their components are:
8% [s = (8), A (To, To [s] = A Too 5))
Let matrix (AA = (1 - 7 Ei (TA) where E [0. 1]. similar to the normalized
discounted distribution d (s), we extend it to X-version and denote it as di (s):
d (s) Eeo-po(-) (1-5) 5)
x(p(A) (s: = =s|s))
where 9 = x(1AA) the probability (P(2) (se = s|80) is the (so, s)-th component of the matrix product 1-7x
(p()))
Finally, we introduce vector d, E R| and its components are: d [s] = d (s).
3.2 Main Results
Theorem (Generalized Policy Performance Difference). For any function sp-)) S for two
arbitrary policies To and T for any p, E [1, o0) such that 1 + 1, we define two error terms:
(a(A) p.g.t (To,
To
!|d
d 11p116 1ll..
P (To, Fo)
EX ()) A (TO, For + p.g.t (TT, Tor)
9(0))
P.g.
t=u
Then, the following bound with respect policy performance difference J ((TO) J (Tg') holds:
p.a. (To. Tg') < J( J( S L8 (To. mg/)
(10)
Proof. See Appendix E.
The bound (10) well-defined, i.e., if no Rg' all the three terms in Eq.(10) are zero identically. From (9),)), we know the performance difference bound L (no. T (10) can be interpreted by (
two distinct difference parts: (i) the first difference part, i.c., the expectation Af (me, To' s), which

---

is determined by the difference between TD errors of To and Te' (ii) the second difference part, ie., the discounted distribution difference g.(2) Ter)) which is determined by the gap between the normalized discounted distribution of and To'. Thus, the difference of both TD errors and discounted distribution determine the policy difference (
The different choices of and lead (10) to be different bounds. p = 1, oc, we denote Fo. 1 maxs,ES (1) (1 then, according to Lemma 2 (see Appendix F.2), when p 1,7 xc, then error (P.(A) (TT,, TTo) is reduced to:
(.(A) 'pastt (To: Te/ p= ( - 1) c
[ where ( [s) is the total variational divergence between action distributions state s, i.e.,
(1 (1- - Es (-)
2Drv(Te To][s] = Σ
IT (als) (a(ass)
aEA
Finally, let yp Va the left side of (10) in Theorem implies lower bound of performance difference, which illustrates the worse case of approximation error, we present it in Proposition 1. Proposition 1. For any two policies no and Tg let co (TT') ) =: suptEN ( : = Vxy }. then
J(TT) - J(TTT 2 1-2 E. Bddd ()ammo(-11s)
A.o ACAB(T.)) (s,
27 (7A(()) - 1) + 1) (TTO) DTV (Tv(TT,70)(s) (1-5)(1- ( 72)
(11)
The refined bound (1I) contains GAE technique that significantly reduces variance while maintaining a tolerable level of bias empirically [Schulman et al., 2016], which implics using the bound (1I) as surrogate function could improve performance potentially for practice. Although GAE has been empirically applied to extensive reinforcement learning tasks, to the best of our knowledge, the result (11) is the first to show rigorous theoretical analysis extend the surrogate functions to GAE
Remark (Unification of [Achiam et al., 2017]). If X - O, then the distribution d² (.) is reduced (-) and the bound (11) is reduced to to deo o
J(mo) - J(TTr) 2 1 1 Bande ().))))(-)))) Ang (s, - 2 1 -2, c (
[o][s]
(12)
which matches the result of [Achiam et al., 2017, Corollary 1]. That is to say the proposed bound (11) unifies the classic bound (12)
Let so = V "o' Theorem 1 implies an upper bound of cost function as presented in the next Proposition
2, we will use it to make guarantee for safe policy optimization.
Proposition 2. For any two policies Th and TT let e (Tg') =: suprEN+ (c : Ve Trg
}. then
(S(TT) - [ < 1 Bandy (.).ammo (-1s)
1
A CAE(T1A)
Tpr.C (s,a)
27 (TA((S) - 1)+ 11) (TO') (1- -
Drv(mgg. mo>[s||
(13)
where calculate A ACAE(T,A) (s)) according to the data sampled from To and the estimator (4). aag C
All above bound results (11) and (13) can be extended for total variational divergence to KL- divergence between policies, which are desirable for policy optimization. Proposition 3. All the bounds in (II) and ( (13) hold if we make the following substitution:
E Badd (-) [Drv(To' No) [s][s]] +
V
B
and? ( (T)))))))))
where KL(,,)) is KL-divergence, and KL(TT [o][s] KL (-1s), no (-(s)))
5

---

4 CUP: Constrained Update Projection
is challenging to implement CMDP (3) directly since requires us to judge whether a proposed policy Ne is in the feasible region Hl. According to the bounds in Proposition 1-3, develop new surrogate functions to replace the objective and constraints. We propose the CUP (constrained update projection) algorithm that is two-step approach contains performance improvement and projection. Due to the limitation of space, we present all the details of the implementation in Appendix C and Algorithm 1.
4.1 Algorithm
Step I: Performance Improvement. According Proposition and Proposition 3, for appropri- ate coefficient og,, we update policy as:
Ta(a
( (s, - Okky Bondta ( [KI( [o)
3OL-3 = arg max
E goed Top amitor (-16)
TyElla
Top (als)
(14)
This step typical
matiiiiiiiimmmiiiiaaiiii (MM) algorithm [Hunter and Lange, 2004], includes return maximization and minimization the distance between old policy and new policy, the expectation (14) by sample averages according to the trajectories collected by Te.
Step 2: Projection. According to Proposition 2 and Proposition 3, for an appropriate coefficient S.. we project the policy Te, x+
onto the safe constraint set,
D (TO. m0 C.To, (To,BB)) S b,
No arg min xeElle
(15)
where D)..)) (e.g., KL divergence or (o-norm) measures distance between To m and To.
E Cro, (To. 3) (T 1 Ba ()a~ma(11s))
(00 (s,
+3 /Bondio ( [KL(TO Te) [
Until now, the particular choice of surrogate function is heuristically motivated, we show the worst- case performance degradation guarantee and approximate satisfaction of safety constraints of CUP in Theorem 2, and its proof is shown Appendix
3((A(()-1))))) If me= and To are Theorem 2. Let X Bsnds () (L T + [s]]] =: (1-9)(1-21
generated according (14)(15), then the lower bound policy improvement, and upper bound constraint violation are
LO0 V2 (TOL),
(T( S bb 1.xV2Xk (TTO)-
((F -(TO))) 2
- xx0+1
- Exoue]
Remark (Asymptotic Safety Guarantee). Let an + 0, + as + oc, Theorem implies monotonic policy improvement with an asymptotic safety guarantee.
4.2 Practical Implementation
Now, we present our sample-based implementation for CUP (14)-(15), Our main idea is to estimate objective and constraints in (14)-(15) with samples collected by current policy Ta, then solving its optimization problem via first-order optimizer.
Let {(s,, a, Tr+1 T o as:
Tea. we denote the empirical KL-divergence with respect Te and
T
DKI
KL(To(ue(st), To (ar():)).

---

We update performance improvement (14) step as follows,
T
xo(a))s)) (t)
As as V (TLL TT)
arg max Tlllly
T
me (arls:)
where A, is am estimator of Are,
A (s, a))
Then we update the projection step by replacing the distance D by Ki-diveegeece,, the next Theorem 3 (for its proof, see Appendix C.2) provides fundamental way for us to solve projection step (15).
Theorem 3. The constrained problem (40) is equivalent to the following primal-dual problem:
max min 220 Elly
{
To. mea+s) + (Cro, ( () B)
-)b
According to Theorem 3, solve the constraint problem (15) by the following primal-dual approach,
(Te 1x-1)) = arg min max mpelle 220
(KII(TTN Te) + vū(TTO me.))
where Č(To, mo.) jo + 1 E estimators for cost-return and cost-advantage. Finally, let
(a(a))))) AÇ + B&V DKL(TO,, TO) b, jCand Aç no, (ae)a.)
be To: To. 0K+g, =: DKLL. DTO To) + (
(16)
we update the parameters (fx+1, 1k-1) as follows,
0k+1 O2
00 Le
no: no, O 0ut D=be.u=K
To,
(17)
Uk+1
up +
f. (TO, To, O 0k+4. dv
0 10-0,,
(18)
where 7 > is step-size, {.)+ denotes the positive part, i.e., if < 0, {x) else fx)+ We have shown all the details the implementation Algorithm 1.
Related Work
Due to the limitation of space, for more discussions and comparisons, see Appendix and Table 2.
5.1 Local Policy Search and Lagrangian Approach
direct way to solve CMDP (3) is to apply local policy search [Peters and Schaal, 2008; Pirotta et al., 2013] over the policy space IIc, i.e.,
#o = arg max J(me), s.t. (TTO) < b, and D(TO, To.) Telll
(19)
where 5 is positive scalar, D(...) some distance measure. For practice, the local policy search (19) challenging to implement because it requires evaluation of the constraint function c to determine whether proposed point is feasible [Zhang et al., 2020]. Besides, Li and Belta [2019]; Cheng al. [2019]; Liu el al. [2020] provide a local policy search via the barrier function. The key idea the proposed CUP is parallel to Barrier functions. When updating policy according samples, local policy search (19) requires off-policy evaluation [Achiam el al. 2017], which is very challenging for high-dimension control problem [Duan er al., 2016; Yang al., 2018, 2021a].
way to solve CMDP (3) is Lagrangian approach that is also known as primal-dual problem:
(T., A.) arg min max (J(TT) ( b)}
(20)
X Eolll
Although extensive canonical algorithms are proposed to solve problem (20), e.g. [Liang et al., 2018:
Tessler et al., 2019; Paternain et al., 2019; Le et al., 2019; Russel et al., 2020; Satija et al., 2020;
7

---

Chen et al., 2021], the policy updated by Lagrangian approach may be infeasible w.r.t. CMDP (3). This is hazardous in reinforcement learning when one needs execute the intermediate policy (which may be unsafe) during training [Chow el al., 2018].
Constrained Policy Optimization (CPO). Recently, Achiam al. [2017] suggest replace the cost constraint with a surrogate cost function which evaluates the constraint (T(a) according to the samples collected from the current policy T see (21)-(33). For a given policy To. CPO [Achiam et al., 2017] updates new policy T0 as follows:
n = arg max sollle
T sondt ())a))o(-(k)) ATO, (s,
(21)
s.t. (TTx) + 1
E8 snnd ())
ko (s, Sb
(22)
DKL(Te: (TT)) = E bed
den (-) [ To [ssl] S.
(23)
Existing recent works (e.g., [Achiam er al., 2017: Vuong er al., 2019; Yang et al., 2020; Han er al., 2020; Bisi et al., 2020; Bharadhwaj et al., 2021J) try to find some convex approximations to replace the term Ano, (x, and (TI, Te,) in (24)-_))) Such first-order and second-order approximations turn non-convex problem (24)-(26) to be convex problem, it seems to make simple solution, but this approach results in many error sources and troubles in practice. Firstly, still lacks a theory analysis to show the difference between the non-convex problem (24)-(26) and its convex approximation. Policy optimization a typical non-convex problem [Yang et al., 2021b]; its convex approximation may introduce some error for its original issue. Secondly, CPO updates parameters according to conjugate gradient [Süli and Mayers, 2003], and its solution involves the inverse Fisher information matrix, which requires expensive computation for each update.
Instead of using convex approximation for the objective function, the proposed CUP algorithm improves CPO and PCPO at least two aspects. Firstly. the CUP directly optimizes the surrogate objective function via the first-order method, and it does not depend on any convex approximation. Thus, the CUP effectively avoids the expensive computation for the inverse Fisher information matrix. Secondly, CUP extends the surrogate objective function to GAE. Although Zhang er al. [2020] has used the GAE technique in experiments, to the best of our knowledge, it still lacks a rigorous theoretical analysis involved GAE before we propose CUP.
Experiment
In this section, we aim to answer the following three issues:
Does CUP satisfy the safety constraints in different environments? Does CUP performs well with different cost limits?
How does CUP compare to the state-of-the-ee safe RL algorithms?
Does CUP play sensibility during the hyper- paramerrrsmeters in the tuning processing?
We train different robotic agents using five MuJoCo physical simulators [Todorov es al., 2012] which are open by OpenAl Gym API [Brockman et al., 2016], and Safety Gym [Ray et al., 2019]. For more details, see Appendix H.2. Baselines includes CPO [Achiam et al., 2017], PCPO [Yang et al., 2020]. TRPO Lagrangian (TRPO-L), PPO Lagrangian (PPO-L) and FOCOPS [Zhang et al., 2020]. TRPO-L and PPO-L are improved by [Chow et al. 2018; Ray et al., 2019], which are based on TRPO [Schulman et al., 2015] and PPO [Schulman el al., 2017]. These two algorithms use the Lagrangian method [Bertsekas, 1997], which applies adaptive penalty coefficients satisfy the constraint.
6.1 Evaluation CUP and Comparison Analysis
We have shown the Learning curves for CUP, and other baselines in Figure 1-2, and Table I sum- marizes the performance of all algorithms. Results show that CUP quickly stabilizes the constraint return around the limit value while converging the objective return to higher values faster. In most cases, the traces of constraint from CUP almost coincide with the dashed black line of the limit. By contrast, the baseline algorithms frequently suffer from over or under the correction.
8

---

Aroov
Hepparwa
Humanadto-a
Swimmmerv
HumandidCrdde
Antra
Hopparea
Humanaiioo
Swinmmeny
HumanoldCrcce
CPO
2P04
TRPP
FOCOPS
PCPD
CUP
Figure 1: Comparison of CUP baseline algorithms over 10 seeds on Mujoco.
Safeepp-rointialll..o
Sefere CariGoll Lvv
Sefeup Carltt--o
CaMerp-CarCoel1-0
Condia
CPO
PPG-L
TSPO-L
TOcoP5
PCh0
CUP
Figure 2: Comparison CUP to to baseline algorithms over seeds on Satety- Gym.
From Figure know initial policies of the baseline algorithms are not guaranteed to be feasible, such as in wwimmer-v3,, while CUP performs the best and keeps safety learning in Swimmer-v3 tasks. In the HumanoidCirele task, all the algorithms learn steadily to obtain a safe policy, except PPO-L. Additionally, we observed that CUP brings the policy back to the feasible range faster than other baselines in the HumanoidCirele task. the Ant-v3 task, only the FOCOPS and the proposed CUP learn safcly, and both CPO and TRPO-L violate the safety constraints significantly. Besides, although FOCOPS and CUP converge to a safe policy, CUP obtains a better reward performance than FOCOPS in the Ant-v3 task. The result of Figure is relatively complex, the initial policies of the CPO and and Dafexp-PointButton piitoon-vdl 1-v0. We think it is not accidental, but it partially provides corroboration of the previous discussions in Appendix B. Both CPO and PCPO use first-order and second-order approximation to approximate a convex problem, which inevitably produces significant deviation from the original RI. problem, and it is more serious in large-scale and complex control systems.
PCPO are not guaranteed to feasible on both Safexp-PointGoall-
non-convex problem
From Table 1, we know although PPO-I achieves reward of 35.58 + 5.68 outperforms CUP in Swimmer-v3. v3, PPO-L obtains cost of 54.91 + 3.93 that violates the cost limit of 24.5 significantly, which implies PPO-L learns a dangerous policy under this setting. On the other hand, Figure I has shown that CUP generally gains higher returns than different baselines while enforcing the cost constraint. Mainly, CUP achieves reward performance of 2025.56 + 122.35 that significantly outperforms all the baseline algorithms. Additionally, after equal iterations, CUP performs a greater speed of stabilizing the constraint return around the limit value and is quicker to find feasible policies to gain a more significant objective return.
6.2 Sensitivity Analysis for Hyper-Parameeter Tuning
flyper-paramette tuning is necessary to achieve efficient policy improvement and enforce constraints. We investigate the performance with respect to the parameters: step-size a, and cost limit b. From Figure (a), we know if the estimated cost under the target threshold h, then keeps calm, which implics is not activated. Such empirical phenomenon gives significant expression to the Ant-v3,

---

Environmantt
CPO
TRPO-L
PPO-L
PCPO
ROCOPS
CUP
Ami-43
Return
1000.17 1. 8.15
450.86
161.05
1012 4 17.26
90.83 17.06
1002.53 d: 17.40
1743.06 40.5
limil: 103
Constraint
120.76 4: 1.80
131,07 4 67.9
112.5 = 15.48
174.80 = 2.55
101.31 4 0.41
99.11 0.93
Hopper-v3
815.89
285.17
1926.49 10.88
16102
211.00
22
1087.72
2025.06
cost limit 83
Constraint
76.6 10.02
40.36 4.75
83.28 4 33.19
36.63
12.54
102.3 1.455
79.98 2,306
Swimmer-x:
Reram
18.77 + 6.56
27 35 10.07
36.58
+ 5,68
37.73 = 3.36
28. 4.30
33.38 01.54
limit 24.5
Constraint
42.07 4 3.91
49.58
7.46
54.91 3.90
74.39 22.71
26.54
24.31 0.052
Humanold-vá
Reram
326.45
16.00
24.71
322
25.54
962.
57.34
642.5
1066.833 266.12
cost limit: 20.0
Constrintt
26. 13 2.13
18
3.04
22:34
4.54
066
3.52
20.04
19:91 0.36
Humanoid-Cain Circle
Retara
237.54 54 23
35.145 47.06
243.35 = 37.90
525 23
48.32
713.04 - 9.25
768.06 +: 63.70
cost limit: 50.0
Constraint
43.64
53.77 1.48
11.17 3,98
50,50
4.57
47.73 4
48. 0.65
Table I: Average results for baseline algorithms and CUP over 10 seeds the last 500 iterations.
>
Comsraint
(a) Cost constraint with respect hyper-parameter (defined in Projection step).
Retun
Conntaam
(b) Performance W.E.l penalty factor
(c) Performance w.r.t. cost limit.
Figure 3: Sensitivity analysis for hyper hyper- parameters tuning with respect to v, a and cost limit.
Humanoid-v3, and Hopper-v3 environments. While if the estimated cost exceeds the target threshold 6, will be activated, which requires the agent to play a policy on the safe region. Those empirical results are consistent with the update rule of y: Ve+1 {i/ + n b)}+, which implies the projection CUP plays an important role for the agent to learn a safe policy. Additionally, Figure (a) provides visualization way to show the difficulty of different tasks, where the task actives much quantification of v, such a task is more challenging to obtain a safe policy. Furthermore, Figure 3 (b) shows that the performance of CUP is still very stable for different settings of a, and the constraint value of CUP also still fluctuates around the target value. The different value achieved by CUP in different setting a affected by the simulated environment and constraint thresholds, which are easy to control. Finally, Figure 3 (c) shows that CUP learns safe policies stably under the cost limit thresholds. We compare policy performance and cost under different cost limit settings. For example, in the Swimmer-v3, v3, we set cost limit b among {10, 15. 20, 25, 30). Different cost limit setting implies different difficulty for learning, results show that CUP is scalable to various complex tasks, which means CUP is robust to different cost limit settings for various safe RL tasks.
7 Conclusion
This paper proposes the CUP algorithm with a theoretical safety guarantee. We derive the CUP based on the newly proposed surrogate functions with respect to objectives and constraints and provide practical implementation of CUP that does not depend on any convex approximation. We compared CUP against comprehensive list of safe RL baselines a wide range of tasks, which shows the effectiveness of CUP where the agent satisfies safe constraints.
Acknowledgements
The National Key R&D Program of China (2021ZZ020000))Naturr) Science Foundation of China (No. 61925603, and No.62102349).
10

---

References
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of International Conference Machine Learning (ICML), volume 70, pages 22-31, 2017.
M Mehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A survey, arXiv preprint arXiv:2101 00200600666..,. 2021.
Eitan Altman. Constrained Markov decision processes. CRC Press, 1999.
Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, 6(5y679
684, 1957.
Dimitri P Bertsekas. Nonlinear programming of the Operational Research Society, 48(3):334- 334, 1997.
Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety critics for exploration. In International Conference on Learning Representations (ICLR), 2021.
Lorenzo Bisi, Luca Sabbioni, Edoardo Vittori, Matteo Papini, and Marcello Restelli. Risk-averse trust region optimization for reward-volatile reduction. In Christian Bessiere, editor, Proceedings of the Twenty-Nitth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 4583-4589, 2020.
Stephen Boyd and Lieven Vandenberghe. Convex optimizution. Cambridge university press, 2004.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 01540, 2016.
Yi Chen, Jing Dong, and Zhaoran Wang. A primal-dual approach to constrained markov decision processes. arXiv preprint arXiv:2101. 10895, 2021.
Richard Cheng, Gábor Orosz, Richard M. Murray, and Joel W. Burdick. End-to-end safe reinforce- ment learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the Thirty-Thine AAAI Conference on Artificial Intelligence. AAAI Press, 2019.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman and Mohammad Ghavamzadeh. A Iyapunov- based approach to safe reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Imre Csiszár and János Körner. Information theory: coding theorems for discrete memoryless systems. Cambridge University Press, 2011.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv: 007000 2018.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. International Conference on Machine Learning (ICML), pages 1329-1338, 2016.
Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research (JMLR), 5(Nov):1471- 1530, 2004.
Minghao Han, Lixian Tian, Yuanand Zhang, Jun Wang, and Wei Pan. Reinforcement learning control of constrained dynamic systems with uniformly ultimate boundedness stability guarantee, arXiv preprint arXiv:200 /. 1.06882, 2020.
David R Hunter and Kenneth Lange. A tutorial on mm algorithms. The American Statistician, 58(1):30 37, 2004.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of International Conference on Machine Learning (ICMI), volume 2, pages 267-274, 2002.
11

---

Bingyi Kang, Shie Mannor, and Jiashi Feng. Learning safe policies with cost-sensitive advantage estimation, 2021. attps://openrevee http:/ppwwrrreeo...eooooordrddd/ wet.://oruum./dd-aaaaaaaaaa..
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Interna-
lional Conference on Machine Learning (ICML), pages 3703-3712, 2019.
Xiao Li and Calin Belta. Temporal logic guided safe reinforcement learning using control barrier functions. arXiv preprint urXiv:1903. 09990,98855 2019.
Qingkai Liang, Fanyu Que, and Eytan Modiano, Accelerated primal-dual policy optimization for safe reinforcement learning. arXiv preprint crXiv:/802.064480 2018.
Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior- policy optimization under constraints. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04), 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
OpenAl. Openai five defeats dota 2 world champions, 2019. https://openai.com Seenai--ive--defetts----
Santiago Paternain, Luiz FO Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained reinforcement learning has zero duality gap. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Jan. Peters and Stefan. Schaal. Reinforcement learning of motor skills with policy gradients. Neural Netw, 21(4):682-66 697, 2008.
M. Pirotta, M. Restelli, A. Pecorino, and D. Calandriello. Safe policy iteration. In International Conference on Machine Learning (ICML), pages 307-315, 2013.
Martin I. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforce- ment Learning. 2019.
Reazul Hasan Russel, Mouhacine Benosman, and Jeroen Van Baar. Robust constrained-mdps: Soft- constrained robust policy optimization under model uncertainty. arXiv preprint arXiv:2010.04870, 2020.
Harsh Satija, Philip Amortila, and Joelle Pineau. Constrained markov decision processes via backward value functions. In International Conference Machine Learning (ICML), pages 8502 1511, 2020.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICML), pages 1889-1897, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. International Conference on Learning Representations (ICLR), 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X1v:1707.06347, 2017.
David Silver, Julian Schrittieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676)3354, 2017.
Endre Süli and David Mayers. introduction to numerical analysis. Cambridge university press, 2003.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 1998.
12

---

Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NeurIPS), pages 1057-1063, 2000.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. International Conference on Learning Representation (ICLR), 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
Quan Vuong, Yiming Zhang, and Keith W Ross. Supervised policy update for deep reinforcement learning. International Conference on Learning Representation (ICLR), 2019.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229- 256, 1992.
Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel. Variance reduction for policy gradient with action-dependent factorized baselines. International Conference on Learning Representation (ICLR), 2018
Long Yang. Minhao Shi, Qian Zheng, Wenjia Meng, and Gang Pan. unified approach for multi-step temporal-difference learning with eligibility traces in reinforcement learning. In Proceedings of the Swenty-Sevenne International Joint Conference on Artificial Intelligence, IJCAI-18, pages 2984 2990, 2018.
Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection- based constrained policy optimization. In International Conference on Learning Representation (ICLR), 2020.
Long Yang, Gang Zheng, Yu Zhang, Qian Zheng, Pengfei Li, and Gang Pan. On convergence of gradient expected sarsa (A). In AAAI, 2021.
Long Yang, Qian Zheng, and Gang Pan. Sample complexity of policy gradient finding second-order stationary points. In AAAI, 2021.
Long Yang, Gang Zheng, Haotian Zhang, Yu Zhang, Qian Zheng, and Gang Pan. Policy optimization with stochastic mirror descent. Association for the Advancement of Artificial Intelligence (AAAI), 2022.
Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020.
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. You are strongly encouraged include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data proprietary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Abstract and Section
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We ensure our paper to conform the ethics review guidelines.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section and Section 4.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix E and Appendix G.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See the URL in the supplementary material for the code, and see H for environments of experiments.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix H
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Appendix H
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
We provide the code for our implementation of CUP in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We use open source safe reinforcement learning environments, see Appendix H.2
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] Our data does not contain any personally identifiable information or offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "mZQtvMZG5dp", "reviewer": "Reviewer_px3X", "summary": "1. A new GAE style surrogate function for solving CMDP problem\n2. The theory covers the previous well-known results\n3. Do not require to compute the second oder information, unlike CPO and PCPO.\n", "questions": "Please answer my questions in the weaknesses. ", "limitations": "1. The similarity to PCPO", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n1. The theory looks technically-sound\n2. The related work section is good. I like the comparison table of different prior algorithm shown in the appendix.\n3. In the experiment section, the claim that PCPO and CPO uses the convex relaxation results of larger errors is back up by the experiment in figure 2.\n4. Nice ablation studies on the projection parameter $v$, the step size, and the cost penalty. It clearly shows that the algorithm is stable, easy to optimize, and can follow the cost constraint threshold well no matter how hard the threshold is.\n\nCons\n1. In algorithm 1 in the appendix, the protection step requires to compute the KL divergence between the old policy and the target policy, does this require to compute second order information to do an update? If this is the case, then the contribution of using less computation would not hold.\n2. The idea of using projection has been extensively used in PCPO paper. The difference between PCPO and this work is the theoretical analysis. In addition, formulating the GAE problem has been discussed in Zhong et al paper. The difference between Zhong et al and this work is again the theoretical analysis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LLrJkTjQoEQ", "reviewer": "Reviewer_8pzD", "summary": "This work extends the performance difference bounds for Constrained Policy Optimization (CPO, Achiam et al, 2017) to a generalized setting that includes GAE in the bound. This provides a theoretical justification for using GAE in CPO-like methods as well as allows for a new approach, CUP, that explicitly uses these new general bounds with an implementation that does not depend on the convex approximation. The authors then validate their new approach on a variety of safe RL tasks and baselines and show the effectiveness of their approach.  ", "questions": "- Significance of Remark 1 is not clear. Comparison with Kakade has already been made in CPO. The generalized bounds with GAE don't really provide any new insights w.r.t. that comparison or do they? Will it be better to instead have a remark on comparison with CPO instead? \n ", "limitations": "The authors don't really discuss the limitations of their approach. Are there any scenarios where it makes sense (say computationally) to use any of the other baseline rather than the CUP approach?", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths :\n\n- The results in the paper are original and bridge the missing justification for the use of GAE in CPO-based methods. The paper is also mostly well written for the most part. \n\n- The empirical study in the paper is done rigorously which further demonstrates the effectiveness of their method. Moreover, the code is provided in the supplemental that will also help future research in this area.\n\n# Weaknesses:\n\n- The aspects of the paper can be improved when it comes to clarity. For instance, one of the major contributions of the work is how the optimize the objective function via the first-order method (L:208-212, L:245), however, most of the details regarding this have been pushed to the Appendix instead. I would recommend the authors to expand a bit more on these aspects. Though the results are novel, most of the proof methodology seems to stem directly from the CPO proof techniques. Maybe some of the details that are direct extensions of CPO like L:177-182 can be moved to the Appendix instead and space can be provided to move the Algorithm to the main paper.\n\n- More details should be provided on the comparison with FOCOPS. It seems like FOCOPS is the most competitive baseline, with some similarities with CUP approach such as the update rule for $\\nu$ is the same as FOCOPS (L:296). In Sec 6.2, while the plots for Fig 3 (a) make sense, they are not particularly interesting as they follow the same rule as FOCOPS. A more interesting comparison could have been on the choice of initial $\\nu$ and the choice of $\\eta$.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IOUIOlcKVlS", "reviewer": "Reviewer_tK1z", "summary": "This paper proposes a new safe RL method based on the constrained updated projection framework(CUP).  The authors use CUP to generalize the surrogate objective and cost(constraints) functions and solve them by the first-order method, which facilities the safety analysis for RL. The authors also compared against some safe RL baselines to show their effectiveness. ", "questions": "Please refer to the detailed weakness points. ", "limitations": "Missing a significant literature review and comparison. \nThe claim of safety guarantees is too strong for the paper. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper derives the theoretical bounds for the surrogate functions of objective and constraints to achieve monotonic policy improvement with an asymptotic safety guarantee. They claimed this work is the first to show a rigorous theoretical analysis to extend the surrogate functions w.r.t GAE. \n\nWeaknesses:\n1. The paper may miss a significant part of the literature review and experiment comparison.  I did a quick search for the related works. \na. CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee(last year's ICML paper), claims it can achieve optimality and zero constraints violation in a sub-linear rate. Although the authors mentioned it in the paper briefly, they didn't compare it in the experiment. In CRPO, the baselines include primal-dual optimization (PDO) and interior point optimization (IPO), the author should explain why they didn't compare with these methods either.\n\nMy understanding of safe RL does not only include the solutions for the constrained optimization problem(like CPO) but also some methods leveraging the formal methods(such as barrier function, and control barrier function, temporal logic), which this paper didn't cover at all. Some references here:\nb. IPO: Interior-point Policy Optimization under Constraints (by adding the barrier penalty)\nc. End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks.\nd. Temporal Logic Guided Safe Reinforcement Learning Using Control Barrier Functions.\n\n2. Another major concern is how the safety 'guarantee' is expressed in this paper. For example, in CRPO, they show that the safety violation is in a sub-linear rate while this paper only claimed they can have asymptotic constraint satisfaction, without quantitative results. It is even worse when compared to the formal methods-based approaches where they can really achieve rigorous safety guarantees.  Thus, the claim of the safety guarantee in the paper is too strong.  \n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Cq_4llzffxw", "reviewer": "Reviewer_iQjH", "summary": "This paper proposes a novel strategy for safe reinforcement learning in constrained MDPs (cMDPs). At a high level, they provide a bound on performance difference that, compared to (Achiam et al., 2017), accounts for the approximation error when applying a Taylor approximation. Then, they propose a novel algorithm based on their theory, and experimentally evaluate its performance.", "questions": "Can you clarify the novelty compared to (Achiam et al., 2017), both in terms of the theory and the algorithm?", "limitations": "None", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n- Important problem\n\nCons\n- Incremental contribution compared to (Achiam et al., 2017)\n\nThe theoretical contributions in this paper appear a bit limited. Compared to prior work (Achiam et al., 2017), they are only providing guarantees on the approximation error. Intuitively, such a guarantee should follow pretty straightforwardly from Taylor’s theorem. If there are additional technical challenges that must be overcome, the authors should make this clear.\n\nThe algorithmic contribution is also unclear. They claim to propose a new algorithm that performs a projection based on their theory to ensure correctness. However, prior work (Achiam et al., 2017) uses a similar strategy. I’m wondering if the authors can clarify how their proposed approach differs from this existing strategy.\n\nFinally, their experimental results also appear mixed. On MuJoCo, they do perform well, but there are several cases where existing baselines outperform their approach. The same issue holes for SafetyGym; here, they perform best on 3/4 environments, but CPO apparently significantly outperforms their approach on CarGoal1-v0 (though it is hard to tell since they cut off the graph).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept