LYAPUNOV-BASED SAFE POLICY OPTIMIZATION
Year: 2018

Abstract:
In many reinforcement learning applications, it is crucial that the agent interacts with the environment only through safe policies, i.e., policies that do not take the agent to certain undesirable situations. These problems are often formulated as a constrained Markov decision process (CMDP) in which the agent’s goal is to optimize its main objective while not violating a number of safety constraints. In this paper, we propose safe policy optimization algorithms that are based on the Lyapunov approach to CMDPs, an approach that has well-established theoretical guarantees in control engineering. We first show how to generate a set of state-dependent Lyapunov constraints from the original CMDP safety constraints. We then propose safe policy gradient (PG) algorithms that train a neural network policy using deep deterministic policy gradient (DDPG) or proximal policy optimization (PPO), while guaranteeing near-constraint satisfaction at every policy update by projecting either the policy parameter or the action onto the set of feasible solutions induced by the linearized Lyapunov constraints. Unlike the existing (safe) constrained policy gradient algorithms, ours are more data efficient as they are able to utilize both on-policy and off-policy data. Furthermore, the action-projection version of our algorithms often leads to less conservative policy updates and allows for natural integration into an end-to-end PG training pipeline. We evaluate our algorithms and compare them with constrained policy optimization (CPO) and the Lagrangian method on several high-dimensional continuous state and action simulated robot locomotion tasks, in which the agent must satisfy certain safety constraints while minimizing its expected cumulative cost.

heading: 1 INTRODUCTION
text: In many real-world reinforcement learning (RL) problems, the agent must satisfy a number of constraints while minimizing its expected cumulative cost. In particular, these constraints might be related to safety, i.e., the policy learned by the agent should not take it to certain undesirable parts of the state and/or action space. In some problems, it is not only important that the policy learned by the agent (that will be deployed) be safe (Amodei et al., 2016), but also crucial that all the policies generated during training (with which the agent interacts with the environment to collect samples) be safe (Achiam et al., 2017). For example, a robot should always avoid taking actions that irrevocably harm its hardware. We often formulate the agent’s interaction with the environment in constrained sequential decision-making problems as a constrained Markov decision process (CMDP). CMDPs are extensions of MDPs in which, in addition to the original cost function, there exists a constraint cost function, whose expected cumulative value should remain bounded. The additional constraint cost function gives more flexibility to CMDPs in modeling problems with trajectorybased constraints compared to approaches that artificially re-cast the cost of MDPs to enforce the constraints (Regan & Boutilier, 2009). Under the CMDP framework, we consider a policy to be safe if it satisfies the (expected) cumulative cost constraints.
A common approach to solve CMDPs is to use the Lagrangian method (Altman, 1998; Geibel & Wysotzki, 2005) that augments the original objective function with a penalty on constraint violation and computes the saddle-point of the constrained policy optimization via primal-dual method (Chow et al., 2017). Although safety is ensured when the policy converges asymptotically, a major drawback of this approach is that it makes no guarantee with regards to the safety of the policies generated during training. To address this issue (safety during training), heuristic algorithms for policy search with safety constraints have been proposed (e.g., Uchibe & Doya 2007). While some of them work reasonably well in practice, none is theoretically grounded to generate constraint-satisfying policies. Achiam et al. (2017) recently proposed the constrained policy optimization (CPO) method that extends the trust-region policy optimization (TRPO) algorithm (Schulman et al., 2015a) to handle CMDP constraints. Although there is no theoretical guarantee that the policies generated by CPO are all safe, the empirical results reported in the paper are promising in terms scalability, performance,
and constraint satisfaction, both during training and after convergence. Moreover, CPO is based on a principled approach to generate constraint-satisfying policies, and not just on heuristics. However, the CPO methodology is closely connected to TRPO. While adopting this methodology to PPO for constrained optimization is straight-forward (this is exactly how we derive our SPPO algorithm in this paper), it is not clear how it can be combined with algorithms that do not belong to the family of proximal PG algorithms (PG algorithms that are regularized with relative entropy), such as DDPG.
Another recent approach to solve CMDPs, while remaining safe during training, is by Chow et al. (2018). This work is based on the notion of Lyapunov function that has a long history in control theory to analyze the stability of dynamical systems (Khalil, 1996; Neely, 2010). Using Lyapunov functions in RL was first studied by Perkins & Barto (2002), where they were used to guarantee closed-loop stability of an agent. In a recent work, Berkenkamp et al. (2017) used Lyapunov functions to guarantee that a model-based RL agent can be brought back to a “region of attraction” during exploration. Using the theoretical underpinnings of the Lyapunov approach, Chow et al. (2018) proposed two dynamic programming (DP) algorithms to solve a CMDP and provided theoretical analyses for the feasibility and performance of the resulted policies. The proposed algorithms correspond to the two celebrated DP algorithms: policy and value iteration. They then extended their algorithms to learning (the scenario in which the MDP model is unknown) and proposed RL algorithms that correspond to approximate policy and value iteration. However, since their algorithms are all value-function-based, applying them to continuous action problems is not straightforward.
In this paper, we extend the Lyapunov-based approach to solving CMDPs of Chow et al. (2018) to continuous action problems that play an important role in control and robotics. Our contributions can be summarized as follows: 1) We formulate the problem of safe RL as a CMDP and propose a Lyapunov-function-based policy optimization framework that can handle continuous action CMDPs. 2) By leveraging the theoretical underpinnings of the Lyapunov-based approach to CMDPs in Chow et al. (2018), we present two classes of safe policy optimization algorithms that can work with any standard policy gradient algorithm such as deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015) and proximal policy optimization (PPO) (Schulman et al., 2017). The first class of our algorithms is based on constrained optimization w.r.t. policy parameter (similar to what is done in constrained policy optimization (CPO)). The second class hinges on the concept of a safety layer introduced by Dalal et al. (2018) and transforms the constrained policy optimization problem into an unconstrained one, by integrating the Lyapunov constraints into the policy network via safety-layer augmentation. 3) We evaluate our algorithms and compare them to two baselines, CPO (Achiam et al., 2017) and the Lagrangian method, on several robot locomotion tasks, in which the agent must satisfy certain safety constraints while minimizing its expected cumulative cost. Our results show that our algorithms outperform the baselines in terms of balancing the performance and constraint satisfaction at every policy update.

heading: 2 PRELIMINARIES
text: We consider the RL problem in which the agent’s interaction with the environment is modeled as a Markov decision process (MDP). A MDP is a tuple (X ,A, γ, c, P, x0), where X is the state space; A is the action space; γ ∈ [0, 1) is the discounting factor; c(x, a) ∈ [0, Cmax] is the immediate cost function (negative reward); P (·|x, a) is the transition probability distribution; and x0 ∈ X is the initial state. Our results easily generalize to random initial states and random costs, but for simplicity we will focus on the case of deterministic initial state and immediate cost. In a more general setting where cumulative constraints are taken into account, we define a constrained Markov decision process (CMDP), which extends the MDP model by introducing additional costs and associated constraints. A CMDP is defined by (X ,A, γ, c, d, P, x0, d0), where the components X ,A, γ, c, P, x0 are the same as in the unconstrained MDP; d(x) ∈ [0, Dmax] is the immediate constraint cost; and d0 ∈ R≥0 is an upper-bound on the expected cumulative (through time) constraint cost. To formalize the optimization problem associated with CMDPs, let ∆ be the set of Markov stationary policies, i.e., ∆(x) = { π(·|x) : X → R≥0s : ∑ a π(a|x) = 1 } for any state x ∈ X . For notational convenience, at each state x ∈ X , we define the generic Bellman operator w.r.t. policy π ∈ ∆ and generic cost function h as Tπ,h[V ](x) = ∑ a π(a|x) [ h(x, a)+γ ∑ x′∈XP (x ′|x, a)V (x′) ] . Given a policy π ∈ ∆ and an initial state x0, the expected cumulative cost is defined as Cπ(x0) := E [∑∞
t=0 γ tc(xt, at) | x0, π ] , and the safety constraint is defined asDπ(x0) ≤ d0, whereDπ(x0) :=
E [∑∞
t=0 γ td(xt) | x0, π
] is the safety constraint function (expected cumulative constraint cost).
The goal in CMDPs is to solve the constrained optimization problem
π∗ ∈ min π∈∆
{ Cπ(x0) : Dπ(x0) ≤ d0 } . (1)
Using the discounting property of a CMDP, Theorem 3.1 in Altman (1999) shows that if the feasibility set is non-empty, then there exists an optimal policy in the class of stationary Markovian policies ∆. To motivate the CMDP formulation, we refer the reader to Chow et al. (2018), which presents two real-world examples in modeling safety using (i) the reachability constraint, and (ii) the constraint that limits the agent’s visits to undesirable states.

heading: 3 A LYAPUNOV APPROACH FOR SOLVING CMDPS
text: In this section, we revisit the Lyapunov approach to solving CMDPs proposed by Chow et al. (2018) and report the mathematical results that are important in developing our safe policy optimization algorithms. To start, without loss of generality, we assume that we have access to a baseline feasible policy of Equation 1, πB ; i.e. πB satisfies DπB (x0) ≤ d0. We define a set of Lyapunov functions w.r.t. initial state x0 ∈ X and constraint threshold d0 as LπB (x0, d0) = {L : X → R≥0 : TπB ,d[L](x)≤ L(x),∀x ∈ X ; L(x0) ≤ d0}, and call the constraints in this feasibility set Lyapunov constraints. For any arbitrary Lyapunov function L ∈ LπB (x0, d0), we denote by FL(x) = { π(·|x) ∈ ∆ : Tπ,d[L](x) ≤L(x) } the set of L-induced Markov stationary policies. Since Tπ,d is a contraction mapping (Bertsekas, 2005), any L-induced policy π has the property Dπ(x) = limk→∞ T kπ,d[L](x) ≤ L(x), ∀x ∈ X . Together with the property that L(x0) ≤ d0, they imply that any L-induced policy is a feasible policy of Equation 1. However, in general, the set FL(x) does not necessarily contain an optimal policy of Equation 1, and thus it is necessary to design a Lyapunov function (w.r.t. a baseline policy πB) that provides this guarantee. In other words, the main goal is to construct a Lyapunov function L ∈ LπB (x0, d0) such that
L(x) ≥ Tπ∗,d[L](x), L(x0) ≤ d0. (2) Chow et al. (2018) show in their Theorem 1 that 1) without loss of optimality, the Lyapunov function can be expressed as L (x) := E [∑∞ t=0 γ t(d(xt) + (xt)) | πB , x ] , where (x) ≥ 0 is some auxiliary constraint cost uniformly upper-bounded by ∗(x) := 2DmaxDTV (π∗||πB)(x)/(1 − γ), and 2) if the baseline policy πB satisfies the condition maxx∈X ∗(x) ≤ Dmax · min{(1− γ)(d0 −DπB (x0))/Dmax, Dmax − (1− γ)D/Dmax + (1− γ)D}, where D = maxx∈X maxπ Dπ(x) is the maximum constraint cost, then the Lyapunov function candidate L ∗ also satisfies the properties of Equation 2, and thus, its induced feasible policy set FL ∗ contains an optimal policy. Furthermore, suppose that the distance between the baseline and optimal policies can be estimated effectively. Using the set of L ∗ -induced feasible policies and noting that the safe Bellman operator T [V ](x) = minπ∈FL ∗ (x) Tπ,c[V ](x) is monotonic and contractive, one can show that T [V ](x) = V (x), ∀x ∈ X has a unique fixed point V ∗, such that V ∗(x0) is a solution of Equation 1, and an optimal policy can be constructed via greedification, i.e., π∗(·|x) ∈ arg minπ∈FL ∗ (x) Tπ,c[V
∗](x). This shows that under the above assumption, Equation 1 can be solved using standard dynamic programming (DP) algorithms. While this result connects CMDP with Bellman’s principle of optimality, verifying whether πB satisfies this assumption is challenging when a good estimate of DTV (π∗||πB) is not available. To address this issue, Chow et al. (2018) propose to approximate ∗ with an auxiliary constraint cost ̃, which is the largest auxiliary cost satisfying the Lyapunov condition L̃(x) ≥ TπB ,d[L̃](x), ∀x ∈ X and the safety condition L̃(x0) ≤ d0. The intuition here is that the larger ̃, the larger the set of policies FL̃ . Thus, by choosing the largest such auxiliary cost, we hope to have a better chance of including the optimal policy π∗ in the set of feasible policies. Specifically, ̃ is computed by solving the following linear programming (LP) problem:
̃ ∈arg max :X→R≥0 {∑ x∈X (x) : d0 −DπB (x0) ≥ 1(x0) >(I − γ{P (x′|x, πB)}x,x′∈X )−1 } , (3)
where 1(x0) represents a one-hot vector in which the non-zero element is located at x = x0. When πB is a feasible policy, this problem has a non-empty solution. Furthermore, according to the derivations in Chow et al. (2018), the maximizer of Equation 3 is an indicator function of the form ̃(x) = (d0 −DπB (x0)) · 1{x = x}/E[ ∑∞ t=0 γ
t1{xt = x} | x0, πB ] ≥ 0, where x ∈ arg minx∈X E [∑∞ t=0 γ t1{xt = x} | x0, πB ] . They also show that by further restricting ̃(x) to be a constant function, the maximizer is given by ̃(x) = (1−γ)·(d0−DπB (x0)), ∀x ∈ X . Using the construction of the Lyapunov function L̃, Chow et al. (2018) propose the safe policy iteration (SPI)
algorithm (see Algorithm 1 in Appendix A) in which the Lyapunov function is updated via bootstrapping, i.e., at each iteration L̃ is recomputed using Equation 3 w.r.t. the current baseline policy. This algorithm has the following properties: 1) Consistent Feasibility, i.e., if the current policy πk is feasible, then πk+1 is also feasible; 2) Monotonic Policy Improvement, i.e., Cπk+1(x) ≤ Cπk(x) for any x ∈ X ; and 3) Asymptotic Convergence. Despite all these nice properties, SPI is still a valuefunction-based algorithm, and thus it is not straightforward to use it in continuous action problems. The main reason is that the greedification step becomes an optimization problem over the continuous set of actions that is not necessarily easy to solve. In Section 4, we show how we use SPI and its nice properties to develop safe policy optimization algorithms that can handle continuous action problems. Our algorithms can be thought as combinations of DDPG or PPO (or any other on-policy or off-policy policy optimization algorithm) with a SPI-inspired critic that evaluates the policy and computes its corresponding Lyapunov function. The computed Lyapunov function is then used to guarantee safe policy update, i.e., the new policy is selected from a restricted set of safe policies defined by the Lyapunov function of the current policy.

heading: 4 SAFE POLICY GRADIENT ALGORITHMS WITH LYAPUNOV FUNCTIONS
text: Policy gradient (PG) algorithms optimize a policy end-to-end by computing sample estimates of the gradient of the cumulative cost induced by the policy and then updating the policy in the gradient direction. In general, stochastic policies that give a probability distribution over actions are parameterized by a κ-dimensional vector θ, so the space of policies can be written as{ πθ(·|x), x ∈ X , θ ∈ Rκ } . Since in this setting a policy π is uniquely defined by its parameter vector θ, policy-dependent functions can be written as a function of θ or π, and they are used interchangeably in this paper.
Recently there are two PG algorithms which have emerged as generally well-performing, namely DDPG and PPO. These algorithms are widely used in many continuous control tasks. DDPG (Lillicrap et al., 2015) is an off-policy Q-learning style algorithm. A deterministic policy πθ(x) and a Q-value approximator Q(x, a;φ) are trained jointly. The Q-value approximator is trained to minimize Bellman errors with respect to π; i.e., it is trained to fit the true Q-value function satisfying Q(x, a) = c(x, a) + γ ∑ x′∈X P (x
′|x, a)Q(x′, πθ(x′)). The policy π is then trained to optimize Q(x, πθ(x);φ) via chain-rule. The PPO algorithm we use is a penalty form of TRPO (Schulman et al., 2017) with an adaptive rule to tune the DKL penalty weight βk. Specifically, PPO trains a Gaussian policy πθ(x) according to the standard policy gradient objective augmented with a penalty on KL-divergence from a previous version of the policy; i.e., the penalty is of the form, DKL(θ, θ
′) = E[DKL(πθ′(·|xt)||πθ(·|xt))|x0, θ′], where θ′ is a previous version of the policy. In CMDPs, the presence of a constraint Dπθ (x0) ≤ d0 may be naively incorporated into the standard forms of DDPG and PPO via the Lagrangian method. That is, one may transform the constrained optimization problem to a penalty form, in which the constraint costs d(x) are added to the task costs c(x, a). The resulting penalized form of the objective is minθ maxλ≥0 E [∑∞ t=0 c(xt, at) + λd(xt)|x0, θ ] − λd0. In this form, both θ and λ must be optimized jointly to find a saddle-point of the objective. The optimization of θ may be performed by either DDPG or PPO on the augmented cost c(x, a) + λd(x). The optimization of λ may be performed by stochastic gradient descent on xt taken from trajectories sampled according to θ.
Although the Lagrangian approach is easy to implement (see Appendix B for details), in practice it does not lead to safety in training. While the objective encourages finding a solution which is safe, any intermediate step in the optimization may lead to an unsafe policy. In contrast, the Lyapunov approaches we propose are guaranteed to return a safe policy, not only at convergence, but also during training. In the subsections below, we elaborate how to transform DDPG and PPO to their Lyapunov safe counterparts below:
θ∗ = arg min θ Cπθ (x0) subject to ∫ a∈A (πθ(a|x)−πB(a|x))QL(x, a)da ≤ ̃(x), ∀x ∈ X . (4)
where QL(x, a) = d(x) + ̃′(x) + γ ∑ x′ P (x
′|x, a)L̃′(x′) is the state-action Lyapunov function. We will describe two approaches to incorporating Lyapunov constraints in PG: θ-projection and a-projection. In Section 4.1, we formulate the Lyapunov-based PG using constrained policy optimization (which we call θ-projection), and in Section 4.2 we show how the Lyapunov constraints can be embedded into the policy network via a safety layer (which we call a-projection). In Section A.1, we also discuss two practical techniques to further enforce safety during policy training.

heading: 4.1 CONSTRAINED OPTIMIZATION APPROACH TO LYAPUNOV-BASED PG
text: For demonstration purposes, we hereby show how one can perform constrained policy optimization with PPO and Lyapunov constraints. With almost identical machinery, this procedure can also be applied to DDPG. Consider the following constrained optimization at iteration k with semi-infinite dimensional Lyapunov constraints for policy update:
θ ∈ arg min θ∈Θ
〈(θ − θk),∇θEx∼µθk ,a∼πθ [ Qθk (x, a) ] + βk〈(θ − θk),∇2θDKL(θ||θk) |θ=θk ·(θ − θk) |θ=θk 〉
s.t. 〈(θ − θk),∇θEa∼πθ [ QLθk (x, a) ] |θ=θk 〉 ≤ ̃(x), ∀x ∈ X ,
where µθk is the γ-visiting distribution w.r.t. πθk , and βk is the adaptive penalty weight of the DKL(θ||θk) regularizer. Clearly if one updates the policy parameter by solving the above optimization, and if the approximation errors from neural network parameterizations of Qθk and QLθk and first-order Taylor series expansion are small, then safety may ensure safety during training. However, the presence of infinite-dimensional Lyapunov constraints makes solving the above optimization (in real-time) numerically intractable. To tackle the issue of infinite dimensionality, (without loss of optimality) we re-write the Lyapunov constraint in the following form: maxx∈X 〈(θ − θk),∇θEa∼πθ [ QLθk (x, a) ] |θ=θk〉 − ̃(x) ≤ 0. This might still lead to numerical instability in gradient descent algorithms, because the max-operator in the constraint is nondifferentiable. Similar to the surrogate constraint used in TRPO (to transform the maxDKL constraint into an average DKL constraint), a more numerically stable way is to approximate the Lyapunov constraint using the following average constraint surrogate:
〈(θ − θk), 1
M M∑ i=1 ∇θEa∼πθ [ QLθk (xi, a) ] |θ=θk 〉 ≤ 1 M M∑ i=1 ̃(xi). (5)
where N is the number of on-policy trajectories of πθk . In practice, if one adopts ̃ = (1 − γ)(d0 − Dπθk (x0)) from Section 4.1, then the linear term in Equation 5 can be simplified as ∇θEa∼πθ [ QLθk (xi, a) ] = ∇θ ∫ a∈A πθ(a|x)∇θ log πθ(a|x)QD,θk(xi, a)da. On the
other hand, by setting M = O(1/(1 − γ)), the constraint threshold becomes 1M ∑M i=1 ̃(xi) ≈ d0 − Dπθk (x0). Collectively, the average constraint surrogate in Equation 5 becomes 〈(θ − θk), 1 M ∑M i=1∇θEa∼πθ [ QD,θk(xi, a) ] |θ=θk〉 ≤ d0 − Dπθk (x0), which is equivalent to the constraint used in the CPO algorithm (see Section 6.1 in Achiam et al. (2017)). This draws the connection between CPO and Lyapunov-based PG with θ-projection.
The Lyapunov-based algorithms with θ-projection in constrained policy update is summarized by Algorithm 4 in Appendix A. In the experiment section, we denote the DDPG version and the PPO version of this algorithm by SDDPG and SPPO respectively.

heading: 4.2 EMBEDDING LYAPUNOV CONSTRAINTS INTO A SAFETY LAYER
text: Notice that the main contribution of the Lyapunov approach is to break down a trajectory-based constraint into a sequence of single-step, state dependent constraints. When the state space is infinite/continuous, it is counter-intuitive to directly enforce these Lyapunov constraints (instead of the original trajectory-based constraint) in the optimization w.r.t. policy parameter θ, because the feasibility set is characterized by infinite dimensional constraints. Rather, leveraging the ideas of a safety layer from Dalal et al. (2018) that was applied to single-step constraints, we propose a novel approach to embed the set of Lyapunov constraints into the policy network. In this way, one reformulates the CMDP problem into an unconstrained one, whose policy parameter θ (of the augmented network) can then be optimized by any standard unconstrained PG algorithms. At every given state, the unconstrained action is first computed and is then passed through the safety layer, where a feasible action mapping is constructed by projecting the unconstrained actions onto the feasibility set w.r.t. the corresponding Lyapunov constraint. Therefore, safety during training w.r.t. the original CMDP problem is guaranteed by the Lyapunov theorem.
We hereby demonstrate how one can find a feasible action mapping using the safety layer with DDPG, whose role is to solve the following projection problem at given state x ∈ X :
a∗(x) ∈ arg min a
{ 1
2 ‖a− πθ,unc(x)‖2 : (a− πB(x))>∇aQL(x, a) |a=πB(x)≤ ̃(x)
} . (6)
In the above optimization problem, πθ,unc is the unconstrained policy whose policy parameter is updated by standard DDPG, πB is the current data-generation policy that is safe, and the left side of the
constraint is the first-order Taylor series approximation of ∫ a∈AQL(x, πθ(x)) − QL(x, πB(x))da over actions, w.r.t. the action πB(x) induced by the baseline policy. As in Section 4.1, since the auxiliary cost ̃ is state-dependent, one can readily find ∇aQL(x, a) |a=πB(x) by computing the gradient of the constraint action value function ∇aQD(x, a) |a=πB(x). Generally, the safety layer perturbs the unconstrained action as little as possible in the Euclidean norm in order to satisfy the Lyapunov constraints. Notice that the objective function is positive-definite and quadratic and the constraint approximation is linear. Therefore, we can find the global solution to this convex problem effectively via an in-graph iterative QP-solver, such as the one from Amos & Kolter (2017). Furthermore, since the optimization in Equation 6 only has a single Lyapunov constraint, one can express a∗(x) using the following analytical solution. Proposition 1. At any given state x ∈ X , the solution to the optimization problem in Equation 6 has the following form: a∗(x) = πθ,unc(x) + λ∗(x)∇aQL(x, a) |a=πB(x), where
λ ∗ (x) = ((( ∇aQL(x, a) |a=πB(x) )> (πθ,unc(x)− πB(x))− ̃(x) ) / ( ∇aQL(x, a) |a=πB(x) )> ∇aQL(x, a) |a=πB(x) ) + .
The closed-form solution is essentially a linear projection of the unconstrained action πθ,unc(x) to the safe hyperplane characterized with slope ∇aQL(x, a) |a=πB(x) and intercept ̃(x) = (1− γ)(d0 − DπB (x0)). Implementing a∗(x) is very simple; it only consists of several arithmetic operations such as matrix products and ReLU. Extending this closed-form solution to handle multiple constraints is possible, under the assumption of having at most one constraint active at a time.
In general, the safety layer approach can also be applied to policy gradient algorithms, such as PPO, that learn a non-deterministic policy. For example in the PPO case when the policy is parameterized with a Gaussian distribution, then one simply need to project both the mean and the standarddeviation vector, in order to obtain a feasible action probability. The Lyapunov-based algorithms with a-projection in safety layer is summarized by Algorithm 5 in Appendix A. In the experiment section, we denote the DDPG version and the PPO version of this algorithm by SDDPG-modular and SPPO-modular respectively.

heading: 5 EXPERIMENTS
text: We empirically validate the Lyapunov-based PG algorithms on several robot locomotion continuous control tasks. In these experiments we aim to address the following questions about our proposed algorithm: (i) How is the performance (in terms of cost and safety during training) of Lyapunovbased PG compared to other baseline methods such as CPO and the naive Lagrangian approach? (ii) In the presence of approximation errors in value functions and policies, how robust is Lyapunovbased PG algorithm with respect to constraint violations?
To understand the performance of these algorithms in terms of both cost and safety guarantees, we designed several interpretable experiments in simulated robot locomotion continuous control tasks, whose notions of safety are motivated by physical constraints. We consider three domains using the MuJoCo simulator (Todorov et al., 2012) with a variety of different agents: (i)HalfCheetah-Safe: The HalfCheetah agent is rewarded for running, but its speed is limited for stability and safety; (ii) Point-Circle: The Point agent is rewarded for running in a wide circle, but is constrained to stay within a safe region defined by |x| ≤ xlim (Achiam et al., 2017); (iii) Point-Gather & Ant-Gather: The agent, which is either a Point or an Ant, is rewarded for collecting target objects in a terrain map, while being constrained to avoid bombs (Achiam et al., 2017).
Visualizations of these tasks as well as more detailed descriptions (immediate cost and constraint cost functions, constraint thresholds) are given in Appendix C. In these experiments there are three different agents: (1) a point-mass (X ⊆ R9, A ⊆ R2); an ant quadruped robot (X ⊆ R32, A ⊆ R8); (3) a half-cheetah (X ⊆ R18, A ⊆ R6). For all experiments, we use two neural networks with two hidden layers of size (100, 50) and ReLU activations to model the mean and log-variance of the Gaussian actor policy, and two neural networks with two hidden layers of size (200, 50) and tanh activations to model the critic and constraint critic. To build a low variance sample gradient estimate, we use GAE-λ (Schulman et al., 2015b) to estimate the advantage and constraint advantage functions, with a hyper-parameter λ ∈ (0, 1) optimized by grid-search.

heading: 5.1 COMPARISON WITH UNCONSTRAINED PG, LAGRANGIAN APPROACH, AND CPO
text: For comparison purposes, we implement the naive Lagrangian approach. Details of this algorithm are available in Appendix B. For fair comparisons, we also optimize the Lagrangian using a natural
policy gradient when needed. To understand the optimal unconstrained performance on these environments, we also include the learning performance of two state-of-the-art unconstrained reinforcement learning algorithms, namely DDPG (Lillicrap et al., 2015) and PPO (Schulman et al., 2017). In the PPO case, we aim to compare the performance of CPO (which is equivalent to Lyapunov-based PG with θ-projection) with Lyapunov-based PG (with a-projection) . Notice that the original implementation of CPO in Achiam et al. (2017) is based on TRPO (Schulman et al., 2015a). Instead of the original CPO algorithm, we use its PPO alternative (which coincides with the SPPO algorithm derived in Section 4.1) as the safe RL baseline for comparison. SPPO preserves the essence of CPO by adding a 1st order constraint to the proximal policy optimization. The main difference between CPO and SPPO is that the latter does not perform backtracking line-search. The decision to compare with SPPO instead of CPO is 1) to avoid the additional computational complexity of line-search in TRPO, while maintaining the performance of PG using the popular PPO algorithm, 2) to have a back-propagatable version of CPO, and 3) to have a fair comparison with other back-propagatable safe RL algorithms, such as the DDPG and safety layer counterparts.
Learning curves for unconstrained DDPG, Lagrangian DDPG, SDDPG and SDDPG-modular are shown in Figure 1, and the learning curves for unconstrained PPO, Lagrangian PPO, SPPO and SPPO-modular are shown in Figure 6. Due to space restrictions, more results are included in Appendix C. From the comparison plots, one can clearly see that the unconstrained DDPG and PPO agents are constraint-violating in these environments. Although the Lagrangian approaches in DDPG and PPO converge to feasible policies with reasonably good performance, in accord with our earlier claims these algorithms cannot guarantee constraint satisfaction during training. Furthermore it is worth-noting that the Lagrangian approach can be sensitive to the initialization of the Lagrange multiplier λ0. If λ0 is too large, it would make policy updates overly conservative, while if λ0 is too small then constraint violation will be more pronounced. By default, we initialize λ0 = 10, which assumes no knowledge about the environment.
Generally in both DDPG and PPO experiments, the Lyapunov-based PG algorithms lead to more stable learning and constraint satisfaction than the Lagrangian approach. The Lyapunov approaches quickly stabilize the constraint cost to be below the threshold, while the constraint costs from the Lagrangian approach tend to jiggle around the threshold. In many cases the a-projection Lyapunovbased PG (SDDPG-modular, SPPO-modular) converges faster than the θ-projection counterpart (SDDPG, SPPO). This corroborates with the hypothesis that the a-projection approach is less conservative during policy updates than the θ-projection approach (which is what CPO is based on).
Finally, in most experiments (HalfCheetah, PointGather, and AntGather) the DDPG class of algorithms tends to have faster learning than the PPO counterpart. This is potentially due to the improved data-efficiency when using off-policy samples in PG updates. Although this benefit is not directly related to the addition of Lyapunov constraints, this also supports our claim that some Lyapunovbased safe PG algorithms (SDDPG, SDDPG-modular) can learn more effectively than CPO (which is analogous to SPPO).

heading: 5.2 CONSTRAINT VIOLATION DURING TRAINING
text: Due to function approximation errors in policies and value functions, in practice most safe learning algorithms including the Lyapunov-based PG methods may still take a bad step and lead to constraint violation. While methods like safeguard policy update and constraint tightening from Section A.1 might help to remedy this issue, it is still unclear how robust each algorithm is regarding constraint satisfaction during training. To study the degree of constraint violation in different PG algorithms, we show their constraint violation plots in Figure 3 for DDPG-based and PPO-based algorithms. To make the comparison fair, we apply both safeguard policy update and the constraint tightening to all safe PG algorithms. From the constraint violation plots, it is clear that the Lagrangian methods (both DDPG and PPO) violate safety constraints more often than the Lyapunov-based counterparts. In many cases the Lyapunov-based PG algorithms with a-projection have lower constraint violation than its θ-projection counterparts. We speculate this is due to the fact that the safety layer generates smoother gradient updates during end-to-end training.

heading: 6 CONCLUSIONS
text: In this paper, we formulated the problem of safe RL as a CMDP and used the notion of Lyapunov function to develop policy optimization algorithms that learn policies that are both safe and have low expected cumulative cost. Our algorithms extend the Lyapunov-based approach to solving CMDPs of Chow et al. (2018) to continuous action problems. Our algorithms combine DDPG or PPO (or any other on-policy or off-policy policy optimization algorithm) with a critic that is inspired by the safe policy iteration (SPI) algorithm of Chow et al. (2018) and both evaluates the policy and computes its corresponding Lyapunov function. The computed Lyapunov function is then used to guarantee safe policy update. We can categorize our algorithms into two classes in terms of the way they perform safe policy update. The first class is based on constrained optimization w.r.t. policy parameter, similar to what is done in CPO. The second class relies on the safety layer concept (Dalal et al., 2018) that integrates the Lyapunov constraints into the policy network by adding an action projection layer to it. We evaluated our algorithms on four high-dimensional simulated robot locomotion tasks and compared them with CPO (Achiam et al., 2017) and the Lagrangian method in terms of minimizing the expected cumulative return and constraint violation during training. Our results indicate that our Lyapunov-based algorithms 1) achieve safe learning, 2) have better data-efficiency, and 3) can be more naturally integrated within the standard end-to-end differentiable policy gradient training pipeline. In general, our work is a step forward in deploying RL to real-world problems in which safety guarantees are of paramount importance.
Future work includes 1) developing more stable and safe learning algorithms by further exploiting the properties of Lyapunov functions, 2) exploring more efficient ways to include Lyapunov constraints in constrained policy optimization, and 3) applying the Lyapunov-based PG algorithms to real-world continuous control problems, particularly in robotics.

heading: A DETAILS OF THE SAFE POLICY GRADIENT ALGORITHMS
text: Algorithm 1 Safe Policy Iteration (SPI) Input: Initial feasible policy π0; for k = 0, 1, 2, . . . do
Step 0: With πb = πk, evaluate the Lyapunov function L k , where k is a solution of Equation 3 Step 1: Evaluate the cost value function Vπk (x) = Cπk (x); Then update the policy by solving the following problem: πk+1(·|x) ∈ argminπ∈FL k (x) Tπ,c[Vπk ](x), ∀x ∈ X
end for Return Final policy πk∗
Algorithm 2 Trajectory-based Policy Gradient Algorithm for CMDP Input: parameterized policy π(·|·; θ) Initialization: policy parameter θ = θ0, and the Lagrangian parameter λ = λ0 while TRUE do
for i = 0, 1, 2, . . . do for j = 1, 2, . . . do
Generate N trajectories {ξj,i}Nj=1 by starting at x0 and following the policy θi. end for
θ Update: θi+1 = θi − α2,i 1
N N∑ j=1 ∇θ log Pθ(ξj,i)|θ=θi ( C(ξj,i) + λiD(ξj,i) ) λ Update: λi+1 = ΓΛ [ λi + α1,i ( − d0 + 1
N N∑ j=1 D(ξj,i) )]
end for if {λi} converges to λmax then
Set λmax ← 2λmax. else
return parameters ν, θ, λ and break end if
end while
Algorithm 3 Actor-Critic Algorithms for CMDP Input: Parameterized policy π(·|·; θ) and value function feature vector φ(·) Initialization: policy parameters θ = θ0; Lagrangian parameter λ = λ0; value function weight v = v0 while TRUE do
for k = 0, 1, 2, . . . do Sample ak ∼ π(·|xk; θk); Cλk (xk, ak) = C(xk, ak) + λkD(xk, ak); xk+1 ∼ P (·|xk, ak); // AC Algorithm:
TD Error: δk(vk) = Cλk (xk, ak) + γV̂φk (xk+1)− V̂φk (xk) (7) Critic Update: vk+1 = vk + ζ3(k)δk(vk)ψ(xk) (8)
θ Update: θk+1 = θk − ζ2(k)∇θ log πθ(ak|xk) · δk(vk)/1− γ (9)
λ Update: λk+1 = ΓΛ ( λk + ζ1(k) ( − d0 + 1
N N∑ j=1 D(ξj,i) ))
(10)
// NAC Algorithm: Critic Update: wk+1 = ( I − ζ3(k)∇θ log πθ(ak|xk)|θ=θk ( ∇θ log πθ(ak|xk)|θ=θk )>) wk
+ ζ3(k)δk(vk)∇θ log πθ(ak|xk)|θ=θk (11) θ Update: θk+1 = θk − ζ2(k)wk/1− γ (12)
Other Updates: Follow from Eqs. 7, 8, and 10.
end for end while
A.1 PRACTICAL IMPLEMENTATIONS OF SAFE PG Due to function approximation errors, even with the Lyapunov constraints in practice the safe PG algorithm may take a bad step and produce an infeasible policy update and cannot automatically recover from such a bad step. To tackle this issue, similar to Achiam et al. (2017)
Algorithm 4 Lyapunov-based Policy Gradient with θ-projection (SDDPG and SPPO) Input: Initial feasible policy π0; for k = 0, 1, 2, . . . do
Step 0: With πb = πθk , generate N trajectories {ξj,k} N j=1 of T steps by starting at x0 and following the policy θk Step 1: Using the trajectories {ξj,k}Nj=1, estimate the criticQθ(x, a) and the constraint criticQD,θ(x, a); • For DDPG, these functions are trained by minimizing the MSE of Bellman residual, and one
can also use off-policy samples from replay buffer (Schaul et al., 2015); • For PPO these functions can be estimated by the generalized advantage function technique from
Schulman et al. (2015b) Step 2: Based on the closed form solution of a QP problem with an LP constraint in Section 10.2 of Achiam et al. (2017), calculate λ∗k with the following formula:
λ∗k = −βk ̃− (∇θQθ(x̄, ā) |θ=θk)>H(θk)−1∇θQD,θ(x̄, ā) |θ=θk( ∇θQD,θ(x̄, ā) |θ=θk )> H(θk)−1∇θQD,θ(x̄, ā) |θ=θk  + ,
where
∇θQθ(x̄, ā) = 1
N ∑ x,a∈ξj,k,1≤j≤N T−1∑ t=0 γt∇θ log πθ(a|x)Qθ(x, a),
∇θQD,θ(x̄, ā) = 1
N ∑ x,a∈ξj,k,1≤j≤N T−1∑ t=0 γt∇θ log πθ(a|x)Qθ(x, a),
βk is the adaptive penalty weight of the DKL(π||πθk ) regularizer, and H(θk) = ∇ 2 θDKL(π||πθ) |θ=θk is the Hessian of this term Step 3: Update the policy parameter by following the objective gradient; • For DDPG
θk+1 ← θk−αk· 1 N · T ∑
x∈ξj,k,1≤j≤N
∇θπθ(x) |θ=θk ·(∇aQθk (x, a)+λ ∗ k∇aQD,θk (x, a)) |a=πθk (x)
• For PPO,
θk+1 ← θk − αk Nβk
( H(θk) )−1 ∑ xj,t,aj,t∈ξj,k,1≤j≤N T−1∑ t=0 γt · ∇θ log πθ(aj,t|xj,t) |θ=θk ·
(Qθk (xj,t,aj,t) + λ ∗ kQD,θk (xj,t,aj,t))
Step 4: At any given state x ∈ X , compute the feasible action probability a∗(x) via action projection in the safety layer, that takes inputs ∇aQL(x, a) = ∇aQD,θk (x, a) and (x) = (1 − γ)(d0 − QD,θk (x0, πk(x0))), for any a ∈ A.
end for Return Final policy πθk∗ ,
Algorithm 5 Lyapunov-based Policy Gradient with a-projection (SDDPG-modular and SPPOmodular)
Input: Initial feasible policy π0; for k = 0, 1, 2, . . . do
Step 0: With πb = πθk , generate N trajectories {ξj,k} N j=1 of T steps by starting at x0 and following the policy θk Step 1: Using the trajectories {ξj,k}Nj=1, estimate the criticQθ(x, a) and the constraint criticQD,θ(x, a); • For DDPG, these functions are trained by minimizing the MSE of Bellman residual, and one
can also use off-policy samples from replay buffer (Schaul et al., 2015); • For PPO these functions can be estimated by the generalized advantage function technique from
Schulman et al. (2015b) Step 2: Update the policy parameter by following the objective gradient; • For DDPG
θk+1 ← θk − αk · 1 N · T ∑
x∈ξj,k,1≤j≤N
∇θπθ(x) |θ=θk ·∇aQθk (x, a) |a=πθk (x);
• For PPO,
θk+1 ← θk− αk Nβk
( H(θk) )−1 ∑ xj,t,aj,t∈ξj,k,1≤j≤N T−1∑ t=0 γt·∇θ log πθ(aj,t|xj,t) |θ=θk ·Qθk (xj,t,aj,t)
where βk is the adaptive penalty weight of the DKL(π||πθk ) regularizer, and H(θk) = ∇2θDKL(π||πθ) |θ=θk is the Hessian of this term
Step 3: At any given state x ∈ X , compute the feasible action probability a∗(x) via action projection in the safety layer, that takes inputs ∇aQL(x, a) = ∇aQD,θk (x, a) and (x) = (1 − γ)(d0 − QD,θk (x0, πk(x0))), for any a ∈ A.
end for Return Final policy πθk∗ ,
we propose the following safeguard policy update rule to purely decrease the constraint cost: θk+1 = θk − αsg,k∇θDπθ (x0)θ=θk , where αsg,k is the learning rate for safeguard update. If αsg,k >> αk (learning rate of PG), then with the safeguard update θ will quickly recover from the bad step but it might be overly conservative. This approach is principled because as soon as πθk is unsafe/infeasible w.r.t. CMDP, the algorithm uses a limiting search direction. One can directly extend this safeguard update to the multiple-constraint scenario by doing gradient descent over the constraint that has the worst violation. Another remedy to reduce the chance of constraint violation is to do constraint tightening on the constraint cost threshold. Specifically, instead of d0, one may pose the constraint based on d0 · (1 − δ), where δ ∈ (0, 1) is the factor of safety for providing additional buffer to constraint violation. Additional techniques in cost-shaping have been proposed in Achiam et al. (2017) to smooth out the sparse constraint costs. While these techniques can further ensure safety, construction of the cost-shaping term requires knowledge from the environment, which makes the safe PG algorithms more complicated.

heading: B LAGRANGIAN APPROACH TO SAFE RL
text: There are a number of mild technical and notational assumptions which we will make throughout this section, so we state them here: Assumption 1 (Differentiability). For any state-action pair (x, a), πθ(a|x) is continuously differentiable in θ and∇θπθ(a|x) is a Lipschitz function in θ for every a ∈ A and x ∈ X . Assumption 2 (Strict Feasibility). There exists a transient policy πθ(·|x) such that Dπθ (x0) < d0 in the constrained problem. Assumption 3 (Step Sizes). The step size schedules {α3,k}, {α2,k}, and {α1,k} satisfy∑
k α1,k = ∑ k α2,k = ∑ k
α3,k =∞, (13)∑ k α21,k, ∑ k α22,k, ∑ k α23,k <∞, (14)
α1,k = o ( α2,k ) , ζ2(i) = o ( α3,k ) . (15)
Assumption 1 imposes smoothness on the optimal policy. Assumption 2 guarantees the existence of a local saddle point in the Lagrangian analysis introduced in the next subsection. Assumption 3 refers to step sizes corresponding to policy updates that will be introduced for the algorithms in this paper, and indicates that the update corresponding to {α3,k} is on the fastest time-scale, the updates corresponding to {α2,k} is on the intermediate time-scale, and the update corresponding to {α1,k} is on the slowest time-scale. As this assumption refer to user-defined parameters, they can always be chosen to be satisfied.
To solve the CMDP, we employ the Lagrangian relaxation procedure (Bertsekas, 1999) to convert it to the following unconstrained problem:
max λ≥0 min θ
( L(θ, λ) 4 = Cπθ (x0) + λ ( Dπθ (x0)− d0 )) , (16)
where λ is the Lagrange multiplier. Notice that L(θ, λ) is a linear function in λ. Then there exists a local saddle point (θ∗, λ∗) for the minimax optimization problem maxλ≥0 minθ L(θ, λ), such that for some r > 0, ∀θ ∈ Rκ ∩Bθ∗(r) and ∀λ ∈ [0, λmax], we have
L(θ, λ∗) ≥ L(θ∗, λ∗) ≥ L(θ∗, λ), (17)
where Bθ∗(r) is a hyper-dimensional ball centered at θ∗ with radius r > 0.
In the following, we present a policy gradient (PG) algorithm and an actor-critic (AC) algorithm. While the PG algorithm updates its parameters after observing several trajectories, the AC algorithms are incremental and update their parameters at each time-step.
We now present a policy gradient algorithm to solve the optimization problem Equation 16. The idea of the algorithm is to descend in θ and ascend in λ using the gradients of L(θ, λ) w.r.t. θ and λ, i.e.,
∇θL(θ, λ) = ∇θ ( Cπθ (x0) + λDπθ (x0) ) , ∇λL(θ, λ) = Dπθ (x0)− d0. (18)
The unit of observation in this algorithm is a system trajectory generated by following policy πθk . At each iteration, the algorithm generates N trajectories by following the current policy, uses them to estimate the gradients in Equation 18, and then uses these estimates to update the parameters θ, λ.
Let ξ = {x0, a0, c0, x1, a1, c1, . . . , xT−1, aT−1, cT−1, xT } be a trajectory generated by following the policy θ, where xT = xTar is the target state of the system and T is the (random) stopping time. The cost, constraint cost, and probability of ξ are defined as C(ξ) = ∑T−1 k=0 γ
kC(xk, ak), D(ξ) = ∑T−1 k=0 γ kD(xk, ak), and Pθ(ξ) = P0(x0) ∏T−1 k=0 πθ(ak|xk)P (xk+1|xk, ak), respectively.
Based on the definition of Pθ(ξ), one obtains∇θ log Pθ(ξ) = ∑T−1 k=0 ∇θ log πθ(ak|xk).
Algorithm 2 contains the pseudo-code of our proposed policy gradient algorithm. What appears inside the parentheses on the right-hand-side of the update equations are the estimates of the gradients of L(θ, λ) w.r.t. θ, λ (estimates of the expressions in 18). Gradient estimates of the Lagrangian
function are given by ∇θL(θ, λ) = ∑ ξ Pθ(ξ) · ∇θ log Pθ(ξ) ( Cπθ (ξ) + λDπθ (ξ) ) ,
∇λL(θ, λ) = −d0 + ∑ ξ Pθ(ξ) · D(ξ),
where the likelihood gradient is
∇θ log Pθ(ξ) =∇θ  T−1∑ k=0 logP (xk+1|xk, ak) + log πθ(ak|xk) + log 1{x0 = x0}  =
T−1∑ k=0 ∇θ log πθ(ak|xk) = T−1∑ k=0
1
πθ(ak|xk) ∇θπθ(ak|xk).
In the algorithm, ΓΛ is a projection operator to [0, λmax], i.e., ΓΛ(λ) = arg minλ̂∈[0,λmax] ‖λ − λ̂‖22, which ensures the convergence of the algorithm. Recall from Assumption 3 that the stepsize schedules satisfy the standard conditions for stochastic approximation algorithms, and ensure that the policy parameter θ update is on the fast time-scale { ζ2,i }
, and the Lagrange multiplier λ update is on the slow time-scale { ζ1,i }
. This results in a two time-scale stochastic approximation algorithm, which has shown to converge to a (local) saddle point of the objective function L(θ, λ). This convergence proof makes use of standard in many stochastic approximation theory, because in the limit when the step-size is sufficiently small, analyzing the convergence of PG is equivalent to analyzing the stability of an ordinary differential equation (ODE) w.r.t. its equilibrium point.
In policy gradient, the unit of observation is a system trajectory. This may result in high variance for the gradient estimates, especially when the length of the trajectories is long. To address this issue, we propose two actor-critic algorithms that use value function approximation in the gradient estimates and update the parameters incrementally (after each state-action transition). We present two actor-critic algorithms for optimizing Equation 16. These algorithms are still based on the above gradient estimates. Algorithm 3 contains the pseudo-code of these algorithms. The projection operator ΓΛ is necessary to ensure the convergence of the algorithms. Recall from Assumption 3 that the step-size schedules satisfy the standard conditions for stochastic approximation algorithms, and ensure that the critic update is on the fastest time-scale { α3,k } , the policy and θ-update { α2,k } is on
the intermediate timescale, and finally the Lagrange multiplier update is on the slowest time-scale{ α1,k } . This results in three time-scale stochastic approximation algorithms.
Using the policy gradient theorem from Sutton et al. (2000), one can show that
∇θL(θ, λ) = ∇θVθ(x0) = 1 1− γ ∑ x,a µθ(x, a|x0) ∇ log πθ(a|x) Qθ(x, a), (19)
where µθ is the discounted visiting distribution and Qθ is the action-value function of policy θ. We can show that 11−γ∇ log πθ(ak|xk) · δk is an unbiased estimate of∇θL(θ, λ), where
δk = C̄λ(xk, ak) + γV̂ (xk+1)− V̂ (xk)
is the temporal-difference (TD) error, and V̂ is the value estimator of Vθ.
Traditionally, for convergence guarantees in actor-critic algorithms, the critic uses linear approximation for the value function Vθ(x) ≈ φ>ψ(x) = Ṽθ,φ(x), where the feature vector ψ(·) belongs to a low-dimensional space Rκ2 . The linear approximation Ṽθ,φ belongs to a low-dimensional subspace SV = { Ψφ|φ ∈ Rκ2 } , where Φ is a short-hand notation for the set of features, i.e., Ψ(x) = ψ>(x). Recently with the advances of deep neural networks, it has become increasingly popular to model the critic with a deep neural network architecture, based on the objective function of minimizing the MSE of Bellman residual w.r.t. Vθ or Qθ (Mnih et al., 2013).

heading: C EXPERIMENTAL SETUP
text: Our experiments are performed on safety-augmented versions of standard MuJoCo domains (Todorov et al., 2012).
HalfCheetah-Safe. The agent is a the standard HalfCheetah (a 2-legged simulated robot rewarded for running at high speed) augmented with safety constraints. We choose the safety constraints to be defined on the speed limit. We constrain the speed to be less than 1, i.e., constraint cost is thus 1[|v| > 1] . Episodes are of length 200. The constraint threshold is 50. Point Circle. This environment is taken from (Achiam et al., 2017). The agent is a point mass (controlled via a pivot). The agent is initialized at (0, 0) and rewarded for moving counter-clockwise along a circle of radius 15 according to the reward −dx·y+dy·x
1+| √ x2+y2−15| , for position x, y and velocity
dx, dy. The safety constraint is defined as the agent staying in a position satisfying |x| ≤ 2.5. The constraint cost is thus 1[|x| > 2.5]. Episodes are of length 65. The constraint threshold is 7. Point Gather. This environment is taken from (Achiam et al., 2017). The agent is a point mass (controlled via a pivot) and the environment includes randomly positioned apples (2 apples) and bombs (8 bombs). The agent given a reward of 10 for each apple collected and a penalty of −10 for each bomb. The safety constraint is defined as number of bombs collected during the episode. Episodes are of length 15. The constraint threshold is 2.
Ant Gather. This environment is is the same as Point Circle, only with an ant agent (quadrapedal simulated robot). Each episode is initialized with 8 apples and 8 bombs. The agent given a reward of 10 for each apple collected, a penalty reward of −20 for each bomb collected, and a penalty reward of −20 is incurred if the episode terminates prematurely (because the ant falls). Episodes are of length at most 500. The constraint threshold is 10 for DDPG algorithms and is 5 for PPO algorithms.
Figure 4 shows the visualization of the above domains used in our experiments.
On top of the GAE parameter λ, in all numerical experiments and for each algorithm (SPPO, SDDPG, SPPO-modular, SDDPG-modular, CPO, Lagrangian, and the unconstrained PG counterparts), we systematically explored different parameter settings by doing grid-search over the following factors: (i) learning rates in the actor-critic algorithm, (ii) batch size, (iii) regularization parameters of the policy relative entropy term, (iv) with-or-without natural policy gradient updates, (v) with-orwithout the emergency safeguard PG updates (see Appendix A.1 for more details). Although each algorithm might have a different parameter setting that leads to the optimal performance in training, the results reported here are the best ones for each algorithm, chosen by the same criteria (which is based on value of return plus certain degree of constraint satisfaction). To account for the variability during training, in each learning curve a 95% confidence interval is also computed over 10 separate random runs (under the same parameter setting).
C.1 ADDITIONAL EXPERIMENTAL RESULTS In this section, we include more experimental results of various PG algorithms on safe robot locomotion tasks.
Ant-Gather Point-Circle
Ant-Gather Point-Circle

Review summaries:
In this paper, authors compare different ways to enforce stability constraints on trajectories in dynamic RL problems. It builds on a recent approach by Achiam et al on Constrained Policy Optimization (oft- mentioned "CPO") and an accepted NIPS paper by Chow which introduces Lyapunov constraints as an alternative method. While this approach is reasonable indeed, the novelty of the approach is questionable, not only in light of recent papers but older literature: inference of Markov Decision Processes under constraints is referred to and has been known a long time. Furthermore, the actual tasks chosen are quite simple and do not present complex instabilities. Also, actually creating a Lyapunov function and weighing the relative magnitude of its second derivative (steep/shallow) is not trivial and must influence the behavior of the optimizer. Also worth mentioning that complex nonlinearities might imply that instabilities in the observed dynamics are not seen and learned unless the space exploration is conservative. That is, comparison of CPO and Lagrangian constraint based RL with Lyapunov based method proposed depends on a lot of factors (such as those just mentioned) that are not systematically explored by the paper.