Title: State-wise Constrained Policy Optimization
Venue: NeurIPS 2023
Accepted: no
Keywords: Safe Reinforcement Learning

State-wise Safety Guarantee

Trust Region Optimization
Abstract: Reinforcement Learning (RL) algorithms have shown tremendous success in simulation environments, but their application to real-world problems faces significant challenges, with safety being a major concern. In particular, enforcing state-wise constraints is essential for many challenging tasks such as autonomous driving and robot manipulation. However, existing safe RL algorithms under the framework of Constrained Markov Decision Process (CMDP) do not consider state-wise constraints. To address this gap, we propose State-wise Constrained Policy Optimization (SCPO), the first general-purpose policy search algorithm for state-wise constrained reinforcement learning. SCPO provides guarantees for state-wise constraint satisfaction in expectation. In particular, we introduce the framework of Maximum Markov Decision Process, and prove that the worst-case safety violation is bounded under SCPO. We demonstrate the effectiveness of our approach on training neural network policies for extensive robot locomotion tasks, where the agent must satisfy a variety of state-wise safety constraints. Our results show that SCPO significantly outperforms existing methods and can handle state-wise constraints in high-dimensional robotics tasks.

Abstract

Reinforcement Learning (RL) algorithms have shown tremendous success in simulation environments, but their application to real-world problems faces significant challenges, with safety being a major concern. In particular, enforcing state-wise constraints is essential for many challenging tasks such as autonomous driving and robot manipulation. However, existing safe RL algorithms under the framework of Constrained Markov Decision Process (CMDP) do not consider state-wise constraints. To address this gap, we propose State-wise Constrained Policy Optimization (SCPO), the first general-purpose policy search algorithm for state-wise constrained reinforcement learning. SCPO provides guarantees for state-wise constraint satisfaction in expectation. In particular, we introduce the framework of Maximum Markov Decision Process, and prove that the worst-case safety violation is bounded under SCPO. We demonstrate the effectiveness of our approach on training neural network policies for extensive robot locomotion tasks, where the agent must satisfy a variety of state-wise safety constraints. Our results show that SCPO significantly outperforms existing methods and can handle state-wise constraints in high-dimensional robotics tasks.

1 Introduction

Reinforcement learning (RL) has achieved remarkable progress in games and control tasks [Mnih et al., 2015, Vinyals et al., 2019, Brown and Sandholm, 2018, He et al., 2022, Zhao et al., 2019]. However, one major barrier that limits the application of RL algorithms to real-world problems is the lack of safety assurance. RL agents learn to make reward-maximizing decisions, which may violate safety constraints. For example, an RL agent controlling a self-driving car may receive high rewards by driving at high speeds but will be exposed to high chances of collision. Although the reward signals can be designed to penalize risky behaviors, there is no guarantee for safety. In other words, RL agents may sometimes prioritize maximizing the reward over ensuring safety, which can lead to unsafe or even catastrophic outcomes [Gu et al., 2022].

Emerging in the literature, safe RL aims to provide safety guarantees during or after training. Early attempts have been made under the framework of constrained Markov Decision Process, where the majority of works enforce cumulative constraints or chance constraints [Ray et al., 2019, Achiam et al., 2017a, Liu et al., 2021]. In real-world applications, however, many critical constraints are instantaneous. For instance, collision avoidance must be enforced at all times for autonomous cars [Zhao et al., 2023]. Another example is that when a robot holds a glass, the robot can only release the glass when the glass is on a stable surface. The violation of those constraints will lead to irreversible failures of the task. In this work, we focus on state-wise (instantaneous) constraints.

The State-wise Constrained Markov Decision Process (SCMDP) is a novel formulation in reinforce35 ment learning that requires policies to satisfy hard state-wise constraints. Unlike cumulative or 36 probabilistic constraints, state-wise constraints demand full compliance at each time step as for37 malized by Zhao et al. [2023]. Existing state-wise safe RL methods can be categorized based on 38

whether safety is ensured during training. There is a fundamental limitation that it is impossible to 39 guarantee hard state-wise safety during training without prior knowledge of the dynamic model. The 40 best we can achieve in a model free setting is to learn to satisfy the constraints using as few samples 41 as possible, which is the focus of this paper. We aim to provide theoretical guarantees on state-wise 42 safety violation and worst case reward degredation during training. 43

Our approach is underpinned by a key insight that constraining the maximum violation is equivalent 44 to enforcing state-wise safety. This insight leads to a novel formulation of MDP called the M aximum 45 M arkov D ecision P rocess (MMDP). With MMDP, we establish a new theoretical result that provides 46 a bound on the difference between the maximum cost of two policies for episodic tasks. This result 47 expands upon the cumulative discounted reward and cost bounds for policy search using trust regions, 48 as previously documented in literature [Achiam et al., 2017b]. We leverage this result to design a 49 policy improvement step that not only guarantees worst-case performance degradation but also ensures 50 state-wise cost constraints. Our proposed algorithm, S tate-wise C onstrained P olicy O ptimization 51 (SCPO), approximates the theoretically-justified update, which achieves a state-of-the-art trade-off 52 between safety and performance. Through experiments, we demonstrate that SCPO effectively 53 trains neural network policies with thousands of parameters on high-dimensional simulated robot 54 locomotion tasks; and is able to optimize rewards while enforcing state-wise safety constraints. This 55 work represents a significant step towards developing practical safe RL algorithms that can be applied 56 to many real-world problems. 57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

2 Related Work

2.1 Cumulative Safety

Cumulative safety requires that the expected discounted return with respect to some cost function is upper-bounded over the entire trajectory. One representative approach is constrained policy optimization (CPO) [Achiam et al., 2017a], which builds on a theoretical bound on the difference between the costs of different policies and derives a policy improvement procedure to ensure constraints satisfaction. Another approach is interior-point policy optimization (IPO) [Liu et al., 2019], which augments the reward-maximizing objective with logarithmic barrier functions as penalty functions to accommodate the constraints. Other methods include Lagrangian methods [Ray et al., 2019] which use adaptive penalty coefficients to enforce constraints and projection-based constrained policy optimization (PCPO) [Yang et al., 2020a] which projects trust-region policy updates onto the constraint set. Although our focus is on a different setting of constraints, existing methods are still valuable references for illustrating the advantages of our SCPO. By utilizing MMDP, SCPO breaks the conventional safety-reward trade-off, which results in stronger convergence of state-wise safety constraints and guaranteed performance degradation bounds.

2.2 State-wise Safety

Hierarchical Policy One way to enforce state-wise safety constraints is to use hierarchical policies, 74 with an RL policy generating reward-maximizing actions, and a safety monitor modifying the actions 75 to satisfy state-wise safety constraints. Such an approach often requires a perfect safety critic to 76 function well. For example, conservative safety critics (CSC) [Bharadhwaj et al., 2020] propose 77 a safe critic Q C ( s, a ) , providing a conservative estimate of the likelihood of being unsafe given a 78 state-action pair. If the safety violation exceeds a predefined threshold, a new action is re-sampled 79 from the policy until it passes the safety critic. However, this approach is time-consuming. On 80 the other hand, optimization-based methods such as gradient descent or quadratic programming 81 can be used to find a safe action that satisfies the constraint while staying close to the reference 82 action. Unrolling safety layer (USL) [Zhang et al., 2022a] follows a similar hierarchical structure as 83 CSC but performs gradient descent on the reference action iteratively until the constraint is satisfied 84 based on learned safety critic Q C ( s, a ) . Finally, instead of using gradient descent, Lyapunov-based 85 policy gradient (LPG) [Chow et al., 2019] and SafeLayer [Dalal et al., 2018] directly solve quadratic 86 programming (QP) to project actions to the safe action set induced by the linearized versions of some 87 learned critic Q C ( s, a ) . All these approaches suffer from safety violations due to imperfect critic 88 Q C ( s, a ) , while those solving QPs further suffer from errors due to the linear approximation of the 89 critic. To avoid those issues, we propose SCPO as an end-to-end policy which does not explicitly 90

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

End-to-End Policy End-to-end policies maximize task rewards while ensuring safety at the same time. Related work regarding state-wise safety after convergence has been explored recently. Some approaches [Liang et al., 2018, Tessler et al., 2018] solve a primal-dual optimization problem to satisfy the safety constraint in expectation. However, the associated optimization is hard in practice because the optimization problem changes at every learning step. Bohez et al. [2019] approaches the same setting by augmenting the reward with the sum of the constraint penalty weighted by the Lagrangian multiplier. Although claimed state-wise safety performance, the aforementioned methods do not provide theoretical guarantee and fail to achieve near-zero safety violation in practice. He et al. [2023] proposes AutoCost to automatically find an appropriate cost function using evolutionary search over the space of cost functions as parameterized by a simple neural network. It is empirically shown that the evolved cost functions achieve near-zero safety violation, however, no theoretical guarantee is provided, and extensive computation is required. FAC [Ma et al., 2021] does provide theoretically guaranteed state-wise safety via parameterized Lagrange functions. However, FAC replies on strong assumptions and performs poorly in practice. To resolve the above issues, we propose SCPO as an easy-to-implement and theoretically sound approach with no prior assumptions on the underlying safety functions.

3 Problem Formulation

3.1 Preliminaries

In this paper, we are especially interested in guaranteeing safety for episodic tasks, which falls within in the scope of finite-horizon Markov Decision Process (MDP). An MDP is specified by a tuple ( S , A , γ , R, P, µ ) , where S is the state space, and A is the control space, R : S × A ↦→ R is the reward function, 0 ≤ γ < 1 is the discount factor, µ : S ↦→ R is the initial state distribution, and P : S × A × S ↦→ R is the transition probability function. P ( s ′ | s, a ) is the probability of transitioning to state s ′ given that the previous state was s and the agent took action a at state s . A stationary policy π : S ↦→ P ( A ) is a map from states to a probability distribution over actions, with π ( a | s ) denoting the probability of selecting action a in state s . We denote the set of all stationary policies by Π . Subsequently, we denote π θ as the policy that is parameterized by the parameter θ .

The standard goal for MDP is to learn a policy π that maximizes a performance measure J 0 ( π ) which is computed via the discounted sum of reward:

where H ∈ N is the horizon, τ = [ s 0 , a 0 , s 1 , · · · ] , and τ ∼ π is shorthand for that the distribution over trajectories depends on π : s 0 ∼ µ, a t ∼ π ( ·| s t ) , s t +1 ∼ P ( ·| s t , a t ) .

3.2 State-wise Constrained Markov Decision Process

A constrained Markov Decision Process (CMDP) is an MDP augmented with constraints that restrict the set of allowable policies. Specifically, CMDP introduces a set of cost functions, C 1 , C 2 , · · · , C m , where C i : S × A × S ↦→ R maps the state action transition tuple into a cost value. Analogous to (1), we denote

as the cost measure for policy π with respect to cost function C i . Hence, the set of feasible stationary 128 policies for CMDP is then defined as follows, where d i ∈ R : 129

In CMDP, the objective is to select a feasible stationary policy π θ that maximizes the performance 130 measure: 131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

In this paper, we are interested in a special type of CMDP where the safety specification is to persistently satisfy a hard cost constraint at every step (as opposed to cumulative costs over trajectories), which we refer to as State-wise Constrained Markov Decision Process (SCMDP). Like CMDP, SCMDP uses the set of cost functions C 1 , C 2 , · · · , C m to evaluate the instantaneous cost of state action transition tuples. Unlike CMDP, SCMDP requires the cost for every state action transition to satisfy a hard constraint. Hence, the set of feasible stationary policies for SCMDP is defined as

where w i ∈ R . Then the objective for SCMDP is to find a feasible stationary policy from ¯ Π C that maximizes the performance measure. Formally,

3.3 Maximum Markov Decision Process

Note that for (6), every state-action transition pair corresponds to a constraint, which is intractable to solve using conventional reinforement learning algorithms. Our intuition is that, instead of directly constraining the cost of each possible state-action transition, we can constrain the expected maximum state-wise cost along the trajectory, which is much easier to solve. Following that intuition, we define a novel Maximum Markov-Decision Process (MMDP), which further extends CMDP via (i) a set of up-to-now maximum state-wise costs M . = [ M 1 , M 2 , · · · , M m ] where M i ∈ M ⊂ R , and (ii) a set of cost increment functions, D 1 , D 2 , · · · , D m , where D i : ( S , M m ) × A × S ↦→ [0 , R + ] maps the augmented state action transition tuple into a non-negative cost increment. We define the augmented state ˆ s = ( s, M ) ∈ ( S , M m ) . = ˆ S , where ˆ S is the augmented state space. Formally,

By setting D i ( ˆ s 0 , a 0 , ˆ s 1 ) = C i ( s 0 , a 0 , s 1 ) , we have M it = ∑ t -1 k =0 D i ( ˆ s k , a k , ˆ s k +1 ) for t ≥ 1 . Hence, we define expected maximum state-wise cost (or D i -return) for π :

Importantly, (8) is the key component of MMDP and differs our work from existing safe RL approaches that are based on CMDP cost measure (2). With (8), (6) can be rewritten as:

where J ( π ) = E τ ∼ π [ ∑ H t =0 γ t R (ˆ s t , a t , ˆ s t +1 ) ] and R (ˆ s, a, ˆ s ′ ) . = R ( s, a, s ′ ) . With R ( τ ) being the discounted return of a trajectory, we define the on-policy value function as V π (ˆ s ) . = E τ ∼ π [ R ( τ ) | ˆ s 0 = ˆ s ] , the on-policy action-value function as Q π (ˆ s, a ) . = E τ ∼ π [ R ( τ ) | ˆ s 0 = ˆ s, a 0 = a ] , and the advantage function as A π (ˆ s, a ) . = Q π (ˆ s, a ) -V π (ˆ s ) . Lastly, we define on-policy value functions, action-value functions, and advantage functions for the cost increments in analogy to V π , Q π , and A π , with D i replacing R , respectively. We denote those by V π D i , Q π D i and A π D i .

4 State-wise Constrained Policy Optimization

To solve large and continuous MDPs, policy search algorithms search for the optimal policy within a set Π θ ⊂ Π of parametrized policies. In local policy search [Peters and Schaal, 2008], the policy is iteratively updated by maximizing J ( π ) over a local neighborhood of the most recent policy π k . In local policy search for SCMDPs, policy iterates must be feasible, so optimization is over Π θ ⋂ ¯ Π C . The optimization problem is:

where D ist is some distance measure, and δ > 0 is a step size. For actual implementation, we need 166 to evaluate the constraints first in order to determine the feasible set. However, it is challenging to 167 evaluate the constraints using samples during the learning process. In this work, we propose SCPO 168 inspired by recent trust region optimization methods Schulman et al. [2015]. SCPO approximates 169 (10) using (i) KL divergence distance metric D ist and (ii) surrogate functions for the objective and 170 constraints, which can be easily estimated from samples on π k . Mathematically, SCPO requires 171 the policy update at each iteration is bounded within a trust region, and updates policy via solving 172 following optimization: 173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

Theorem 1 (Trust Region Update State-wise Maximum Cost Bound) . For any policies π ′ , π , with /epsilon1 π ′ D . = max ˆ s | E a ∼ π ′ [ A π D (ˆ s, a )] | , and define ¯ d π = ∑ H t =0 P (ˆ s t = ˆ s | π ) as the non-discounted augmented state distribution using π , then the following bound holds:

The proof for Theorem 1 is summarized in Appendix A. Next, we note the following relationship 190 between the total variation divergence and the KL divergence [Boyd et al., 2003, Achiam et al., 2017a]: 191

E ˆ s ∼ ¯ d π [ D TV ( p ‖ q )[ˆ s ]] ≤ √ 1 2 E ˆ s ∼ ¯ d π [ D KL ( p ‖ q )[ˆ s ]] . The following bound then follows directly from 192 Theorem 1: 193

194

195

196

197

198

199

200

By Equation (13), we have a guarantee for satisfaction of maximum state-wise constraints: Proposition 1 (SCPO Update Constraint Satisfaction) . Suppose π k , π k +1 are related by (11) , then D i -return for π k +1 satisfies

Proposition 1 presents the first constraint satisfaction guarantee under MMDP. Unlike trust region methods such as CPO and TRPO, which assume a discounted sum characteristic, MMDP's nondiscounted sum characteristic invalidates these theories. As the maximum state-wise cost is calculated where D KL ( π ′ ‖ π )[ˆ s ] is KL divergence between two policy ( π ′ , π ) at state ˆ s , the set { π ∈ Π θ : E ˆ s ∼ ¯ d π k [ D KL ( π ‖ π k )[ˆ s ]] ≤ δ } is called trust region , d π k . = (1 -γ ) ∑ H t =0 γ t P (ˆ s t = ˆ s | π k ) , ¯ d π k . = ∑ H t =0 P (ˆ s t = ˆ s | π k ) and /epsilon1 π D i . = max ˆ s | E a ∼ π [ A π k D i (ˆ s, a )] | . We then show that SCPO guarantees (i) worst case maximum state-wise cost violation, and (ii) worst case performance degradation for policy update, by establishing new bounds on the difference in returns between two stochastic policies π and π ′ for MMDPs.

Theoretical Guarantees for SCPO We start with the theoretical foundation for our approach, i.e. a new bound on the difference in state-wise maximum cost between two arbitrary policies. The following theorem connects the difference in maximum state-wise cost between two arbitrary policies to the total variation divergence between them. Here total variation divergence between discrete probability distributions p, q is defined as D TV ( p ‖ q ) = 1 2 ∑ i | p i -q i | . This measure can be easily extended to continuous states and actions by replacing the sums with integrals. Thus, the total variation divergence between two policy ( π ′ , π ) at state ˆ s is defined as: D TV ( π ′ ‖ π )[ˆ s ] = D TV ( π ′ ( ·| ˆ s ) ‖ π ( ·| ˆ s )) .

201

202

203

204

205

206

207

208

through a summation of non-discounted increments, analysis must be performed on a finite horizon to upper bound the worst-case summation. In contrast, the theory behind CPO relies on infinite horizon analysis with discounted constraint assumptions, which is not applicable for MMDP settings.

Next, we provide the performance guarantee of SCPO. Previous analyses of performance guarantees have focused on infinite-horizon MDP. We generalize the analysis to finite-horizon MDP, inspired by previous work [Kakade and Langford, 2002, Schulman et al., 2015, Achiam et al., 2017a], and prove it in Appendix B. The infinite-horizon case can be viewed as a special case of the finite-horizon setting.

Proposition 2 (SCPO Update Worst Performance Degradation) . Suppose π k , π k +1 are related by 209 (11) , with /epsilon1 π k +1 . = max ˆ s | E a ∼ π k +1 [ A π k (ˆ s, a )] | , then performance return for π k +1 satisfies 210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

5 Practical Implementation

In this section, we show how to (a) implement an efficient approximation to the update (11), (b) encourage learning even when (11) becomes infeasible, and (c) handle the difficulty of fitting augmented value V π D i which is unique to our novel MMDP formulation. The full SCPO pseudocode is given as algorithm 1 in appendix C.

Practical implementation with sample-based estimation We first estimate the objective and constraints in (11) using samples. Note that we can replace the expected advantage on rewards using an importance sampling estimator with a sampling distribution π k [Achiam et al., 2017a] as

(14) allows us to replace A π k with empirical estimates at each state-action pair (ˆ s, a ) from rollouts by the previous policy π k . The empirical estimate of reward advantage is given by R (ˆ s, a, ˆ s ′ ) + γ V π k (ˆ s ′ ) -V π k (ˆ s ) . V π k (ˆ s ) can be computed at each augmented state by taking the discounted future return. The same can be applied to the expected advantage with respect to cost increments, with the sample estimates given by D i (ˆ s, a, ˆ s ′ ) + V π k D i (ˆ s ′ ) -V π k D i (ˆ s ) . V π k D i (ˆ s ) is computed by taking the non-discounted future D i -return. To proceed, we convexify (11) by approximating the objective and cost constraint via first-order expansions, and the trust region constraint via second-order expansions. Then, (11) can be efficiently solved using duality [Achiam et al., 2017a].

Infeasible constraints An update to θ is computed every time (11) is solved. However, due to approximation errors, sometimes (11) can become infeasible. In that case, we follow [Achiam et al., 2017a] to propose an recovery update that only decreases the constraint value within the trust region. In addition, approximation errors can also cause the proposed policy update (either feasible or recovery) to violate the original constraints in (11). Hence, each policy update is followed by a backtracking line search to ensure constraint satisfaction. If all these fails, we relax the search condition by also accepting decreasing expected advantage with respect to the costs, when the cost constraints are already violated. Denoting c i . = J D i ( π k )+2( H +1) /epsilon1 π D √ δ / 2 -w i , the above criteria can be summarized as

Note that the previous expected advantage E ˆ s ∼ ¯ d π k ,a ∼ π k [ A π k D i (ˆ s, a ) ] is also estimated from rollouts by π k and converges to zero asymptotically, which recovers the original cost constraints in (11).

Imbalanced cost value targets A critical step in solving (11) is to fit the cost increment value 238 functions V π k D i (ˆ s t ) . By definition, V π k D i (ˆ s t ) is equal to the maximum cost increment in any future 239 state over the maximal state-wise cost so far. In other words, the true V π k D i will always be zero for all 240 ˆ s t : H when the maximal state-wise cost has already occurred before time t . In practice, this causes 241 the distribution of cost increment value function to be highly zero-skewed and makes the fitting very 242 hard. To mitigate the problem, we sub-sample the zero-valued targets to match the population of 243 non-zero values. We provide more analysis on this trick in Q3 in section 6.2. 244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

6 Experiments

In our experiments, we aim to answer these questions:

Q1 How does SCPO compare with other state-of-theart methods for safe RL?

Q2 What benefits are demonstrated by constraining the maximum state-wise cost?

Q3 How do the sub-sampling trick of SCPO impact its performance?

6.1 Experiment Setups

New Safety Gym To showcase the effectiveness of our state-wise constrained policy optimization approach, we enhance the widely recognized safe reinforcement learning benchmark environment, Safety Gym Ray et al. [2019], by incorporating additional robots and constraints. Subsequently, we perform a series of experiments on this augmented environment.

Our experiments are based on five different robots: (i) Point: Figure 2a A point-mass robot ( A ⊆ R 2 ) that can move on the ground. (ii) Swimmer: Figure 2b A three-link robot ( A ⊆ R 2 ) that can move on the ground. (iii) Walker: Figure 2d A bipedal robot

(b) Walker-Hazard-8

3.0

Reward_Performance

2.5

Reward_Performance

2.0

1.5

1.0

0

0.5

0.0

TRPO-Lagrangian

TRPO

TRPO-SL

0.5

TotalEnvinteracts

1.0

1.5

TRPO-USL

TRPO-IPO

TRPO-FAC

2.0

2.5

CPO

SCPO

PCPO

3.0

1e7

1

TRPO

TRPO-Lagrangian

TRPO-SL

0.5

1.0

TotalEnvlnteracts

1.5

TRPO-USL

TRPO-IPO

TRPO-FAC

2.0

2.5

CPO

PCPO

SCPO

3.0

1e7

6

TRPO-Lagrangian

TRPO-SL

TRPO-USL

TAPOVAC

RPO-IPO

CPO

PCPO

SCPO

8

TRPO

TRPO-Lagrangian

TRPO-SL

TRPO-USL

TRPO-IPO

TRPO-FAC

SCPO

CPO

PCPO

0

0

0.5

TotalEnvlnteracts

1.0

1.5

2.0

2.5

3.0

1e7

0.5

TotalEnvlnteracts

1.0

1.5

2.0

2.5

1e7

3.0

1.2

1.0

1e-2

TRPO-Lagrangian

TRPO

TRPO-FAC

TRPO-IPO

SCPO

CPO

PCPO

1.4

1e2

TRPO

TRPO-Lagrangia

TRPO-SL

TRPO-USL

TRPO-IPO

CPO

PCPO

SCPO

0.2

0°0

0.5

1.0

1.5

2.0

2.5

3.0

0.5

1.0

1.5

2.0

2.5

3.0

TotalEnvlnteracts

1e7

TotalEnvlnteracts

1e7

( A ⊆ R 10 ) that can move on the ground. (iv) Ant: Figure 2c A quadrupedal robot ( A ⊆ R 8 ) that can move on the ground. (v) Drone: Figure 2e A quadrotor robot ( A ⊆ R 4 ) that can move in the air.

All of the experiments are based on the goal task where the robot must navigate to a goal. Additionally, since we are interested in episodic tasks (finite-horizon MDP), the environment will be reset once the goal is reached. For the robots that can move in 3D spaces (e.g, the Drone robot), we also design a new 3D goal task with a sphere goal floating in the 3D space. Three different types of constraints are considered: (i) Hazard : Dangerous areas as shown in Figure 3a. Hazards are trespassable circles on the ground. The agent is penalized for entering them. (ii) 3D Hazard : 3D Dangerous areas as shown in Figure 3b. 3D Hazards are trespassable spheres in the air. The agent is penalized for entering them. (iii) Pillar : Fixed obstacles as shown in Figure 3c. The agent is penalized for hitting them.

(a) Point

(b) Swimmer

(c) Ant

(d) Walker

(e) Drone

->

X

(b) 3D Hazard

(c) Pillar

(a) Point-Hazard-8

(b) Point-Pillar-4

(c) Swimmer-Hazard-8

3.0

Reward_Performance

Reward_Performance

Reward_Performance

2.5

Reward_Performance

0

1

2.0

0

0

1.5

2

-4

-1

1.0

-2

0.5

-6

TRPO

TRPO-Lagrangian

TRPO-USL

TRPO-IPO

CPO

PCOP

3

TRPO

TRPO-Lagrangian

TRPO-IPO

CPO

PCOP

0.0

TRPO-Lagrangian

TRPO

TRPO-USL

TRPO-IPO

CPO

PCOP

TRPO

TRPO-Lagrangian

TRPO-USL

TRPO-IPO

CPO

PCOP

-8

TRPO-SL

1

TotalEnvinteracts

2

3

TRPO-FAC

4

5

SCPO

1e6

TRPO-SL

1

TotalEnvinteracts

2

3

TRPO-FAC

4

5

SCPO

1e6

0.5

TRPO-SL

1

TotalEnvinteracts

2

3

TRPO-FAC

4

5

SCPO

1e6

-6

TRPO-SL

1

TotalEnvinteracts

2

E

TRPO-FAC

4

5

1e6

5

TRPO-Lagrangian

TRPO-SL

TRPO

TRPO-IPO

TRPO-FAC

TRPO-USL

CPO

PCOP

SCPO

3.0

TRPO-Lagrangian

TRPO

TRPO-SL

TRPO-IPO

TRPO-USL

TRPO-FAC

CPO

SCPO

PCOP

5

4

TRPO-Lagrangian

TRPO-SL

TRPO

TRPO-IPO

TRPO-FAC

PCOP

SCPO

4

TRPO-Lagrangian

TRPO-SL

TRPO

TRPO-USL

TRPO-IPO

TRPO-FAC

PCOP

CPO

0.5

0.0

TotalEnvlnteracts

2

3

4

1e6

6

TotalEnvlnteracts

3

4

5

1e6

6

TotalEnvlnteracts

2

3

4

1e6

TotalEnvlnteracts

2

3

4

1e6

6

1e-2

1e3

1e3

1e3

TRPO-Lagrangian

TRPO-SL

TRPO

TRPO-FAC

TRPO-IPO

CPO

SCPO

PCOP

8

TRPO

TRPO-Lagrangian

TRPO-SL

TRPO-FAC

TRPO-USL

TRPO-IPO

SCPO

PCOP

一

TRPO

TRPO.Lagrangian

TRPO.9

TRPO-FAC

TRPO-IPO

TRPO-USL

一

SCPO

PCOP

CPO

TRPO-SL

TRPO

TRPO-Lagrangian

TRPO-USL

TRPO-FAC

TRPO-IPO

PCOP

CPO

YR

3

2

3

4

6

2

3

4

5

6

2

4

5

6

2

3

4

6

TotalEnvlnteracts

1e6

TotalEnvlnteracts

1e6

TotalEnvlnteracts

1e6

TotalEnvlnteracts

1e6

Considering different robots, constraint types, and constraint difficulty levels, we design 14 test suites 276 with 5 types of robots and 9 types of constraints, which are summarized in Table 1 in Appendix. We 277 name these test suites as {Robot}-{Constraint Type}-{Constraint Number} . 278

279

280

281

282

283

284

285

286

287

288

289

Comparison Group The methods in the comparison group include: (i) unconstrained RL algorithm TRPO [Schulman et al., 2015] (ii) end-to-end constrained safe RL algorithms CPO [Achiam et al., 2017a], TRPO-Lagrangian [Bohez et al., 2019], TRPO-FAC [Ma et al., 2021], TRPO-IPO [Liu et al., 2020], PCPO [Yang et al., 2020b], and (iii) hierarchical safe RL algorithms TRPO-SL (TRPO-Safety Layer) [Dalal et al., 2018], TRPO-USL (TRPO-Unrolling Safety Layer) [Zhang et al., 2022b]. We select TRPO as our baseline method since it is state-of-the-art and already has safety-constrained derivatives that can be tested off-the-shelf. For hierarchical safe RL algorithms, we employ a warm-up phase ( 1 / 3 of the whole epochs) which does unconstrained TRPO training, and the generated data will be used to pre-train the safety critic for future epochs. For all experiments, the policy π , the value ( V π , V π D ) are all encoded in feedforward neural networks using two hidden layers of size (64,64) with tanh activations. More details are provided in Appendix D.

290

291

292

293

294

295

296

297

Evaluation Metrics For comparison, we evaluate algorithm performance based on (i) reward performance, (ii) average episode cost and (iii) cost rate. Comparison metric details are provided in Appendix D.3. We set the limit of cost to 0 for all the safe RL algorithms since we aim to avoid any violation of the constraints. For our comparison, we implement the baseline safe RL algorithms exactly following the policy update / action correction procedure from the original papers. We emphasize that in order for the comparison to be fair, we give baseline safe RL algorithms every advantage that is given to SCPO, including equivalent trust region policy updates.

6.2 Evaluating SCPO and Comparison Analysis

Low Dimension System We select four representative test suites on low dimensional system 298 (Point, Swimmer, Drone) and summarize the comparison results on Figure 4, which demonstrate 299 that SCPO is successful at approximately enforcing zero constraints violation safety performance 300 in all environments after the policy converges. Specifically, compared with the baseline safe RL 301 methods, SCPO is able to achieve (i) near zero average episode cost and (ii) significantly lower 302 cost rate without sacrificing reward performance. The baseline end-to-end safe RL methods (TRPO303 Lagrangian, TRPO-FAC, TRPO-IPO, CPO, PCPO) fail to achieve the near zero cost performance 304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

even when the cost limit is set to be 0. The baseline hierarchical safe RL methods (TRPO-SL, TRPO-USL) also fail to achieve near zero cost performance even with an explicit safety layer to correct the unsafe action at every time step. End-to-end safe RL algorithms fail since all methods rely on CMDP to minimize the discounted cumulative cost while SCPO directly work with MMDP to restrict the state-wise maximum cost by Proposition 1. We also observe that TRPO-SL fails to lower the violation during training, due to the fact that the linear approximation of cost function C (ˆ s t , a, ˆ s t +1 ) [Dalal et al., 2018] becomes inaccurate when the dynamics are highly nonlinear like the ones we used in MuJoCo [Todorov et al., 2012]. More detailed metrics for comparison and experimental results on test suites with low dimension systems are summarized in Appendix D.3.

High Dimension System To demonstrate the scalability and performance of SCPO in high-dimensional systems, we conducted additional tests on the Ant-Hazard-8 and Walker-Hazard-8 suites, with 8-dimensional and 10-dimensional control spaces, respectively. The comparison results for high-dimensional systems are summarized in Figure 1, which show that SCPO outperforms all other baselines in enforcing zero safety violation without compromising performance in terms of return. SCPO rapidly stabilizes the cost return around zero and significantly reduces the cost rate, while the other baselines fail to converge to a policy with near-zero cost. The comparison results of both low dimension and high dimension systems answer Q1 .

Maximum State-wise Cost As pointed in Section 3.3, the under- lying magic for enabling near-zero safety violation is to restrict the maximum state-wise cost to stay around zero. To have a better understanding of this process, we visualize the evolution of maximum state-wise cost for SCPO on the challenging high-dimensional Ant-Hazard-8 and Walker-Hazard-8 test suites in Figure 5 , which answers Q2 .

Ablation on Sub-sampling Imbalanced Cost Increment Value Targets As pointed in Section 5, fitting V π k D i (ˆ s t ) is a critical step towards solving SCPO, which is challenging due to zero-skewed distribution of cost increment value function. To demonstrate the necessity of sub-sampling for solving this challenge, we compare the performance of SCPO with and without sub-sampling trick on the aerial robot test suite, summarized in Figure 6. It is evident that with sub-sampling, the agent achieves higher rewards and more importantly, converges to near-zero costs.

_Performance

0

Cost_Performance

8

with downsampling

without downsampling

-2

Reward

-6

4

4

-8

2

-10

with downsampling

12

without downsampling

1

TotalEnvlnteracts

2

E

4

5

1e6

5

TotalEnvlnteracts

2

3

4

5

1e6

6

That is because sub-sampling effectively balances the cost increment value targets and improves the fitting of V π k D i (ˆ s t ) . We also attempted to solve the imbalance issue via over-sampling non-zero targets, but did not observe promising results. This ablation study provides insights into Q3 .

7 Conclusion and Future Work

This paper proposed SCPO, the first general-purpose policy search algorithm for state-wise constrained RL. Our approach provides guarantees for state-wise constraint satisfaction at each iteration, allows training of high-dimensional neural network policies while ensuring policy behavior, and is based on a new theoretical result on Maximum Markov Decision Process. We demonstrate SCPO's effectiveness on robot locomotion tasks, showing its significant performance improvement compared to existing methods and ability to handle state-wise constraints.

Limitation and future work One limitation of our work is that, although SCPO satisfies state-wise constraints, the theoretical results are valid only in expectation, meaning that constraint violations are still possible during deployment. To address that, we will study absolute state-wise constraint satisfaction, i.e. bounding the maximal possible state-wise cost, which is even stronger than the current result (satisfaction in expectation).

le-1

Cost

1.50

Ant

Walker

State-wise

1.25

1.00

0.75

Maximum

0.50

0.25

0.00

0.5

1.0

1.5

2.0

2.5

3.0

TotalEnvlnteracts

1e7

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

References

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science , 359(6374):418-424, 2018.

Tairan He, Yuge Zhang, Kan Ren, Minghuan Liu, Che Wang, Weinan Zhang, Yuqing Yang, and Dongsheng Li. Reinforcement learning with automated auxiliary loss search. arXiv preprint arXiv:2210.06041 , 2022.

Wei-Ye Zhao, Xi-Ya Guan, Yang Liu, Xiaoming Zhao, and Jian Peng. Stochastic variance reduction for deep q-learning. arXiv preprint arXiv:1905.08152 , 2019.

Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330 , 2022.

Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. CoRR , abs/1910.01708, 2019.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31. PMLR, 2017a.

Yongshuai Liu, Avishai Halev, and Xin Liu. Policy learning with constraints in model-free reinforcement learning: A survey. In The 30th International Joint Conference on Artificial Intelligence (IJCAI) , 2021.

Weiye Zhao, Tairan He, Rui Chen, Tianhao Wei, and Changliu Liu. State-wise safe reinforcement learning: A survey. The 32nd International Joint Conference on Artificial Intelligence (IJCAI) , 2023.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning , pages 22-31. PMLR, 2017b.

Yongshuai Liu, Jiaxin Ding, and Xin Liu. IPO: interior-point policy optimization under constraints. CoRR , abs/1910.09615, 2019. URL http://arxiv.org/abs/1910.09615 .

Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Projection-based constrained policy optimization. CoRR , abs/2010.03152, 2020a. URL https://arxiv.org/ abs/2010.03152 .

Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety critics for exploration. arXiv preprint arXiv:2010.14497 , 2020.

Linrui Zhang, Qin Zhang, Li Shen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Evaluating model-free reinforcement learning toward safety-critical tasks. arXiv preprint arXiv:2212.05727 , 2022a.

Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. ICML 2019 Workshop RL4RealLife , abs/1901.10031, 2019.

Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. CoRR , abs/1801.08757, 2018.

Qingkai Liang, Fanyu Que, and Eytan Modiano. Accelerated primal-dual policy optimization for safe reinforcement learning. arXiv preprint arXiv:1802.06480 , 2018.

Chen Tessler, Daniel J Mankowitz, and Shie Mannor. arXiv preprint arXiv:1805.11074 , 2018.

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

Steven Bohez, Abbas Abdolmaleki, Michael Neunert, Jonas Buchli, Nicolas Heess, and Raia Hadsell. Value constrained model-free continuous control. arXiv preprint arXiv:1902.04623 , 2019.

Tairan He, Weiye Zhao, and Changliu Liu. Autocost: Evolving intrinsic cost for zero-violation reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence , 2023.

Haitong Ma, Yang Guan, Shegnbo Eben Li, Xiangteng Zhang, Sifa Zheng, and Jianyu Chen. Feasible actor-critic: Constrained reinforcement learning for ensuring statewise safety. arXiv preprint arXiv:2105.10682 , 2021.

Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks , 21(4):682-697, 2008.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

Stephen Boyd, Lin Xiao, and Almir Mutapcic. Subgradient methods. lecture notes of EE392o, Stanford University, Autumn Quarter , 2004:2004-2005, 2003.

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning , pages 267-274, 2002.

Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-point policy optimization under constraints. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 4940-4947, 2020.

Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. arXiv preprint arXiv:2010.03152 , 2020b.

Linrui Zhang, Qin Zhang, Li Shen, Bo Yuan, and Xueqian Wang. Saferl-kit: Evaluating efficient reinforcement learning methods for safe autonomous driving. arXiv preprint arXiv:2206.08528 , 2022b.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033. IEEE, 2012.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses an important topic about safe reinforcement learning, which explores the state-wise issue. It is a significant problem because, in the real world, state-wise constraints are one of the most common and challenging constraints in safety-critical applications. Most safe RL methods focus on cumulative safety, which may need to be improved to ensure safety during deploying RL in real-robot applications.\n\nSTRENGTHS:\n1. The writing quality is good.\n2. Theoretical results sound good to me.\n3. The code is provided in this study.\n\nWEAKNESSES:\n1. The experimental results are very weird, especially for the reward performance. For example, in Figure 1, all the baselines’ reward values are almost the same; please check the environments' reward settings. This is also shown in Figure 4.\n2. The method cannot ensure safety while deploying the method in real-world applications; demos that the authors provided also confirm this point, that is, sometimes the agent will violate safety constraints and crash into obstacles.\n\nQUESTIONS:\n1. How does the study consider high-dimensional optimization? Generally, RL algorithms can handle 8- or 10-dimensional tasks on MuJoCo tasks, e.g., CPO, PPO Lagrangian, and CRPO.\n2. How does the study define the reward and cost settings? \n3. Why does this study categorize CSC and safety Layer methods as hierarchical policy? Does this study mean these methods are HRL?\n\nLIMITATIONS:\n1. It is hard for me to find some new insights from this study; the problem and the solution are not new.  As for the problem, it is already defined in many papers, e.g., CPO, PCPO, and CRPO; as for the solution, it just revises the cost setting from a trajectory, and does not ensure safety at each step.\n2. As for the theoretical analysis, most of the findings could be known from CPO by revising the cost optimization settings.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This work tackles the problem of state-wise safety in the reinforcement learning problem. To this end it introduces the framework of Maximum Markov Decision Process and an algorithm State-wise Constrained Policy Optimization (SCPO) to solve the problem. Numerical results illustrate the performance of the authors algorithm.

STRENGTHS:
The paper is well written and supported by extensive numerical results.

WEAKNESSES:
Novelty is slightly limited. Similar state augmentation mechanisms have been studied before. The algorithm developed here arises from a somewhat straightforward application of trust region methods to such state augmentation mechanisms.

QUESTIONS:
Additional details on the motivation behind the Maximum Markov Decision Process formulation would be beneficial. At first read it is slightly confusing as to why state-wise constraints are introduced under this formulation.

The framework of Maximum Markov Decision Process introduces a state augmentation. These mechanisms have been used in the past [Sootla, Calvo-Fullana]. Specially as the augmentation introduced by the authors is similar to augmenting the state space with the dual variables. While the algorithm introduced by the authors here is clearly different, a comparison with these augmentation methods should be done.

A. Sootla, A. I. Cowen-Rivers, J. Wang, and H. B. Ammar. Enhancing safe exploration using safety state augmentation. arXiv preprint arXiv:2206.02675, 2022
M. Calvo-Fullana, S. Paternain, L. F. O. Chamon, and A. Ribeiro. State augmented constrained reinforcement learning: Overcoming the limitations of learning with rewards. arXiv preprint arXiv:2102.11941, 2021.

LIMITATIONS:
Addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper introduces a novel approach to solve safe RL tasks. This approach is based on a new method SCPO and corresponding MMDP framework. Authors show the efficacy of this method by providing mathematical guarantees. Additionally, authors give useful practical implementation tips to improve reproducibility of their work.

STRENGTHS:
originality
  - This paper extends CMDP framework to account for state-wise safety guarantees. It's a fairly natural extension, and authors did a great job of explaining the shortcomings of existing methods and how theirs addresses the gaps. 

quality
  - The paper is well-written and includes thorough mathematical analysis and practical tips for implementation. 
  - Experimental section includes fair comparison with existing SOTA methods.

clarity
  - The paper uses consistent notation with previous papers (i.e. TRPO).
  - Including pseudocode (in appendix) is useful.
 
significance.
  - The paper has a good significant impact on safe RL. The presented method is novel and beats existing methods on the selected benchmarks. Authors also provided concrete future direction for this work.

WEAKNESSES:
- Not necessarily a weakness, but including intuitive explanation of proposition 1 and 2 would be helpful to increase overall readability of the paper.
- Paper heavily relied on concepts TRPO, but didn't give explanations about them in the background section.

QUESTIONS:
N/A

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework called Maximum MDP to address the problem of state-wise constrained policy optimization, namely the authors considers limiting the expected maximum state-wise cost rather than the cost for each state. Similar to the TRPO/CPO framework, the authors derived a worse-case constraint violation guarantee and practical algorithm which were shown in be effective in a set of robotic locomotive tasks.\n\nSTRENGTHS:\n- I really like the simplicity of the idea proposed by the authors, state-wise safe RL is a much more common requirement for real-world safety critical problems but most literature on safe RL focus on constraining the cumulative cost. Reformulating the problem as constraining expected maximum state-wise cost eliminates the scalability issues associated with state-wise safe RL and effectively transforms the problem into a cumulative cost problem.\n- Furthermore, this formulation allows the authors to easily adapt existing algorithms for cumulative constraints (CPO) with little modifications.\n- Experiment section is well-structured and answers key practical questions associated with the proposed algorithms.\n\nWEAKNESSES:\n- Theorem 1 doesn't really depend on the specific formulation of MMDP and is really a finite-horizon variant of the policy improvement theorem from TRPO/CPO (which I think is a nice contribution in itself), I think the authors should try to convey this\n- There is a major error in the proof in the appendix, note that equation 20 is not correct since $I - P$ is not invertible\n\nQUESTIONS:\n- One related reference which seems to be missing is [1] which similarly uses a trust region based method but for cumulative costs, one question I have here is can SCPO be applied to other trust-region based algorithms such as PCPO or FOCOPS?\n- Overall, I do like the main idea proposed by the authors and I think the general approach is sound and the empirical improvements are valid. However the issue in the proof essentially invalidates Theorem 1. While I am fairly sure a finite horizon variant of the policy improvement theorem from CPO/TRPO does exist, the authors need to make sure the results are correct. At present, I cannot recommend this paper for acceptance at this venue, however I am willing to revise my score if the authors do fix the mathematical issues during the rebuttal period.\n- [Minor] Achiam et al.'s Constrained Policy Optimization seem to appear twice in your list of references\n\n[1] Zhang et al. (2020) First Order Constrained Optimization in Policy Space\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}