Title: Constrained Proximal Policy Optimization
Venue: NeurIPS 2023
Accepted: no
Keywords: constrained reinforcement learning
Abstract: The problem of constrained reinforcement learning (CRL) holds significant importance as it provides a framework for addressing critical safety satisfaction concerns in the field of reinforcement learning (RL). However, with the introduction of constraint satisfaction, the current CRL methods necessitate the utilization of second-order optimization or primal-dual frameworks with additional Lagrangian multipliers, resulting in increased complexity and inefficiency during implementation. To address these issues, we propose a novel first-order feasible method named Constrained Proximal Policy Optimization (CPPO). By treating the CRL problem as a probabilistic inference problem, our approach integrates the Expectation-Maximization framework to solve it through two steps: 1) calculating the optimal policy distribution within the feasible region (E-step), and 2) conducting a first-order update to adjust the current policy towards the optimal policy obtained in the E-step (M-step). We establish the relationship between the probability ratios and KL divergence to convert the E-step into a convex optimization problem. Furthermore, we develop an iterative heuristic algorithm from a geometric perspective to solve this problem. Additionally, we introduce a conservative update mechanism to overcome the constraint violation issue that occurs in the existing feasible region method. Empirical evaluations conducted in complex and uncertain environments validate the effectiveness of our proposed method, as it performs at least as well as other baselines.

Abstract

The problem of constrained reinforcement learning (CRL) holds significant importance as it provides a framework for addressing critical safety satisfaction concerns in the field of reinforcement learning (RL). However, with the introduction of constraint satisfaction, the current CRL methods necessitate the utilization of second-order optimization or primal-dual frameworks with additional Lagrangian multipliers, resulting in increased complexity and inefficiency during implementation. To address these issues, we propose a novel first-order feasible method named Constrained Proximal Policy Optimization (CPPO). By treating the CRL problem as a probabilistic inference problem, our approach integrates the ExpectationMaximization framework to solve it through two steps: 1) calculating the optimal policy distribution within the feasible region (E-step), and 2) conducting a firstorder update to adjust the current policy towards the optimal policy obtained in the E-step (M-step). We establish the relationship between the probability ratios and KL divergence to convert the E-step into a convex optimization problem. Furthermore, we develop an iterative heuristic algorithm from a geometric perspective to solve this problem. Additionally, we introduce a conservative update mechanism to overcome the constraint violation issue that occurs in the existing feasible region method. Empirical evaluations conducted in complex and uncertain environments validate the effectiveness of our proposed method, as it performs at least as well as other baselines.

1 Introduction

In recent years, reinforcement learning (RL) has achieved huge success in various aspects (Le et al., 2022; Li et al., 2022; Silver et al., 2018), especially in the field of games. However, due to the increased safety requirements in practice, researchers are starting to consider the constraint satisfaction in RL. Compared with unconstrained RL, constrained RL (CRL) incorporates certain constraints during the process of maximizing cumulated rewards, which provides a framework to model several important topics in RL, such as safe RL (Paternain et al., 2022), highlighting the importance of this problem in industrial applications.

The current methods for solving the CRL problem can be mainly classified into two categories: 29 primal-dual method (Paternain et al., 2022; Stooke et al., 2020; Zhang et al., 2020; Altman, 1999) and 30 feasible region method (Achiam et al., 2017; Yang et al., 2020). The primal-dual method introduces 31 the Lagrangian multiplier to convert the constrained optimization problem into an unconstrained dual 32 problem by penalizing the infeasible behaviours, promising the CRL problem to be resolved in a 33 first-order manner. Despite the primal-dual framework providing a way to solve CRL in first-order 34 manner, the update of the dual variable, i.e., the Lagrangian multiplier, tends to be slow and unstable, 35 affecting the overall convergent speed of the algorithms. In contrast, the feasible region method 36 provides a faster learning method by introducing the concept of the feasible region into the trust 37 region method. With either searching in the feasible region (Achiam et al., 2017) or projecting into 38

the feasible region (Yang et al., 2020), the feasible region method can guarantee the generated policies 39 stay in the feasible region. However, the introduction of the feasible region in the proposed method 40 relies on computationally expensive second-order optimization using the inverse Fisher information 41 matrix. This approach can lead to inaccurate estimations of the feasible region and potential constraint 42 violations, as reported in previous studies (Ray et al., 2019). 43

44

45

46

47

48

49

50

To address the existing issues mentioned above, this paper proposed the Constrained Proximal Policy Optimization (CPPO) algorithm to solve the CRL problem in a first-order, easy-to-implement way. CPPO employs a two-step Expectation-Maximization approach to solve the problem by firstly calculating the optimal policy (E-step) and then conducting a first-order update to reduce the distance between the current policy and the optimal policy (M-step), eliminating the usage of the Lagrangian multiplier and the second-order optimization. The main contributions of this work are summarized as follows:

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

To our best knowledge, the proposed method is the first first-order feasible region method without using dual variables or second-order optimization, which significantly reduces the difficulties in tuning hyperparameters and the computing complexity.

An Expectation-Maximization (EM) framework based on advantage value and probability ratio is proposed for solving the CRL problem efficiently. By converting the CRL problem into a probabilistic inference problem, the CRL problem can solved in first order manner without dual variables.

To solve the convex optimization problem in E-step, we established the relationship between the probability ratios and KL divergence, and developed an iterative heuristic algorithm from a geometric perspective.

A recovery update is developed when the current policy encounters constraint violation. Inspired by Bang-bang control, this update strategy can improve the performance of constraint satisfaction and reduce the switch frequency between normal update and recovery update.

The proposed method is evaluated in several benchmark environments. The results manifest its comparable performance over other baselines in complex environments.

This paper is organized as follows. Section 2 introduces the concept of constrained markov decision process and present an overview of related works in the field. The Expectation-Maximization framework and the technical details about the proposed constrained proximal policy optimization method are proposed in Section 3. Section 4 verifies the effectiveness of the proposed method through several testing scenarios and an ablation study is conducted to show the effectiveness of the proposed recovery update. Section 5 states the limitations and the boarder impact of the proposed method. Finally, a conclusion is drawn in Section 6.

2 Preliminary and Related Work

2.1 Constrained Markov Decision Process

Constrained Markov Decision Process(CMDP) is a mathematical framework for modelling decisionmaking problems subjected to a set of cost constraints. A CMDP can be defined by a tuple ( S , A , P, r, γ, µ, C ), where S is the state space, A is the action space, P : S × A × S → (0 , 1) is the transition kernel, r : S × A → R is the reward function, γ → (0 , 1) is the discount factor, µ : S → (0 , 1) is the initial state distribution, and C := { c i ∈ C | c i : S ×A → R , i = 1 , 2 , . . . , m } is the set of m cost functions. For simplicity, we only consider a CRL problem with one constraint in the following paper and use c to represent the cost function. Note that, although we restrict our discussion to the case with only one constraint, the method proposed in this paper can be naturally extended to the multiple constraint case. However, the result may not as elegant as the one constraint case. Compared with the common Markov Decision Process(MDP), CMDP introduces a constraint on the cumulated cost to restrict the agent's policies. Considering a policy π ( s | a ) : S×A → (0 , 1) , the goal of MDP is to find the π that maximizes the expected discounted returns J r ( π ) = E τ [ ∑ ∞ t =0 γ t r ( s t )] , where τ is the trajectories generated based on π . Based on these settings, CMDP applied a threshold d on the expected discounted cost returns J c ( π ) = E τ [ ∑ ∞ t =0 γ t c ( s t )] . Thus, the CMDP problem can be formed as finding policy π ∗ that π ∗ = argmax π J r ( π ) s.t. J c ( π ∗ ) ≤ d . The advantage function A and the cost advantage function A c is defined as A ( s t , a t ) = Q ( s t , a t ) -V ( s t )

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

and A c ( s t , a t ) = Q c ( s t , a t ) -V c ( s t ) where Q ( s t , a t ) = E τ [ ∑ ∞ t =0 γ t r | s 0 = s t , a 0 = a t ] and V ( s t ) = E τ [ ∑ ∞ t =0 γ t r | s 0 = s t ] are the corresponding Q-value and V-value for reward function, and Q c ( s t , a t ) = E τ [ ∑ ∞ t =0 γ t c | s 0 = s t , a 0 = a t ] and V c ( s t ) = E τ [ ∑ ∞ t =0 γ t c | s 0 = s t ] are the corresponding Q-value and V-value for cost function. Note that both A and A c in the batch are centered to moves theirs mean to 0, respectively.

2.2 Related Work

2.2.1 Proximal Policy Optimization (PPO)

Proximal policy optimization (PPO) (Schulman et al., 2017) is a renowned on-policy RL algorithm for its stable performance and easy implementation. Based on the first-order optimization methodology, PPO addresses the challenge of the unconstrained RL problem through the surrogate objective function that proposed in Trust Region Policy Optimization (TRPO) (Schulman et al., 2015a). With the clipping and early stop trick, PPO can keep the new policy to stay within the trust region. Thanks to its stability and superior performance, the PPO algorithm has been employed in various subfields of RL like multi-agent RL (Yu et al., 2021), Meta-RL (Yu et al., 2020). However, due to the extra constraint requirements, the direct application of PPO in CRL problems is not feasible. The extra constraint requirements cause PPO not only restricted by the trust region but also the constraint feasible region, which significantly increases the challenge in conducting first-order optimization. Despite the difficulties in the direct application of PPO in CRL, researchers are still searching for a PPO-like method to solve CRL problems with stable and superior performance.

2.2.2 Constrained Reinforcement Learning

The current methods for solving the CRL problem can be mainly divided into two categories: primaldual method (Paternain et al., 2022; Stooke et al., 2020; Zhang et al., 2020) and feasible region method (Achiam et al., 2017; Yang et al., 2020). The primal-dual method converts the original problem into a convex dual problem by introducing the Lagrangian multiplier. By updating the policy parameters and Lagrangian multiplier iteratively, the policies obtained by the primal-dual method will gradually converge towards a feasible solution. However, the usage of the Lagrange multiplier introduces extra hyperparameters into the algorithm and slows down the convergence speed of the algorithm due to the characteristic of the integral controller. Stooke et al. (2020) tries to solve this issue by introducing PID control into the update of the Lagrangian multiplier, but this modification will introduce more hyperparameters and cause the algorithm to be complex. Different from the primal-dual method, the feasible region method estimates the feasible region within the trust region using linear approximation and subsequently determines the new policy based on the estimated feasible region. A representative method is constrained policy optimization (CPO). By converting the CRL to a quadratically constrained linear program, CPO (Achiam et al., 2017) can solve the problem efficiently. However, the uncertainties inside the environment may cause an inaccurate cost assessment, which will affect the estimation of the feasible region and cause the learned policy to fail to meet the constraint requirements, as shown in Ray et al. (2019). Another issue of CPO is that it uses the Fisher information matrix to estimate the KL divergence in quadratic approximation, which is complex in computing and inflexible in network structure.

To address the second-order issue in CRL, several researchers (Zhang et al., 2020; Liu et al., 2022) proposed the EM-based algorithm in a first-order manner. FOCOPS (Zhang et al., 2020) obtain the optimal policy from advantage value, akin to the maximum entropy RL, and perform a first-order update to reduce the KL divergence between the current policy and the optimal policy. Despite its significant improvement in performance compared to CPO, FOCOPS still necessitates the use of a primal-dual method to attain a feasible optimal policy, which introduces a lot of hyperparameters for tuning, resulting in a more complex tuning process. CVPO (Liu et al., 2022) extends the maximum However, this algorithm still requires the primal-dual framework in optimal policy calculation and a posteriori policy optimization (MPO) (Abdolmaleki et al., 2018) method to the CRL problem, allowing for the efficient calculation of the optimal policy from Q value in an off-policy manner. necessitates additional samplings during the training, increasing the complexity of implementation. Thus, the development of a simple-to-implement, first-order algorithm with superior performance, remains a foremost goal for researchers in the CRL subfield.

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

3 Constrained Proximal Policy Optimization (CPPO)

As mentioned in Section 2, existing CRL methods often require second-order optimization for feasible region estimation or the use of dual variables for cost satisfaction. These approaches can be computationally expensive or result in slow convergence. To address these challenges, we proposed a two-step approach in an EM fashion named Constrained Proximal Policy Optimization (CPPO), the details will be shown in this section.

3.1 Modelling CRL as Inference

Instead of directly pursuing an optimal policy to maximize rewards, our approach involves conceptualizing the problem of Constrained Reinforcement Learning (CRL) as a probabilistic inference problem. This is achieved by assessing the reward performance and constraint satisfaction of stateaction pairs and subsequently increasing the likelihood of those pairs that demonstrate superior reward performance while adhering to the constraint requirement. Suppose the event of state-action pairs under policy π θ can maximize reward is represented by optimality variable O , we assume the likelihood of state-action pairs being optimal is proportional to the exponential of its advantage value: p ( O = 1 | ( s, a )) ∝ exp ( A ( s, a ) /α ) where α is a temperature parameter. Denote q ( a | s ) is the feasible posterior distribution estimated from the sampled trajectories under current policy π , p π ( a | s ) is the probability distribution under policy π , and θ is the policy parameters. We can have following evidence lower bound(ELBO) J ( q, θ ) using surrogate function(see Appendix B for detailed proof)

where d π is the state distribution under current policy π , p ( θ ) is a prior distribution of policy 162 parameters. Considering q ( a | s ) is a feasible policy distribution, we also have following constraint 163 (Achiam et al., 2017) 164

where d is the cost constraint. By performing iterative optimization of the feasible posterior distribution q (E-step) and the policy parameter θ (M-step), the lower bound J ( q, θ ) can be increased, resulting in an enhancement in the likelihood of state-action pairs that have the potential to maximize rewards.

3.2 E-Step

3.2.1 Surrogate Constrained Policy Optimization

As mentioned in the previous section, we will firstly optimize the feasible posterior distribution q to maximize ELBO in E-step. The feasible posterior distribution q plays a crucial role in determining the upper bound of the ELBO since the KL divergence is non-negative. Consequently, q needs to be theoretically optimal to maximize the ELBO. By converting the soft KL constraint in Equation (1) into a hard constraint and combining the cost constraint in Equation (2),the optimization problem of q can be expressed as follows:

165

166

167

168

169

170

171

172

173

174

175

176

where δ is the reverse KL divergence constraint that determine the trust region. During the E-step, it is 177 important to note that the optimization is independent of θ , meaning that the policy π θ remains fixed 178 to the current sampled policy π . Even we know the closed-form expression of p π θ , it is impractical to 179 solve the closed-form expression of q from Equation (3), as we still needs the closed-form expression 180 of d π for calculating. Therefore, we we opt to represent the solution of q in a non-parametric manner 181

by calculating the probability ratio v = q ( a | s ) p π ( a | s ) for the sampled state-action pairs, allowing us to 182

avoid explicitly parameterizing q and instead leverage the probability ratio to guide the optimization 183 process. After relaxing the reverse KL divergence constraint with the estimated reverse KL divergence 184 calculated through importance sampling, we can obtain 185

where d ′ the scaled cost margin d ′ = (1 -γ )( d -J c ( π )) . Although Equation (4) is convex optimization problem that can be directly solved through existing convex optimization algorithm, the existence of non-polynomial KL constraint tends to cause the optimization to be computationally expensive. To overcome this issue, the following proposition is proposed to relax Equation (4) into an linear optimization problem with quadratic constraint.

Proposition 3.1. Denote v as the probability ratios q ( a | s ) p π ( a | s ) calculated from sampled trajectories. If there are a sufficient number of sampled v , we have E [ v ] = 1 and E [ v log v ] ≤ Var ( v -1) .

With Proposition 3.1, the relationship between reverse KL divergence and l 2 -norm of vector v -1 is constructed. Also, consider that the expectation of v equals 1, the optimization variable can be changed from v to v -1 . Let v denote the vector consists of v -1 and replace the reverse KL divergence constraint with the l 2 -norm constraint, Equation (4) can be rewritten in the form of vector multiplication

where A and A c are the advantage value vectors for reward and cost (for all sampled state-action pairs in one rollout) respectively, N is the number of state-action pair samples, δ ′ is l 2 -norm constraint, and the element-wise lower bound of v is -1 , as v > 0 . Thus, the optimal feasible posterior distribution q expressed through v can be obtained by solving the aforementioned optimization problem.

Remark 3.2 . By replacing the non-polynomial KL constraint with an l 2 -norm constraint, the original optimization problem in Equation (4) can be reformulated as a geometric problem. This reformulation enables the use of the proposed heuristic method to efficiently solve the problem without the need for dual variables .

Remark 3.3 . Our proposed method builds upon the idea presented in CVPO (Liu et al., 2022) of treating the CRL problem as a probabilistic inference problem. However, our approach improves upon their idea in two significant ways. Firstly, the probabilistic inference problem in our method is constructed based on advantage value , which is more effective in reducing the bias in estimating the cost return, compared to the Q-value used in CVPO. Secondly, while CVPO tries to directly calculate the value of q ( a | s ) , our method employs the probability ratio v to represent q . By replacing q ( a | s ) with v , our method only needs to find a vector of v whose elements are positive and E [ v ] = 1 , thereby negating the need to sample multiple actions in one state to calculate the extra normalizer that ensures q is a valid distribution. This results in a significant reduction in computational complexity.

3.2.2 Recovery update

Although the optimal solution q in Section 3.2.1 is applicable when the current policy is out of the feasible region, the inconsistent between optimal q and π θ and the inaccurate cost evaluations tends to result in the generation of infeasible policies, as demonstrated in Ray et al. (2019) where CPO fail to satisfy constraint. To overcome this issue, a recovery update strategy is proposed for pushing the agent back to the feasible region. This strategy aims to minimize costs while preserving or minimizing any reduction in overall reward return. In the event that it is not possible to recover from the infeasible region without compromising the reward return, the strategy aims to identify an optimal policy within the feasible region that minimizes the adverse impact on the reward return. The optimization problem in recovery update can be expressed as

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

Figure 1 illustrates the recovery update strategy from the perspective of geometry. The blue, red, 225 and yellow arrows represent the direction of minimizing cost, maximizing reward and the recovery 226 update, respectively. The reward preservation region is defined by the zero reward boundary, which is 227 depicted as the dashed line perpendicular to the red arrow. As a result, the semi-circle encompassing 228 the red arrow indicates a positive increment in reward. Case 1 and Case 3 illustrate the case when the 229 reward preservation region has an intersection with the feasible region. In these cases, we choose 230 the direction of minimizing cost within the reward preservation region, e.g., the recovery update 231 direction is coincident with the dashed line in Case 1, and the recovery update direction is coincident 232 with the blue arrow in Case 3. Case 2 shows the case when there is no intersection between the 233 reward preservation region and the feasible region. In this case, the direction with the least damage 234 to reward is chosen. If we use an angle α to represent the direction of update, then we can have 235 α = Clip( α, max( θ f , θ A + π/ 2) , π ) , where θ A represents the direction of A , θ f is the minimum 236 angle that can point toward the feasible region. 237

Infeasible region

A

-Ac

A

-Ac

Update direction

A

-Ac

Feasible region

Zero reward boundary

Case 1

Case 2

Case 3

To further improve the constraint satisfaction performance, a switching mechanism inspired by bang238 bang control (Lasalle, 1960) is introduced. As shown in Figure 2, the agent will initially conduct 239 normal update in Section 3.2.1; when the agent violates the cost constraint, it will switch to recovery 240 update to reduce the cost until the cost is lower than the lower switch cost. By incorporating this 241 switching mechanism, a margin is created between the lower switch cost and the cost constraint. 242 This margin allows for a period of normal updates before the recovery update strategy is invoked. 243 As a result, this mechanism prevents frequent switching between the two strategies, leading to 244 improved performance in both reward collection and cost satisfaction. This switching mechanism 245 effectively balances the exploration of reward-maximizing actions with the need to maintain constraint 246 satisfaction. 247

Cost

Cost

constraint

Switch

cost

Normal

update

Recovery

update

3.3 Heuristic algorithm from geometric interpretation

248

Section 3.2 and Section 3.4 provide a framework for solving CRL problem in theory. However, 249 solving Equation (5) and Equation (6) in Section 3.2 is a tricky task in practice. To reduce the 250 computation complexity, an iterative heuristic algorithm is proposed to solve this optimization 251 problem from geometric interpretation. Recall Equation (5), the l 2 -norm can be interpreted as a radius 252 constraint from the geometric perspective. Additionally, both the objective function and the cost 253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

function are linear, indicating that the optimal solution lies on the boundary of the feasible region. By disregarding the element-wise bounds in Equation (5), we can consider the optimization problem as finding a optimal angle θ ′ on the A -A c plane, in accordance with Theorem 3.4. The optimal solution can be expressed as v = 2 Nδ ′ (cos θ ′ ˜ A c +sin θ ′ ˜ A ) , where ˜ A and ˜ A c are the orthogonal unit vectors of A and A c respectively. Considering Assumption 3.5, we proposed a iterative heuristic algorithm to solve Equation (5) by firstly calculating the optimal angle θ ′ regardless the element-wise bound and obtain a initial solution v , then clip v according to the element-wise bound and mask the clipped value, and iteratively update the rest unmasked elements according to aforementioned steps until all elements in v are satisfy the element-wise bound. The detailed steps are outlined in Appendix C. For the recovery update in Section 3.2.2, the same algorithm can be used to find the angle that satisfy v · A c = Nd ′ or v · A = 0 .

Theorem 3.4. Given a feasible optimization problem of the form:

where v , A , and A c are N -dimensional vectors, then the optimal solution v will lie in the A -A c plane determined by A c and A .

Assumption 3.5. If the optimization problem in Theorem 3.4 has a optimal solution v opt = [ v 1 , v 2 , . . . ] , and the same problem with element-wise lower bound constraint b has a optimal solution v ′ opt = [ v ′ 1 , v ′ 2 , . . . ] , then v ′ t = b where v t ≤ b .

Remark 3.6 . By utilizing the proposed heuristic algorithm, the optimal solution to Equation (5) can be obtained in just a few iterations. The time complexity of each iteration is O ( n ) , where n represents the number of unmasked elements. As a result, the computational complexity is significantly reduced compared to conventional convex optimization methods.

3.4 M-Step

After determining the optimal feasible posterior distribution q to maximize the upper bound of ELBO, an M-step is implemented to maximize ELBO by updating policy parameters θ in a supervised learning manner. Recall the definition of ELBO in Equation (1) in Section 3.1, by dropping the part that independent from θ , we will obtain following optimization problem

Note that if we assume p ( θ ) is a Gaussian distribution, then log p ( θ ) can be converted into D KL ( π ‖ π θ ) (see Appendix B for details). Using the same trick in Section 3.2.1 to convert soft KL constraint to hard KL constraint, the supervised learning problem in M-step can be expressed as

Note that D KL ( π θ ‖ π ) is chosen to lower than δ so that the current policy π can be reached during the E-step in next update iteration to achieve robust update.

For Equation (7), it is a common practice for researchers to directly minimize the KL divergence, like CVPO (Liu et al., 2022) and MPO (Abdolmaleki et al., 2018). However, recall Equation (6), it is evident that the value of surrogate reward and cost are deeply connected to the projection of v onto the A -A c plane, while KL divergence can hardly reflect this kind of relationship between v and surrogate value. Consequently, Consequently, we choose to replace the original KL objective function with the l 2 -norm E [ ‖ v -p π θ /p π ‖ 2 ] , where v is the optimal probability ratio obtained in E-step and p π θ /p π is the probability ratio under policy parameter θ . With this replacement, the optimization problem can be treated as a fixed-target tracking control problem. This perspective enables us to plan tracking trajectories that can consistently satisfy the cost constraint, enhancing the ability to maintain cost satisfaction throughout the learning process. The optimization problem after replacement can be rewritten as

To ensure the tracking trajectories can satisfy cost constraint at nearly all locations, we calculated the 296 several recovery v ′ under different δ ′′ and guide p π θ p π to different v according to the l 2 -norm of p π θ p π , 297 so that even ‖ p π θ p π ‖ 2 is much smaller than 2 Nδ ′ , the new policy can still satisfy the cost constraint. 298 Moreover, inspired by the proportional navigation (Yanushevsky, 2018), we also modify the recovery 299 update gradient from ( v -p π θ p π ) ∂π θ ∂θ to (( β ( v -p π θ p π ) + (1 -β ) A ′ c ) ∂π θ ∂θ to reduce the cost during 300 the tracking, where A ′ c is the projection of v -p π θ p π on cost advantage vector A c . In according with 301 Theorem 3.7, the lower-bound clipping mechanism similar with PPO is applied on updating p π θ p π in 302 M-step to satisfy the forward KL constraint (see Appendix C for details). 303

304

305

306

Theorem 3.7. For a probability ratio vector v , if the variance of v is constant, then the upper bound of the approximated forward KL divergence D KL ( π θ ‖ π ) , will decrease as the element-wise lower bound of v increase.

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

Apart from E-step and M-step introduced in Section 3.2 and Section 3.4, our method shares the same Generalized Advantage Estimator (GAE) technique (Schulman et al., 2015b) with PPO in calculating the advantage value A and A c . The main steps of CPPO are summarized in Appendix C.

4 Experiment

In this section, Safety Gym (Ray et al., 2019) benchmark environments and Circle environment (Achiam et al., 2017) are used to verify and evaluate the performance of the proposed method. Five test scenarios, namely CarPush, PointGoal, PointPush, PointCircle, and AntCircle are evaluated. The detailed information about the test scenarios can be seen in Appendix D. Three algorithms are chosen as the benchmarks to compare the learning curves and the constraint satisfaction: CPO (Achiam et al., 2017), PPO-Lagrangian method (simplified as PPO_lag), and TRPO-Lagrangian method (simplified as TRPO_lag) (Ray et al., 2019). CPO is chosen as the representative of the feasible region method. PPO_lag and TRPO_lag are treated as the application of the primal-dual method in first-order optimization and second-order optimization. TRPO and PPO are also used in this section as unconstrained performance references. For a fair comparison, all of the algorithms use the same policy network and critic network. The detail of the hyperparameter setting is listed in Appendix E.

1

0

1

2

3

Performance

carpush1

0

10

20

pointgoal1

2.5

0.0

2.5

5.0

pointpush1

0

50

100

pointcircle

0

100

antcircle

0.00

0.25

0.50

0.75

1.00

TotalEnvInteracts

1e7

20

40

AverageEpCost

0.00

0.25

0.50

0.75

1.00

TotalEnvInteracts

1e7

20

40

0.00

0.25

0.50

0.75

1.00

TotalEnvInteracts

1e7

0

20

40

0.0

0.5

1.0

1.5

2.0

TotalEnvInteracts

1e5

0

5

10

0.00

0.25

0.50

0.75

1.00

TotalEnvInteracts

1e7

0

100

200

300

cppo

cpo

ppo_lag

trpo_lag

cost constraint

Performance and Constraint Satisfaction: Figure 3 compares the learning curves of the proposed 323 method and other benchmark algorithms in terms of the episodic return and the episodic cost. The 324 first row records the undiscounted episodic return for performance comparison, and the second row is 325 the learning curves of the episodic cost for constraint satisfaction analysis, where the red dashed line 326 indicates the cost constraint. The learning curves for the Push and Goal environments are averaged 327 over 6 random seeds, while those for the Circle environments are averaged over 4 random seeds. 328 The curve itself represents the mean value, and the shadow indicates the standard deviation. In 329 terms of performance comparison, it was observed that CPO can achieve the highest reward return 330 in PointGoal and PointCircle. The proposed CPPO method, on the other hand, achieves similar or 331 even higher reward return in the remaining test scenarios. However, when considering constraint 332 satisfaction, CPO fails to satisfy the constraint in all four tasks due to approximation errors, as 333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

previously reported in Ray et al. (2019). In contrast, CPPO successfully satisfies the constraint in all five environments, showing the effectiveness of the proposed recovery update . Referring to the learning curves in Circle scenarios, it can be seen that the primal-dual based CRL methods, i.e., PPO_lag and TRPO_lag, suffer from the slow and unstable update of the dual variable, causing the conservative performance in PointCircle and slow cost satisfaction in AntCircle. On the other hand, CPPO can achieves a faster learning speed in Circle environment by eliminating the need for the dual variable . Overall, the experimental results demonstrate the effectiveness of CPPO in solving the CRL problem.

Ablation Study: An ablation study was conducted to investigate the impact of the recovery update in CPPO. Figure 4 presents the reward performance and cost satisfaction of CPPO with and without the recovery update in the PointCircle environment. The results indicate that without the recovery update, CPPO achieves higher reward performance; however, the cost reaches 15, which significantly violates the cost constraint. In contrast, when the recovery update is applied, CPPO successfully satisfies the constraint, thereby demonstrating the importance of the recovery update in ensuring constraint satisfaction.

0.0

0.5

1.0

1.5

2.0

TotalEnvInteracts

1e5

0

100

200

300

Performance

0.0

0.5

1.0

1.5

2.0

TotalEnvInteracts

1e5

5

10

15

20

AverageEpCost

cppo

cppo w/o recovery

cost constraint

5 Limitations and Boarder Impact

Although our proposed method has shown its ability in test scenarios, there still exist some limitations. Firstly, CPPO method is an on-policy constrained RL, which suffers from lower sampling efficiency compared to other off-policy algorithms, potentially limiting its applicability in real-world scenarios. Additionally, the convergence of our method is not yet proven. However, we believe that our work will offer researchers a new EM perspective for using PPO-like algorithms to solve the problem of constrained RL, thereby leading to the development of more efficient and stable constrained RL algorithms.

6 Conclusion

In this paper, we have introduced a novel first-order Constrained Reinforcement Learning (CRL) method called CPPO. Our approach avoids the use of the primal-dual framework and instead treats the CRL problem as a probabilistic inference problem. By utilizing the Expectation-Maximization (EM) framework, we address the CRL problem through two key steps: the E-step, which focuses on deriving a theoretically optimal policy distribution, and the M-step, which aims to minimize the difference between the current policy and the optimal policy. Through the non-parametric representation of the policy using probability ratios, we convert the CRL problem into a convex optimization problem with a clear geometric interpretation. As a result, we propose an iterative heuristic algorithm that efficiently solves this optimization problem without relying on the dual variable. Furthermore, we introduce a recovery update strategy to handle approximation errors in cost evaluation and ensure constraint satisfaction when the current policy is infeasible. This strategy mitigates the impact of approximation errors and strengthens the capability of our method to satisfy constraints. Notably, our proposed method does not require second-order optimization techniques or the use of the primal-dual framework, which simplifies the optimization process. Empirical experiments have been conducted to validate the effectiveness of our proposed method. The results demonstrate that our approach achieves comparable or even superior performance compared to other baseline methods. This showcases the advantages of our method in terms of simplicity, efficiency, and performance in the field of Constrained Reinforcement Learning.

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

References

Abdolmaleki, A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., and Riedmiller, M. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920 , 2018.

Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained policy optimization. In International conference on machine learning , pp. 22-31. PMLR, 2017.

Altman, E. Constrained Markov decision processes , volume 7. CRC press, 1999.

Lasalle, J. The 'bang-bang' principle. IFAC Proceedings Volumes , 1(1):503-507, 1960. ISSN 1474-6670. 1st International IFAC Congress on Automatic and Remote Control, Moscow, USSR, 1960.

Le, N., Rathour, V. S., Yamazaki, K., Luu, K., and Savvides, M. Deep reinforcement learning in computer vision: a comprehensive survey. Artificial Intelligence Review , 55(4):2733-2819, 2022.

Li, Q., Peng, Z., Feng, L., Zhang, Q., Xue, Z., and Zhou, B. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence , 2022.

Liu, Z., Cen, Z., Isenbaev, V ., Liu, W., Wu, S., Li, B., and Zhao, D. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning , pp. 13644-13668. PMLR, 2022.

Paternain, S., Calvo-Fullana, M., Chamon, L. F., and Ribeiro, A. Safe policies for reinforcement learning via primal-dual methods. IEEE Transactions on Automatic Control , 2022.

Ray, A., Achiam, J., and Amodei, D. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708 , 7:1, 2019.

Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International conference on machine learning , pp. 1889-1897. PMLR, 2015a.

Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015b.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419):1140-1144, 2018.

Stooke, A., Achiam, J., and Abbeel, P. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning , pp. 9133-9143. PMLR, 2020.

Yang, T.-Y., Rosca, J., Narasimhan, K., and Ramadge, P. J. Projection-based constrained policy optimization. arXiv preprint arXiv:2010.03152 , 2020.

Yanushevsky, R. Modern missile guidance . CRC Press, 2018.

Yu, C., Velu, A., Vinitsky, E., Wang, Y ., Bayen, A., and Wu, Y . The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955 , 2021.

Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning , pp. 1094-1100. PMLR, 2020.

Zhang, Y., Vuong, Q., and Ross, K. First order constrained optimization in policy space. Advances in Neural Information Processing Systems , 33:15338-15349, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the constrained version of PPO that is devised for solving constrained MDP problems in the discounted reward setting with discounted cost constraints.\n\nSTRENGTHS:\nThe paper presents the constrained PPO. It also gives some analytical results and presents a heuristic algorithm that is seen to perform well numerically.\n\nWEAKNESSES:\n1. Inadequate literature survey. The authors should broaden the scope of their survey - in fact constrained actor critic was proposed in the paper: \"V.S.Borkar, An actor-critic algorithm for constrained Markov decision processes, Systems and Control Letters, 54(3):207-213, 2005'', for the full state case and \"S.Bhatnagar, An actor–critic algorithm with function approximation for discounted cost constrained Markov decision processes, Systems and Control Letters, 59(12): 760-766, 2010, in the function approximation setting.\n\n2. The authors point to the fact that they have not provided a convergence analysis for their algorithm even though other works in the literature such as the ones listed above do possess an asymptotic analysis of convergence.\n\n3. The results given in the text carry imprecise statements, for instance, Prop. 3.1 says \"if there are a sufficient number of sampled v, then E[v]=1 and E[vlog v] <= var(v-1). This has to be made precise. Does it  mean that in the limit that the number of samples goes to infinity, the statement is valid? Then, if so, the question will be what will be the form of the statement in terms of the number of samples N.\n\n4. The recovery update problem in Sec 3.2 is not clear - how it has been arrived at? \n\n5. Subsequently a heuristic procedure has been proposed and the authors say that an optimal solution to (5) is provably obtained, But if it is provably obtained, how is the procedure a heuristic procedure? \n\n6. After (4), it is said that (4) can be directly solved through existing convex optimization techniques. This is not clear since one needs A(s,a), A_c(s,a) etc. to be known in order to solve it directly. But that is not known and needs to be estimated. So not clear what the authors mean?\n****\n7. On further reading of the paper and supplementary material post the response of the authors, I feel the technical results are imprecise and flawed. For instance, there is no way to verify Assumption 3.5. Moreover, the statement of Proposition 3.1 seems to suggest that it depends on a \"sufficient number of sampled v\" but gives a bound on the true expectation. Why should the true expectation depend on the number of samples of v? Also, when you say \"sufficient number of sampled v\", what does it mean? How many v is a sufficient number?\n\nQUESTIONS:\n1. On page 2, is it \\pi(a|s) instead of \\pi(s|a)?\n2. Beginning of page 3: definitions of Q(s_t,a_t), V(s_t), Q_c(s_t,a_t) and V_c(s_t) are confusing. This is because you have a summation of t from 0 to \\infty, and then you condition on s_0=s_t, a_0=a_t etc. What does it mean? It is too confusing.\n3. After (1), how is d^\\pi defined?\n4. In (3), is there a relationship between d and \\delta? Can one be found from the other?\n\nLIMITATIONS:\nThe major limitation is in terms of a lack of credible analysis. Even the theoretical results presented are not precise, so nothing much can be said about the algorithm. In addition there are several typos and grammatical errors throughout the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Constrained Proximal Policy Optimization (CPPO) for Constrained Reinforcement Learning (CRL). The CPPO method is designed to overcome the limitations of existing methods by offering a first-order feasible region method that doesn't require dual variables or second-order optimization, and it is an incremental extension of the  CVPO algorithm (Constrained Variational Policy Optimization for Safe Reinforcement Learning). The improvement seems incremental, improving the computational efficiency of CVPO. The method is evaluated in different environment,  comparable or even superior performance to other baseline methods.  . However, the paper does not provide  a direct comparison between CPPO and CVPO.\n\nSTRENGTHS:\n- The authors propose a new first-order method for constrained RL. The method seems to be designed to be simple, and overcome  limitations of existing methods such as CPO and CVPO (e.g., the authors do not require the usage of a dual variable).\n\n- The proposed method demonstrates comparable or even superior performance compared to other baseline methods.\n\nWEAKNESSES:\n- Clarity: the paper could benefit from improvements in clarity and readability. The presentation of their ideas is somewhat dense and could be difficult for readers to follow. A concise description of the algorithm is missing.\n\n- Novelty and results: The method builds directly upon CVPO. The authors need to clearly highlight this fact when introducing their method, and clearly explain what are the differences. As of now, the changes seem incremental, and therefore the paper seems to lack in novelty. Furthermore, the paper does not provide  a direct comparison between CPPO and CVPO, especially in terms of computational complexity\n\n- No theoretical results are provided, therefore I'd have expected more extensive numerical results.\n\n- Typos and notation: there are several typos and errors in notation throughout the paper.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nAuthors briefly discuss limitations and broader impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel first-order feasible method, CPPO, for efficient constrained reinforcement learning. The proposed approach integrates the Expectation-Maximization (EM) framework to solve the policy optimization problem by treating the CRL as the probabilistic inference. In the E-step, CPPO calculates the optimal policy distribution within the feasible region. In the M-step, CCPO conducts a first-order update for policy optimization. The authors also propose an iterative heuristic algorithm from a geometric perspective to efficiently solve the E-step and a recovery update strategy to improve constraint satisfaction performance. They evaluate their algorithm in several benchmark environments. The reported results show comparable performance over other baselines in complex environments.\n\nSTRENGTHS:\n(1) The proposed algorithm converts the CRL problem into a convex optimization problem with a clear geometric interpretation which mitigates the impact of approximation errors and strengthens the capability of the proposed method to satisfy constraints.\n\n(2) Since the proposed method does not require second-order optimization techniques or the use of the primal-dual framework, the policy optimization process is largely simplified.\n\nWEAKNESSES:\n(1) I suggest further polishing and improving the mathematical formulation and notations to make them more rigorous. For example, the objectives and constraints in (5) (6) are represented by the dot product of two vectors. In my understanding, it should be easier to understand with a transpose on the top of the first vector. \n\n(2) The policy update strategy looks overly conservative. Figure 1 shows that only in case 3, the policy can be updated toward seeking a higher reward. Does this strategy result in over-conservativeness? I am also wondering in cases 1 and 3 why the solution is not at the feasible boundary (that is to find the feasible distribution to maximize A).\n\n(3) It is very impressive that CPPO can work well in the AntCircle and Push tasks. However, the experiment lacks sufficient baselines and environments for comparison. For example, the recovery update in Section 3.2.2 is similar to the idea in Yang et. al [1, 2], where an additional projection step is introduced to recover the safe policy, so I am wondering how does the author compare CPPO against these methods, both in theory and empirically. In addition, the results only present a subset of tasks in SafetyGym, so I wonder how the algorithm performs in other tasks.\n\n[1] Yang, Tsung-Yen, et al. \"Projection-based constrained policy optimization.\" arXiv preprint arXiv:2010.03152 (2020).\n\n[2] Yang, Tsung-Yen, et al. \"Accelerating safe reinforcement learning with constraint-mismatched baseline policies.\" International Conference on Machine Learning. PMLR, 2021.\n\nQUESTIONS:\n(1) This work seems like an on-policy version of CVPO, so I am wondering how does the proposed approach compare against CVPO in the experiments?\n(2) Could the authors provide more details about baseline implementations, such as the PPO-Lag and TRPO-lag? I am not able to find them in the supplementary material.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel CPPO method to solve the contained RL (CRL) problem. Specifically, CPPO leverages the probabilistic inference and converts the CRL problem formulation based on the probabilistic ratio, resulting in the first-order optimization solution. CPPO also develops the recovery update method to safely optimize the policy when there are inaccurate cost evaluations and infeasible solutions. \nThe resulting EM-based framework of CPPO shows its effectiveness across various safety gym scenarios compared to baselines.\n\nSTRENGTHS:\n1. CPPO solves the constrained RL based on first-order and does not use dual variables and second-order optimization, resulting in a simpler, intuitive (i.e., geometric perspective), and computationally efficient method. \n2. The recovery update method is developed thanks to CPPO's first-order optimization/geometric perspectives and shows its effectiveness (as shown in Figure 4).\n\nWEAKNESSES:\n1. As stated in Section 3.2.1, CPPO builds on CVPO and has two main differences (using advantage instead of q and using the probability ratio instead of directly calculating q). While  Some readers may understand these as a limited novelty. Possibly adding comparisons against CVPO in the evaluation section can convey the importance of these differences better. \n2. While I agree that CPPO reduces computational complexity, there are no empirical results in the evaluation section. Possibly, adding computation time results in the evaluation section can help. \n3. I understand the benefits of converting the CRL problem into first-order optimization, but it is unclear what potential limitations/disadvantages the first-order optimization may have. Could there be an approximation error compared to second-order optimization? Related to this concern, could the authors clarify further why Assumption 3.5 is a fair assumption?\n\nQUESTIONS:\n1. Could the CPO baseline or other baselines also apply a similar recovery update strategy (i.e., cases 1-3 described in Section 3.2.2)?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

comments: SUMMARY:
This paper studies constrained reinforcement learning problems. It proposes a new EM-type algorithm and designs a heuristic version for practical use. The authors also conduct some numerical experiments to validate the performance of the algorithm.

STRENGTHS:
1. The algorithm is first-order and thus computationally efficient in practice.
2. The numerical results look convincing.

WEAKNESSES:
1. The algorithm looks very similar to CVPO and only has some small modifications.
2. This paper does not have convincing theoretical analysis. For example, the authors claim that the heuristic algorithm in Section 3.3 will stop in just a few iterations (remark 3.6) but there are not any theoretical proofs. It would be better if the paper can provide some theoretical study, even only for simple cases.

QUESTIONS:
None.

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27

comments: SUMMARY:
This paper focuses on constrained reinforcement learning and proposes a method called Constrained Proximal Policy Optimization (CPPO). Experimental results demonstrate the improved performance in terms of episodic return and episodic cost.

STRENGTHS:
+ The proposed CPPO achieves improved balance between return and cost empirically.

WEAKNESSES:
- The novelty and contribution of this work is not clear. The difference between the proposed CPPO and CVPO does not seem to be very significant. Adopting the advantage value instead of Q-value is a natural extension. Besides, it is unclear how much sample complexity can be reduced by replacing $q$ by $v$. After all, $v$ should still satisfy the expectation constraint.
- The paper does not have any theoretical characterization of the proposed CPPO algorithm.

QUESTIONS:
- It is unclear why replacing Lagrangian based approach by the feasible region based approach can reduce the computational complexity. Solving a hard-constrained optimization problem seems to be more challenging. Please clarify.
- Can an off-policy version be developed for CPPO to improve its sampling efficiency?

LIMITATIONS:
The paper mentions that CPPO method is an on-policy constrained RL, which suffers from lower sampling efficiency compared to other off-policy algorithm.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-28