Title: Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Safe Reinforcement Learning

Conditioned Reinforcement Learning

Multi-task Reinforcement Learning
Abstract: Safe reinforcement learning (RL) focuses on training reward-maximizing agents subject to pre-defined safety constraints. Yet, learning versatile safe policies that can adapt to varying safety constraint requirements during deployment without retraining remains a largely unexplored and challenging area. In this work, we formulate the versatile safe RL problem and consider two primary requirements: training efficiency and zero-shot adaptation capability. To address them, we introduce the Conditioned Constrained Policy Optimization (CCPO) framework, consisting of two key modules: (1) Versatile Value Estimation (VVE) for approximating value functions under unseen threshold conditions, and (2) Conditioned Variational Inference (CVI) for encoding arbitrary constraint thresholds during policy optimization. Our extensive experiments demonstrate that CCPO outperforms the baselines in terms of safety and task performance while preserving zero-shot adaptation capabilities to different constraint thresholds data-efficiently. This makes our approach suitable for real-world dynamic applications.

Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning

Abstract

Safe reinforcement learning (RL) focuses on training reward-maximizing agents subject to pre-defined safety constraints. Yet, learning versatile safe policies that can adapt to varying safety constraint requirements during deployment without retraining remains a largely unexplored and challenging area. In this work, we formulate the versatile safe RL problem and consider two primary requirements: training efficiency and zero-shot adaptation capability. To address them, we introduce the Constraint-Conditioned Policy Optimization (CCPO) framework, consisting of two key modules: (1) Versatile Value Estimation (VVE) for approximating value functions under unseen threshold conditions, and (2) Conditioned Variational Inference (CVI) for encoding arbitrary constraint thresholds during policy optimization. Our extensive experiments demonstrate that CCPO outperforms the baselines in terms of safety and task performance, while preserving zero-shot adaptation capabilities to different constraint thresholds data-efficiently. This makes our approach suitable for real-world dynamic applications.

1 Introduction

Safe reinforcement learning (RL) has emerged as a promising approach to address the challenges faced by agents operating in complex, real-world environments [1], such as autonomous driving [2], home service [3], and UAV locomotion [4]. Safe RL aims to learn a reward-maximizing policy within a constrained policy set [5-10]. By explicitly accounting for safety constraints during policy learning, agents can better reason about the trade-off between task performance and safety constraints, making them well-suited for safety-critic applications [11].

Despite the advances in safe RL, the development of a versatile policy that can adapt to varying safety constraint requirements during deployment without retraining remains a largely unexplored area. Investigating versatile safe RL is crucial due to the inherent trade-off between task reward and safety requirement [12, 13]: stricter constraints typically lead to more conservative behavior and lower task rewards. For example, an autonomous vehicle can adapt to different thresholds for driving on an empty highway and crowded urban area to maximize transportation efficiency. Consequently, learning a versatile policy allows agents to efficiently adapt to diverse constraint conditions, enhancing their applicability and effectiveness in real-world scenarios [14].

This paper studies the problem of training a versatile safe RL policy capable of adapting to tasks with different cost thresholds. The primary challenges are two-fold:

(1) Training efficiency. A straightforward approach is to train multiple policies under different constraint thresholds, then the agent can switch between policies for different safety requirements. However, this method is sampling inefficient, making it unsuitable for most practical applications, as the agent may only collect data under a limited number of thresholds during training.

(2) Zero-shot adaptation capability. Constrained optimization-based safe RL approaches rely on fixed thresholds during training [15], while recovery-based safe RL methods require a pre-defined backup policy to correct agent's unsafe behaviors [5-7]. Therefore, current safe RL training paradigms face challenges in adapting the learned policy to accommodate unseen safety thresholds.

To tackle the challenges outlined above, we introduce the Conditioned Constrained Policy Optimization (CCPO) framework, a sampling-efficient algorithm for versatile safe reinforcement learning that achieves zero-shot generalization to unseen cost thresholds during deployment. Our method consists of two integrated components: Versatile Value Estimation (VVE) and Conditioned Variational Inference (CVI). The first VVE module is inspired by transfer learning in RL [16, 17], which utilizes value function representation learning to estimate value functions for the versatile policy under unseen threshold conditions. The second CVI module aims to encode arbitrary threshold conditions during policy training. Our key contributions are summarized as follows:

1. We frame safe RL beyond pre-defined constraint thresholds as a versatile learning problem. This perspective highlights the limitations of most existing constrained-optimization-based approaches and motivates the development of CCPO based on conditional variational inference. Importantly, CCPO can generalize to diverse unseen constraint thresholds without retraining the policy.

2. We introduce two key techniques, VVE and CVI, for safe and versatile policy learning. To the best of our knowledge, our method is the first successful online safe RL approach capable of achieving zero-shot adaptation for unseen thresholds while preserving safety. Our theoretical analysis further provides insights into our approach's data efficiency and safety guarantees.

3. We conduct comprehensive evaluations of our method on various safe RL tasks. The results demonstrate that CCPO outperforms baseline methods in terms of both safety and task performance for varying constraint conditions. The performance gap is notably larger in tasks with the highdimensional state and action space, wherein all baseline methods fail to realize safe adaptation.

2 Related Work

Safe RL has been approached through various methods. Some techniques leverage domain knowledge of the target problem to enhance the safety of an RL agent [18-26]. Another line of work employs constrained optimization techniques to learn a constraint-satisfaction policy [27, 1, 28], such as the Lagrangian-based approach [29-31], where the Lagrange multipliers can be optimized via gradient descent along with the policy parameters [32, 33, 12]. Alternatively, other works approximate the nonconvex constrained optimization problem with low-order Taylor expansions [15] or through variational inference [34], then solve for the dual variable using convex optimization [35-38]. However, most existing approaches consider a fixed constraint threshold during training, which can hardly be deployed for different threshold conditions after training.

Transfer learning in RL. The concept of transfer learning, also recognized as knowledge transfer, denotes a sophisticated technique that exploits external knowledge harnessed from various domains to enhance the learning trajectory of a specified target task [39]. Transfer learning in RL can be categorized from multiple perspectives, such as skill composition for novel tasks [40-43], parameter transfer [44], and feature representation transfer [45-47]. Among these, the methodologies leveraging Successor Features (SFs) [48, 17] are particularly relevant to this work. These methodologies operate under the assumption that the reward function can be deconstructed into a linear combination of features, and they further extend the successor representation to decouple environmental dynamics from rewards [16, 49]. However, most existing works using SFs consider transfer learning problems among tasks with different reward functions but not with different task conditions.

3 Problem Formulation

3.1 Safe RL with Constrained Markov Decision Process

Constrained Markov Decision Process (CMDP) M is defined by the tuple ( S , A , P , r, c, µ 0 ) [50], where S is the state space, A is the action space, P : S × A × S - → [0 , 1] is the transition function, r : S × A × S - → R is the reward function, and µ 0 : S - → [0 , 1] is the initial state distribution. CMDP augments MDP with an additional element c : S × A × S - → R + to characterize the cost of violating the constraint. Note that in this work we use a single constraint for ease of demonstration.

Versatile CriticLearning

Versatile Actor Training

Replay buffer

VVE:versatilevalue

CVI:conditionalvariationalinference

S,s',a,r,c,Ei

functionestimation

Conditioned E-step

(convexoptimization)

Q(s,ale) =y(s,a)z(e)

q*(· Is,ei), vei E E

S,Ei

=π(·)

Versatile M-step

∑Vπ(l∈i)

Versatile safe RL agent π(·I)

(supervised learning)

A safe RL problem is specified by a CMDP and a constraint threshold /epsilon1 - → [0 , + ∞ ) . Let π : S × A → [0 , 1] denote the policy and τ = { s 1 , a 1 , ... } denote the trajectory. We use shorthand f t = f ( s t , a t , s t +1 ) , f ∈ { r, c } for simplicity. The value function is V π f ( µ 0 ) = E τ ∼ π ,s 0 ∼ µ 0 [ ∑ ∞ t =0 γ t f t ] , f ∈ { r, c } , which is the expectation of discounted return under the policy π and the initial state distribution µ 0 . The goal of safe RL is to find the policy that maximizes the reward return while limiting the cost return under the pre-defined threshold /epsilon1 :

3.2 Versatile Safe RL beyond a Single Threshold

We consider the versatile safe RL problem beyond a single pre-defined constraint threshold. Specifically, we consider a set of thresholds /epsilon1 ∈ E and a constraint-conditioned policy space: π ( ·| /epsilon1 ) : S × A × E → [0 , 1] . We can then formulate the versatile safe RL problem as finding the optimal versatile policy π ∗ ( ·| /epsilon1 ) that maximizes the reward while sticking to the corresponding threshold condition on a range of constraint thresholds /epsilon1 ∈ E , i.e.,

The generated action is subsequently based on the state and threshold: a ∼ π ( s | /epsilon1 ) . A successful versatile training algorithm should satisfy as least two core properties:

Thresholds Sampling Efficiency: The training dataset D = ⋃ N i =1 D i is collected through a limited set of thresholds D i ∼ π ( ·| ˜ /epsilon1 i ) , ∀ /epsilon1 i ∈ ˜ E , with ˜ E ⊂ E , | ˜ E| = N , where N denotes the number of behavior policies with pre-specified constraint conditions during training.

Safety for Varying Thresholds: Given that the agent is trained under a restricted range of threshold conditions, ensuring safety when adapting to unseen thresholds is of significant importance.

We identify two key challenges to meet these requirements: (1) Q function estimation for unseen threshold conditions with limited behavior policy data and (2) encoding arbitrary safety constraint conditions in the versatile policy training. To address these challenges, we propose the ConstraintConditioned Policy Optimization (CCPO) method as follows.

4 Method

As illustrated in Figure. 1, CCPO is comprised of two parts. The first versatile critic learning part involves data collection via several behavior agents, each under their respective target constraint thresholds. The goal is to learn feature representations for both the state-action pair feature and the target thresholds, facilitating linear disentanglement of Q values and hence enabling generalization to unseen thresholds. This step is inspired by the concept of successor feature in RL transfer learning [16], and we term it Versatile Value Estimation (VVE).

In the second versatile agent training part, we train the policy to be responsive to a range of unseen thresholds based on the well-trained value functions. Our key insight is to adopt the variational inference safe RL framework [34], which allows us to compute the optimal non-parametric policy

distributions for various thresholds through convex optimization, and, subsequently, use them to train the constraint-conditioned policy via supervised learning. With this Conditional Variational Inference (CVI) step, the policy can achieve zero-shot adaptation to unseen thresholds without the necessity for behavior agents to collect data under corresponding conditions. We introduce each step as follows.

4.1 Versatile value estimation

It has been shown that inherent trade-offs exist between the reward and cost values for the optimal policy under varying cost thresholds [13]. Typically, stricter safety requirements tend to be accompanied by reduced rewards. Hence, the estimation of Q-value functions becomes increasingly crucial when dealing with unseen thresholds that are not encountered by the behavior agents. To this end, we propose the versatile critics learning module, which disentangles observations and target thresholds within a latent feature space. The assumption regarding the decomposition is as follows.

Assumption 1 (Critics linear decomposition) . The versatile Q functions Q ∗ f with respect to the optimal versatile policy π ∗ can be represented as:

where || ψ f ( s, a ) || ∞ ≤ K f is the feature for the function of the state-action pair ( s, a ) , K f are constants, and z ∗ f is the optimal constraint-conditioned policy feature, which only depends on the policy condition /epsilon1 for a specified task. The dimension of ψ f ( s, a ) and z ∗ f ( /epsilon1 ) are both M .

Note that Assumption 1 is reasonable and widely accepted in the RL transfer learning literature with successor features [17], as it is reasonable to find a high-dimensional feature space to decompose the Q functions into the product of feature functions ψ f ( s, a ) and the latent vectors z f ( /epsilon1 ) [51]. As shown in Theorem 1, by learning ψ f ( s, a ) and z f ( /epsilon1 ) jointly and adding norm constraints on the feature function || ψ f ( s, a ) || ∞ ≤ K f , we can efficiently encode the threshold information /epsilon1 into the Q functions and achieve accurate estimations for unseen thresholds. This is the basis for our method's data-efficient training. To further facilitate theoretical analysis, we assume that the optimal constraint-conditioned policy feature z ∗ f ( /epsilon1 ) can be approximated by polynomial functions:

Assumption 2 (Polynomial feature space) . The optimal constraint-conditioned policy feature z ∗ f can be approximated by z ∗ f ( /epsilon1 ) = Poly ( /epsilon1 , p ) + e , meaning each element of z ∗ f is a p -degree polynomial of /epsilon1 , and e is the remainder. Each component for e follows e j ∼ N (0 , σ 2 j ) , j = i, ..., M , and denote σ = max j σ j .

Note that the degree p corresponds to the z ( /epsilon1 ) model representation capability. Based on the above assumptions, we can derive the Q function estimation error bound as follows.

Theorem 1 (Bounded estimation error) . Denote /epsilon1 L and /epsilon1 H are the lower and upper bound of the target threshold interval for E . Suppose the threshold conditions { ˜ /epsilon1 i } i =1 , 2 ,...,N for behavior policies are selected to divide the interval [ /epsilon1 L , /epsilon1 H ] evenly, then with confidence level 1 -α , the estimation error of versatile Q functions conditioned on arbitrary /epsilon1 ∈ [ /epsilon1 L , /epsilon1 H ] can be bounded by:

where B ( p ) and β ( p ) are both functions of the polynomial degree p , and z α / 2 is the upper alpha quantile for the standard Gaussian distribution. The proof and detailed discussion of Theorem 1 and functions B ( p ) , β ( p ) are shown in Appendix B.1. It is worth noting that we normalize the threshold conditions /epsilon1 ∈ [ /epsilon1 L , /epsilon1 H ] to the interval [0 , 1] by /epsilon1 = ( /epsilon1 -/epsilon1 L ) / /epsilon1 H for numerical stability. Theorem 1 establishes that by decomposing the Q function into the product of ψ ( s, a ) and z ( /epsilon1 ) and jointly learning these components, we can guarantee a bounded estimation error for Q functions under unseen threshold conditions. Furthermore, we can derive the bounded cost violation for unseen thresholds and /epsilon1 -sample complexity analysis based on the theorem, both of which are discussed in proposition 1 and remark 1. We also provide empirical verification of Q function estimation in Appendix D.1.

4.2 Conditioned variational inference

Given well-trained versatile Q functions in the previous VVE module, we aim to encode arbitrary threshold constraints during policy learning in the versatile policy learning part. We utilize the safe

RL as inference framework to achieve this goal, as it decomposes safe RL to a convex optimization followed by supervised learning, both stages readily accommodating varying target thresholds. In contrast to the classical view of safe RL aiming to find the most-rewarding actions while satisfying the constraints, the probabilistic inference perspective finds the feasible (constraint-satisfying) actions most likely to have been taken given future success in maximizing task rewards [34].

Following the RL as inference literature [52, 53], we consider an infinite discounted reward formulation. In the condition that the constraint threshold is /epsilon1 i , we denote O = O ( s, a ) as the optimality variable of a state-action pair ( s, a ) , which indicates the reward-maximizing (optimal) event by choosing an action a at a state s . Then for a given trajectory τ , the likelihood of being optimal is proportional to the exponential of the discounted cumulative reward: p ( O = 1 | τ ) ∝ exp( ∑ t γ t r t / α ) , where α is a temperature parameter. Since the probability of getting a trajectory τ under the conditioned policy π ( ·| /epsilon1 i ) can be expressed as p π ( ·| /epsilon1 i ) ( τ ) = p ( s 0 ) ∏ t ≥ 0 p ( s t +1 | s t , a t ) π ( a t | s t , /epsilon1 i ) , the lower bound for the log-likelihood of optimality given the conditioned policy π ( ·| /epsilon1 i ) is:

where the inequality follows Jensen's inequality, and q ( τ | /epsilon1 i ) is an auxiliary trajectory-wise variational distribution conditioned on /epsilon1 i . J ( q, π | /epsilon1 i ) in equation (5) is the evidence lower bound (ELBO) to reach the reward optimality under condition /epsilon1 i . Since q ( τ | /epsilon1 i ) = p ( s 0 ) ∏ t ≥ 0 p ( s t +1 | s t , a t ) q ( a t | s t , /epsilon1 i ) , we have the following ELBO over the state and constraint conditioned action distribution q ( a | s, /epsilon1 i ) :

where ρ q ( s | /epsilon1 i ) is the stationary state distribution induced by q ( ·| s, /epsilon1 i ) and ρ 0 , θ refers to the parameters for policy π , and p ( θ ) is a prior distribution over the parameters. Note we overload q by using it both in q ( a | s, /epsilon1 i ) and q ( τ | /epsilon1 i ) .

We utilize the Expectation-Maximization (EM)-based RL algorithms, which alternate to improve J ( q, π | /epsilon1 i ) in terms of q ( τ | /epsilon1 i ) and p π ( ·| /epsilon1 i ) ( τ ) to improve the likelihood of optimality [54, 55, 34]. Denote the feasible distribution family for the conditioned variational distribution q ( ·| s, /epsilon1 i ) as:

which is a set of all the state-conditioned action distributions that satisfy the safety constraint specified by /epsilon1 i . Then the E-step optimizes q ( τ | /epsilon1 i ) to maximize the reward return within the trust region of the old policy and within Π /epsilon1 i Q , while the M-step aims to minimize the KL divergence between p π ( ·| /epsilon1 i ) ( τ ) and q ( τ | /epsilon1 i ) by updating the parametrized policy in a supervised learning fashion. Since Off-policy deep RL techniques can be used during training, the EM updating steps are more data efficient.

The key strength of using the variational inference framework lies in its ability to encode arbitrary threshold conditions during policy learning, as shown in (7), a feat that is challenging for other methods, such as those based on primal-dual algorithms. Optimizing the factorized lower bound J ( q, θ | /epsilon1 i ) w.r.t q ( ·| /epsilon1 i ) within the feasible distribution family and the policy parameter θ iteratively with one single threshold /epsilon1 i via EM yields the CVPO method [34], which is the basis of our method. We introduce the modified constraint-conditioned E-step and M-step as follows.

Constraint-Conditioned E-step: The conditioned E-step aims to find the optimal variational distribution q ( ·| /epsilon1 i ) ∈ Π /epsilon1 i Q that maximizes the reward return while satisfying the safety condition defined by /epsilon1 i . At the j -th iteration, We can write the ELBO objective w.r.t q as a constrained

optimization problem (see Appendix B for proofs):

where ˆ Q f ( ·| /epsilon1 i ) is the versatile Q functions as introduced in section 4.1, the first inequality constraint represents the constraint defined in (7) and the last term in the constraint is the trust region with the old policy defined by KL distance κ . Inspired by [54], we use the solution of the optimal variational distribution q ∗ i = q ∗ i ( a | s, /epsilon1 i ) for arbitrary safety constraint /epsilon1 i , which has the closed form:

where Z ( s, /epsilon1 i ) is a normalizer to make sure q ∗ i is a valid distribution, and the dual variables η ∗ i and λ ∗ i are the solutions of the following convex optimization problem (see Appendix B for details):

Then we can encode arbitrary safety constraints by calculating the optimal distribution q ∗ i ( a | s, /epsilon1 i ) regarding the corresponding condition /epsilon1 i efficiently with (9). The term q ∗ i ( a | s, /epsilon1 i ) means when conditioned on /epsilon1 i , the probability of taking a at s for the optimal feasible policy.

Versatile M-step: After the constraint-conditioned E-step, we obtain a set of optimal feasible variational distribution q ∗ i = q ∗ i ( ·| s, /epsilon1 i ) for each constraint threshold /epsilon1 i . In the versatile M-step, we aim to improve the ELBO (5) w.r.t the policy parameter θ for /epsilon1 i ∈ E .

Using a Gaussian prior for each threshold-conditioned policy, this problem can be further converted to the following supervised-learning problem with KL-divergence constraints [54, 55]:

where E is the set for all the sampled versatile policy conditions { /epsilon1 i } in fine-tuning stage of training. The constraint in (12) is a regularizer to stabilize the policy update. The proposed CCPO for versatile safe RL and implementation details are summarized in Appendix C.

4.3 Theoretical analysis

Proposition 1 (Bounded safety violation) . With the threshold conditions ˜ /epsilon1 i ∈ ˜ E for behavior policies selected to divide the target condition interval [ /epsilon1 L , /epsilon1 H ] evenly, and with confidence level 1 -α , the cost violation of versatile policy under arbitrary threshold condition /epsilon1 ∈ [ /epsilon1 L , /epsilon1 H ] is bounded as:

The proof is shown in Appendix B.1. Proposition 1 ensures that the cost violation of the versatile safe RL agent on unseen thresholds can be bounded if the selected behavior policy conditions divide the interval [ /epsilon1 L , /epsilon1 H ] evenly. We can observe that the bound (13) is proportional to √ K 2 c M . Since larger K c and M correspond to a wider range of threshold conditions, this safety violation bound is related to the interval range. Also, we provide the complexity analysis for the /epsilon1 -sample, i.e., the estimation error corresponding to the number of behavior policies N as shown in remark 1.

Remark 1 ( /epsilon1 -sample complexity analysis) . The estimation error for Q functions and safety violation bound decreases as the number of behavior policies N increases. The decreasing rate is proportional to 1 N β ( p ) , where the exponent of N is related to p , which is the representation capabilities of the model to represent the constraint-conditioned policy feature z ( /epsilon1 ) . When p increases, β also increases as shown in the Appendix B.1, which means when the model capability is high, the proposed method significantly becomes more data-efficient.

The functions β ( p ) , B ( p ) are shown in Appendix B.1. Since CCPO is under the 'RL as inference' framework, we also enjoy many benefits as revealed in previous works, such as the optimality guarantees and training robustness [34], see Appendix A for details.

5 Experiment

We aim to answer five major questions in the experiment section: (1) Can we achieve versatile safe RL by simply applying a linear combination of single-threshold policies (2) Can we combine safe RL algorithms with a constraint-conditioned actor to achieve versatile safe RL? (3) What is the performance of our proposed CCPO method in versatile safe RL tasks in terms of constraint violation and reward? (4) How /epsilon1 -efficient CCPO is compared to exhaustively training the safe RL agents? (5) What is the contribution of each component in CCPO contribute to the overall performance? We adopt the following experiment setting to address these questions.

Task. The simulation environments are from a publicly available benchmark [56]. We consider two tasks (Run and Circle) and four robots (Ball, Car, Drone, and Ant) which have been used in many previous works as the testing ground [13-15]. For the Run task, the agents are rewarded for running fast between two boundaries and are given constraint violation cost if they run across the boundaries or exceed an agent-specific velocity threshold. For the Circle task, the agents are rewarded for running in a circle but are constrained within a safe region smaller than the target circle's radius. We name the tasks as Ball-Circle , Car-Circle , Drone-Circle , Drone-Run , and Ant-Run .

Constraint-conditioned Baselines. We design these baselines by directly integrating the threshold as a part of the state in the CMDP tuple, ¯ s = [ s ; /epsilon1 ] . The policy is optimized with behavior policy conditions only. We adopt commonly used off-policy safe RL algorithms, SAC-Lag and DDPG-Lag , and name the proposed baselines as V-SAC-Lag and V-DDPG-Lag .

Policy linear combination baselines. We also compare our method with single-threshold policy combinations. Denote /epsilon1 as an unseen target threshold for adaptation, and /epsilon1 1 , /epsilon1 2 as two behavior policy conditions closest to /epsilon1 . Then the policy for /epsilon1 is the combination of π ( ·| /epsilon1 1 ) , π ( ·| /epsilon1 2 ) :

This method is designed for both threshold interpolation and extrapolation, i.e., the coefficients w 1 or w 2 can be negative. This baseline draws inspiration from the safe control theory, which suggests the safe input component is proportional to the conservativeness level [57]. To this end, we use two strong on-policy methods PPO-Lag and TRPO-Lag to train the single-threshold agents and named the corresponding baselines as C-PPO-Lag and C-TRPO-Lag . More baselines and results can be found in the Appendix.

Metrics: We compare the methods in terms of episodic reward (the higher, the better) and episodic constraint violation cost (the lower, the better) on each evaluated threshold condition, which have been used in many related works [14, 58]. For all the results shown in section 5.1 and 5.3, the behavior policy conditions are ˜ E = { 20 , 40 , 60 } and the threshold conditions for evaluation are set to be { 10 , 15 , ..., 70 } . We take the average of the episodic reward (Avg. R) and constraint violation (Avg. CV) as the main comparison metrics. The constraint violation for threshold /epsilon1 is defined as:

As mentioned in previous works [58], the general evaluation criteria in safe RL are: (1) method A is better than method B if A achieves better safety performance than B. (2) If both A and B satisfy constraints, the one with the higher reward is better. We also report the average performance solely on unseen thresholds (Avg. R-G and Avg. CV-G) to characterize the adaptation capability. More experiments can be found in Appendix D.

5.1 Main Results and Analysis

The evaluation results are shown in Figure. 2 and Table 1. We shade the two safest agents with the lowest averaged cost violation values.

First, we can observe that our method outperforms the modified versatile safe RL baselines V-SAC-Lag and V-DDPG-Lag that directly concatenate the threshold condition into the state. Although V-SAC-Lag or V-DDPG-Lag gets higher rewards on simple-dynamics tasks Ball-Circle ,

CCPO

V-DDPG-Lag

C-TRPO-Lag

Thresholdcondition

Ball-Circle

Car-Circle

Drone-Circle

Drone-Run

Ant-Run

Reward

900

800

700

600

500

450

400

800

700

600

500

450

400

800

009

700

500

350

500

350

500

400

300

000

250

400

300

300

250

400

300

200

200

200

200

200

10

20

30

40

50

60

70

150

10

20

30

40

50

60

70

10

20

304050

60

150

10

20

30

40

50

60

70

10

20

30

40

50

60

70

100

120

175

140

70

50

60

80

09

100

80

150

125

100

120

100

80

40

40

60

75

60

30

40

OS

40

20

20

20

25

20

10

0

0

0

10

20

30

40

50

60

70

10

20

30

40

50

60

70

10203040506070

10

20

30

40

50

60

70

10

20

30

40

50

60

70

Thresholdcondition

and Car-Circle , their cost violation values are significantly larger than the proposed CCPO cost. In the Drone-Circle , Drone-Run , and Ant-Run tasks characterized by highly-nonlinear robot dynamics, all the baseline methods exhibit poor generalization when being exposed to different thresholds, thus leading to high cost violation values. This limitation arises due to the inadequacy of utilizing only a limited number of behavior policies for versatile policy training in tasks with high-dimensional observation and action spaces. Insufficient conditions prevent the actor from effectively distinguishing between different constraint conditions, thus resulting in large performance variance even for behavior policies, and large cost violations on unseen thresholds. The poor safety

performance when generalizing to different threshold conditions indicates the necessity of studying versatile safe RL methods.

Second, although the linear combination of single-threshold policies performs well on Ball-Circle and Car-Circle with simple robot dynamics, it can hardly handle Drone-Circle , Drone-Run , and Ant-Run tasks - the interpolation method has a significant reward drop at unseen thresholds. It is because, for Ball-Circle and Car-Circle task, the action dimensionality is low and the dynamics are simple, thus directly applying linear combination may work in these settings. However, for Drone-Circle and Drone-Run , the high nonlinearity in agent dynamics and the large action space will make this naive approach fail to perform well. These results show that the concepts from the control theory that the safety-critical control component is proportional to the conservativeness level can not be directly used in versatile safe RL with high-dimensional settings, which further indicates the necessity of the versatile safe RL method.

Finally, from Table 1, we can clearly see that our proposed CCPO method learns a versatile safe RL policy that can generalize well to unseen thresholds with low cost-violations and high rewards. Also, from Figure. 2, we can observe the performance of the CCPO method has a smooth relation with respect to threshold conditions, which indicates it can efficiently encode the threshold conditions into the versatile safe RL agent.

5.2 Evaluation of /epsilon1 -sampling efficiency

The proposed algorithm is /epsilon1 -sampling-efficient as it satisfies the first requirement thresholds sampling efficiency mentioned in section 3.2: it is able to train the versatile safe RL agent with limited behavior policies for data collection. In this experiment, we aim to answer the question: How /epsilon1 -efficient CCPO is compared to exhaustively training the safe RL agents? We compare our method with C-TRPO , which is the strongest baseline method as shown in Table. 1 with different behavior policy set ˜ E = { 20 , 40 , 60 } and ˜ E ′ = { 20 , 30 , 40 , 50 , 60 , 70 } , where | ˜ E ′ | = 2 | ˜ E| . The algorithms are evaluated on threshold conditions E = { 10 , 15 , ..., 70 } , and the averaged performance is reported in Table. 2.

We observe that CCPO exhibits significant /epsilon1 -efficiency over C-TRPO-Lag : CCPO outperforms C-TRPO-Lag significantly in terms of both safety and reward, even with a smaller set of behavior policy conditions. This comparison showcases the remarkable /epsilon1 -efficiency of our approach. In fact, our method demonstrates at least 2 times greater /epsilon1 -sampling efficiency compared to exhaustively training safe RL agents using C-TRPO-Lag .

5.3 Ablation Study

To study the influence of VVE, and CVI components of CCPO introduced in the section 4.1 and 4.2, we conduct an ablation study by removing each component from the full CCPO algorithm. The ablation experiment results are shown in Table. 3. We shade the safest agent with the lowest averaged cost violation value. We can also observe significant safety performance (cost violation) degradation and task performance (reward) drop if we remove the VVE module (versatile critic learning) or CVI module (versatile actor learning) since they help us learn versatile Q functions more accurately and

improve the generalizability of learned actors. Removing them will result in the bad estimation of for state-action pair value function and lead to poor policy generalization capability.

6 Conclusion

In this study, we pioneered the concept of versatile safe reinforcement learning (RL), presenting the Constraint-Conditioned Policy Optimization (CCPO) algorithm. This approach adapts efficiently to different and unseen cost thresholds, offering a promising solution to safe RL beyond pre-defined constraint thresholds. With its core components, Versatile Value Estimation (VVE) and Conditioned Variational Inference (CVI), CCPO facilitates zero-shot generalization for constraint thresholds. Our theoretical analysis further offers insights into the constraint violation bounds for unseen thresholds and the sampling efficiency of the employed behavior policies. The extensive experimental results reconfirm that CCPO effectively adapts to unseen threshold conditions and is much safer and more data-efficient than baseline methods. The limitations include that our method would be more computationally expensive than the primal-dual-based safe RL approaches due to the convex optimization problems in the constraint-conditioned E-step. One potential negative social impact is that the misuse of this work in safety-critical scenarios can cause unexpected damage. We hope our findings can inspire more research in studying the generalization capability in safe RL.

7 Acknowledgment

The work is partially supported by Google Deepmind with an unrestricted grant. The authors also want to acknowledge the support from the National Science Foundation under grants CNS-2047454.

References

Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330 , 2022.

David Isele, Alireza Nakhaei, and Kikuo Fujimura. Safe reinforcement learning on autonomous vehicles. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 1-6. IEEE, 2018.

Hongyuan Ding, Yan Xu, Benjamin Chew Si Hao, Qiaoqiao Li, and Antonis Lentzakis. A safe reinforcement learning approach for multi-energy management of smart home. Electric Power Systems Research , 210:108120, 2022.

Zengyi Qin, Kaiqing Zhang, Yuxiao Chen, Jingkai Chen, and Chuchu Fan. Learning safe multiagent control with decentralized neural barrier certificates. arXiv preprint arXiv:2101.05436 , 2021.

Tsung-Yen Yang, Tingnan Zhang, Linda Luu, Sehoon Ha, Jie Tan, and Wenhao Yu. Safe reinforcement learning for legged locomotion. arXiv preprint arXiv:2203.02638 , 2022.

Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters , 6(3):4915-4922, 2021.

Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety critics for exploration. arXiv preprint arXiv:2010.14497 , 2020.

Vanshaj Khattar, Yuhao Ding, Bilgehan Sel, Javad Lavaei, and Ming Jin. A cmdp-within-online framework for meta-safe reinforcement learning. In The Eleventh International Conference on Learning Representations , 2022.

Yuhao Ding and Javad Lavaei. Provably efficient primal-dual reinforcement learning for cmdps with non-stationary objectives and constraints. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 7396-7404, 2023.

Honghao Wei, Arnob Ghosh, Ness Shroff, Lei Ying, and Xingyu Zhou. Provably efficient model-free algorithms for non-stationary cmdps. In International Conference on Artificial Intelligence and Statistics , pages 6527-6570. PMLR, 2023.

Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems , 5, 2021.

Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708 , 7, 2019.

Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Jie Tan, Bo Li, and Ding Zhao. On the robustness of safe reinforcement learning under observational perturbations. arXiv preprint arXiv:2205.14691 , 2022.

Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. arXiv preprint arXiv:2302.07351 , 2023.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning , pages 22-31. PMLR, 2017.

André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems , 30, 2017.

Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning , pages 501-510. PMLR, 2018.

Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757 , 2018.

William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention. arXiv preprint arXiv:1707.05173 , 2017.

Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Thirty-Second AAAI Conference on Artificial Intelligence , 2018.

Haonan Yu, Wei Xu, and Haichao Zhang. Towards safe reinforcement learning with a safety editor policy. arXiv preprint arXiv:2201.12427 , 2022.

Zuxin Liu, Hongyi Zhou, Baiming Chen, Sicheng Zhong, Martial Hebert, and Ding Zhao. Safe model-based reinforcement learning with robust cross-entropy method. arXiv preprint arXiv:2010.07968 , 2020.

Yuping Luo and Tengyu Ma. Learning barrier certificates: Towards safe reinforcement learning with zero training-time violations. Advances in Neural Information Processing Systems , 34, 2021.

Aivar Sootla, Alexander I Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning , pages 20423-20443. PMLR, 2022.

Baiming Chen, Zuxin Liu, Jiacheng Zhu, Mengdi Xu, Wenhao Ding, Liang Li, and Ding Zhao. Context-aware safe reinforcement learning for non-stationary environments. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 10689-10695. IEEE, 2021.

Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Yihang Yao, Hanjiang Hu, and Ding Zhao. Towards robust and safe reinforcement learning with benign off-policy data. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 21586-21610. PMLR, 23-29 Jul 2023.

Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(1):1437-1480, 2015.

Yannis Flet-Berliac and Debabrota Basu. Saac: Safe reinforcement learning as an adversarial game of actor-critics. arXiv preprint arXiv:2204.09424 , 2022.

Shalabh Bhatnagar and K Lakshmanan. An online actor-critic algorithm with function approximation for constrained markov decision processes. Journal of Optimization Theory and Applications , 153(3):688-708, 2012.

Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research , 18(1):6070-6120, 2017.

Yarden As, Ilnura Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy optimization via bayesian world models. arXiv preprint arXiv:2201.09802 , 2022.

Qingkai Liang, Fanyu Que, and Eytan Modiano. Accelerated primal-dual policy optimization for safe reinforcement learning. arXiv preprint arXiv:1802.06480 , 2018.

Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074 , 2018.

Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning , pages 13644-13668. PMLR, 2022.

Ming Yu, Zhuoran Yang, Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning. arXiv preprint arXiv:1910.12156 , 2019.

Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. arXiv preprint arXiv:2010.03152 , 2020.

Shangding Gu, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang, Zheng Tian, Jun Wang, Alois Knoll, and Yaodong Yang. Multi-agent constrained policy optimisation. arXiv preprint arXiv:2110.02793 , 2021.

Dohyeong Kim and Songhwai Oh. Efficient off-policy safe reinforcement learning using trust region conditional value at risk. IEEE Robotics and Automation Letters , 7(3):7644-7651, 2022.

Zhuangdi Zhu, Kaixiang Lin, Anil K Jain, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. arXiv preprint arXiv:2009.07888 , 2020.

Pashootan Vaezipoor, Andrew C Li, Rodrigo A Toro Icarte, and Sheila A Mcilraith. Ltl2action: Generalizing ltl instructions for multi-task rl. In International Conference on Machine Learning , pages 10497-10508. PMLR, 2021.

Brandon Araki, Xiao Li, Kiran Vodrahalli, Jonathan DeCastro, Micah Fry, and Daniela Rus. The logical options framework. In International Conference on Machine Learning , pages 307-317. PMLR, 2021.

Emanuel Todorov. Compositionality of optimal control laws. Advances in neural information processing systems , 22, 2009.

Geraud Nangue Tasse, Steven James, and Benjamin Rosman. A boolean task algebra for reinforcement learning. Advances in Neural Information Processing Systems , 33:9497-9507, 2020.

Janarthanan Rajendran, Aravind Srinivas, Mitesh M Khapra, P Prasanna, and Balaraman Ravindran. Attend, adapt and transfer: Attentive deep architecture for adaptive transfer from multiple sources in the same domain. arXiv preprint arXiv:1510.02879 , 2015.

Chen Ma, Dylan R Ashley, Junfeng Wen, and Yoshua Bengio. Universal successor features for transfer reinforcement learning. arXiv preprint arXiv:2001.04025 , 2020.

Jaekyeom Kim, Seohong Park, and Gunhee Kim. Constrained gpi for zero-shot transfer in reinforcement learning. Advances in Neural Information Processing Systems , 35:4585-4597, 2022.

Mark Nemecek and Ronald Parr. Policy caches with successor features. In International Conference on Machine Learning , pages 8025-8033. PMLR, 2021.

Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural computation , 5(4):613-624, 1993.

Ida Momennejad, Evan M Russek, Jin H Cheong, Matthew M Botvinick, Nathaniel Douglass Daw, and Samuel J Gershman. The successor representation in human reinforcement learning. Nature human behaviour , 1(9):680-692, 2017.

Eitan Altman. Constrained Markov decision processes: stochastic modeling . Routledge, 1999.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):17981828, 2013.

Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909 , 2018.

Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Yihang Yao, Hanjiang Hu, and Ding Zhao. Towards robust and safe reinforcement learning with benign off-policy data. 2023.

Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920 , 2018.

Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Heess, and Martin Riedmiller. Relative entropy regularized policy iteration. arXiv preprint arXiv:1812.02256 , 2018.

Sven Gronauer. Bullet-safety-gym: Aframework for constrained reinforcement learning. 2022.

Tianhao Wei and Changliu Liu. Safe control algorithms using energy functions: A uni ed framework, benchmark, and new directions. In 2019 IEEE 58th Conference on Decision and Control (CDC) , pages 238-243. IEEE, 2019.

Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09303 , 2023.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. Advances in Neural Information Processing Systems , 30, 2017.

Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. Advances in neural information processing systems , 29, 2016.

Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. arXiv preprint arXiv:1912.13465 , 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n1. This paper introduces the Conditioned Constrained Policy Optimization (CCPO) framework, consisting of two modules: (1) Versatile Value Estimation (VVE) for approximating value functions under unseen threshold conditions, and (2) Conditioned Policy Inference (CPI) for encoding arbitrary constraint thresholds during policy optimization.\n2. The proposed method can enable learning versatile safe policies that can adapt to varying safety constraint requirements during deployment without retraining.\n\nSTRENGTHS:\n1. training efficiency and zero-shot adaptation capability: the trained policy can adapt to varying safety constraint requirements during deployment without retraining.\n2. The theoretical analysis for the q-function estimation and constraint violations.\n3. The extensive experiments are implemented to demonstrate the effectiveness of the method.\n\nWEAKNESSES:\nSee questions.\n\nQUESTIONS:\n1. How reasonable/strong is assumption 2? Especially if latent vectors are highly dimensional. In experiments, what model do you use to estimate the latent vector z(\\epsilon)? \n2. What is the main technical challenge in extending CVPO [30] to the constraint-conditional scenarios?\n3. How does this method scale when there are multiple constraints?\n4. How does this work compare with the following missing literature with similar topics:\nMeta CMDP: \nKhattar, V., Ding, Y., Sel, B., Lavaei, J., & Jin, M. (2022, September). A CMDP-within-online framework for Meta-Safe Reinforcement Learning. In The Eleventh International Conference on Learning Representations.\nNon-stationary CMDP:\nDing, Y., & Lavaei, J. (2023, June). Provably efficient primal-dual reinforcement learning for CMDPs with non-stationary objectives and constraints. In Proceedings of the AAAI Conference on Artificial Intelligence.\nWei, H., Ghosh, A., Shroff, N., Ying, L., & Zhou, X. (2023, April). Provably Efficient Model-Free Algorithms for Non-stationary CMDPs. In International Conference on Artificial Intelligence and Statistics (pp. 6527-6570). PMLR.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses Safe Reinforcement Learning with varying safety constraint requirements (Versatile Safe Reinforcement Learning). The paper proposes to learn a versatile Q function, which is decomposed as the product of feature functions of state and action,  and the latent features of the constraint thresholds. The Q function will be used in a variational EM algorithm to find the constraint-satisfying policy that maximizes the return. The proposed method shows good safety and decent returns in experimental robotic tasks such as Run and Circle, outperforming simple constraint-conditioned and linear combination baselines.\n\nSTRENGTHS:\n- The target problem is interesting and seems novel in the setting of Safe RL\n- The paper has relevant theoretical analysis to back up the method\n\nWEAKNESSES:\n- The writing is unclear, especially in the pretraining phase. It is better to have an algorithm in the main text to summarize the process and connect the components in the method. It is hard to connect Fig. 1 to the method section. For example, where are the two bottom boxes in the pretraining phase described in the method section?\n- The experiment result overall is not really good. Although the safety criteria show good performance, the reward counterpart is not. In many tasks, the proposed method underperforms the baselines regarding the rewards. How to ensure that the trade-off is acceptable and the behaviour is desirable in Safe RL setting? Overall, more tasks should be tested to confirm if the lower performance is a consistent issue in the approach. Furthermore, there are claims on sample efficiency, yet no report on it can be found in the main text. It should be noted that running time should be considered as well because as admitted in the discussion, the method can be slow due to complicated optimization steps.\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nThe method is complicated with different stages of optimization. Can it work in higher dimensional domains such as visual state space?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSafe reinforcement learning approaches have so far focused on the case where\ncost thresholds are static and known a priori. In contrast, this work addresses\nRL environments where cost constraints may change over time. To do this, the\nproposed algorithm learns a policy which is conditioned on the given cost\nthreshold, allowing the policy to adapt to new cost thresholds at test time.\nThe policy is derived from a pair of learned Q functions, one for rewards and\none for costs. The Q functions are decomposed into a feature for state-action\npairs and a feature for the safety threshold. The safety threshold feature uses\na relatively low-dimensional model in order to provide better generalization\nwithout needing to see a large set of thresholds at training time, making\ntraining more efficient. In experiments, the proposed algorithm outperforms\nLagrangian approaches to safety in a suite of relatively high-dimensional\nbenchmarks.\n\nSTRENGTHS:\nThe problem of versatlie safe RL is interesting, potentially important, and (to\nmy knowledge) novel. The threshold in CMDP-based safe RL work is often\narbitrary, so it seems useful to be able to modify that threshold without\nretraining the system. Moreover, there may be some environments where the cost\nthreshold changes over time.\n\nThe proposed high-level approach is a very elegant solution to this problem. I\nparticularly like the idea of separating out the effect of $\\epsilon$ into a\nlower-parameter model which should generalize better than a full neural model.\nThe way this allows for efficient training without sacrificing generalizability\nto new safety thresholds is quite impressive. Similarly, separating out Q-value\nlearning from policy construction is an interesting way to generalize while\nmaintaining the power of neural learning.\n\nThe experiments are fairly extensive and show impressive results. In\nparticular, CPPO demonstrates much better safety than other approaches on most\nbenchmarks, while maintaining comparable performance. I think Figure 2 is a\ngood illustration of the advantages of CPPO, showing how the number of observed\ncost closely tracks the allowed cost. This suggests that CPPO is using all of\nits available safety budget to achieve high rewards.\n\nWEAKNESSES:\nI think the problem statement could do with a bit more motivation. While I do\nbelieve the versatile safe RL problem is interesting, it's not clear to me how\noften these kinds of problems actually arise. The given example of a car\noperating in highway vs. urban environments provides some intuition, but I'm\nnot clear on what the cost signal is that would be changing in that case. An\nextra example or more explanation might help clarify this.\n\nI found the development in Section 4 to be difficult to follow. I'm not very\nfamiliar with the work on RL as inference, so maybe this explanation is\nsufficient for more expert readers. For less expert readers, I think it would\nhelp to add a bit more intuition or maybe some examples if feasible.\n\nFor the experiments, I think it would help to include some baselines from the\nsafe RL literature beyond basic Lagrangian approaches. For example, you might\ncompare with CPO or Conservative Safety Critics (among others). These\napproaches are generally tailored to a single threshold, but a comparison could\nbe made by training them with a strict threshold then using a more relaxed\nthreshold at test time. In principle, CPPO should achieve a higher reward since\nit is able to take advantage of the increased safety budget.\n\nQUESTIONS:\nDo you have any intiution about why TRPO-Lag achieves better safety on Ant-Run\nthan CCPO? Is there some feature of the benchmark or the algorithm which makes\nthis a particularly challening case for CCPO?\n\nLIMITATIONS:\nThe authors have adequately addressed some limitations and potential societal\nimpacts in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce the versatile safe RL problem, in which we want to maximize the expected reward, subject to an expected constraint, but for a range of constraint thresholds $\\epsilon\\in \\mathcal E$ (compared to the traditional safe RL problem, which only aims to find a solution for one $\\epsilon$). This is an interesting problem setting, with good motivation and potential real world application in learning a policy that works under many circumstances. The authors also provide an algorithm CVPO,  that nicely combines the control as inference framework with the EM framework to improve the likelihood of optimality while trying to stay within a feasible set.\n\nQUESTIONS:\nOn Page 4, Line 134 - what is $K_f$? is it a constant for $f \\in \\{r, c\\}$? Similarly, on line 146, I understand that $Poly(\\epsilon, p)$ is a $p$-degree polynomial of $\\epsilon$, but shouldn't it have a vector output since $z_f^*$ is a vector? Also, on line 158, why is it okay to normalize epsilon to [0, 1] since realistically constraint thresholds can be arbitrary?\n\nOn Page 5, Line 197, the authors say - \"The key strength of using the variational inference framework lies in its ability to encode arbitrary threshold conditions during policy learning, as shown in (7), a feat that is challenging for other methods, such as those based on primal-dual algorithms.\" Could you elaborate why this is challenging for other methods?\n\nFor testing generalization to unseen thresholds, the authors train on data from $\\epsilon$=20,40,60 and test with $\\epsilon$=10,20,30,40,50,60,70. This is relatively close to the thresholds in the training data. It would be nice to see this algorithm trained with thresholds in [20, 30, 40] and tested with thresholds in [50, 60, 70]. Ideally, the algorithm may be able to provide feasible policies, but it is also important to test if the reward achieved is highest possible. If it is able to handle thresholds well outside the training set, that would be a good thing and it will show that the algorithm is quite versatile. If it is unable to do so, then, it should be discussed a bit further i.e. what unseen thresholds can we realistically expect good performance on?\n\nSuggestions related to language and typos:\n1. On Page 5 - \"We utilize the safe RL as inference framework ...\" - since this is proposed by this work, it would be better to say \"we introduce the ...\".\n\nEDIT (16 Aug 2023): Updated score from 6->7.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The study investigated a versatile, safe reinforcement learning problem and proposed the Conditioned Constrained Policy Optimization (CCPO) algorithm. CCPO may be helpful in ensuring safety under the settings of various constraint limits. Moreover, the experiment results demonstrate that CCPO works well.

STRENGTHS:
1. The problem is interesting.
2. The method may be useful for safe RL research.

WEAKNESSES:
1. The code the study provided is not convenient to run; we have to download the code file by file. Could you provide the whole code? Such that we can examine the algorithm better.
2. Please check the whole paper, and improve the writing quality, e.g., on page 3, line 101, please check the sentence, and revise “z(1)” as “(1)”.

QUESTIONS:
1. How can we handle non-convex optimization while ensuring safety?
2. If the cost limits are linear, what will happen?
3. Could you compare it with CPO-based methods?

LIMITATIONS:
1. The balance between reward and cost is not handled well.
2. Although the Lagrangian-based methods can not perform well on various cost-limit settings, their reward performance is better than CCPO.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25