Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning
Authors: Yihang Yao
Zuxin Liu
Zhepeng Cen
Jiacheng Zhu
Wenhao Yu
Tingnan Zhang
Ding Zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: Safe Reinforcement Learning
Conditioned Reinforcement Learning
Multi-task Reinforcement Learning

Abstract: Safe reinforcement learning (RL) focuses on training reward-maximizing agents subject to pre-defined safety constraints. Yet, learning versatile safe policies that can adapt to varying safety constraint requirements during deployment without retraining remains a largely unexplored and challenging area. In this work, we formulate the versatile safe RL problem and consider two primary requirements: training efficiency and zero-shot adaptation capability. To address them, we introduce the Conditioned Constrained Policy Optimization (CCPO) framework, consisting of two key modules: (1) Versatile Value Estimation (VVE) for approximating value functions under unseen threshold conditions, and (2) Conditioned Variational Inference (CVI) for encoding arbitrary constraint thresholds during policy optimization. Our extensive experiments demonstrate that CCPO outperforms the baselines in terms of safety and task performance while preserving zero-shot adaptation capabilities to different constraint thresholds data-efficiently. This makes our approach suitable for real-world dynamic applications.

Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning
Yihang Yao*!, Zuxin Liu"!, Zhepeng Cen', Jiacheng Zhu 1.3
Wenhao Yu², Tingnan Zhang", Ding Zhao?
Carnegie Mellon University, 2 Google DeepMind, 8 Massachusetts Institute of Technology Equal contribution, fyihangya, zuzinl)aandrew. cmu. edu
Abstract
Safe reinforcement learning (RL) focuses training reward-maximzzing agents subject to pre-defined safety constraints. Yet, learning versatile safe policies that can adapt to varying safety constraint requirements during deployment without re- training remains a largely unexplored and challenging area. In this work, we formu- late the versatile safe RI. problem and consider two primary requirements: training efficiency and zero-shot adaptation capability. To address them, we introduce the Constraint-Conditoone Policy Optimization (CCPO) framework, consisting of two key modules: (I) Versatile Value Estimation (VVE) for approximating value func- tions under unseen threshold conditions, and (2) Conditioned Variational Inference (CVT) for encoding arbitrary constraint thresholds during policy optimization. Our extensive experiments demonstrate that CCPO outperforms the baselines terms of safety and task performance, while preserving zero-shot adaptation capabilities to different constraint thresholds data-efficiently. This makes our approach suitable for real-world dynamic applications.
Introduction
Safe reinforcement learning (RL) has emerged as a promising approach to address the challenges faced by agents operating complex, real-world environments [1. such as autonomous driving (2]. home service 3. and UAV locomotion [4]. Safe RL. aims to learn a neward-maximizie
policy within a constrained policy set 15-C By explicitly accounting for safety constraints during policy learning. agents can better reason about the trade-off between task performance and safety constraints, making them well-suited for safety-critic applications [Л].
Despite the advances in safe RL, the development of a versatile policy that can adapt to varying safety constraint requirements during deployment without retraining remains largely unexplored area. Investigating versatile safe RL is crucial due to the inherent trade-off between task reward and safety requirement 212 [K] stricter constraints typically lead to more conservative behavior and lower task rewards. For example, an autonomous vehicle can adapt to different thresholds for driving an empty highway and crowded urban area to maximize transportation efficiency. Consequently. learning versatile policy allows agents to efficiently adapt to diverse constraint conditions, enhancing their applicability and effectiveness real-world scenarios [4].
This paper studies the problem of training a versatile safe RL policy capable of adapting to tasks with different cost thresholds. The primary challenges two-fold:
(1) Training efficiency. A straightforward approach to train multiple policies under different constraint thresholds, then the agent can switch between policies for different safety requirements. However, this method is sampling inefficient, making it unsuitable for most practical applications, as the agent may only collect data under limited number of thresholds during training.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(2) Zero-shot adaptation capability, Constrained optimization based safe RL approaches rely on fixed thresholds during training [T]. while recovery-based safe RL methods require a pre-defined backup policy to correct agent's unsafe behaviors [5 Therefore, current safe RI. training paradigms face challenges in adapting the learned policy to accommodate unseen safety thresholds.
To tackle the challenges outlined above, we introduce the Conditioned Constrained Policy Optimiza- tion (CCPO) framework, sampling-efficient algorithm for versatile safe reinforcement learning that achieves zero-shot generalization to unseen cost thresholds during deployment. Our method consists of two integrated components: Versatile Value Estimation (VVE) and Conditioned Variational Infer- ence (CVI). The first VVE module is inspired by transfer learning in RL [I 17]. which utilizes value function representation learning to estimate value functions for the versatile policy under unseen threshold conditions. The second CVI module aims to encode arbitrary threshold conditions during policy training. Our key contributions are summarized as follows:
1. We frame safe RL beyond pre-defined constraint thresholds as a versatile learning problem.
This perspective highlights the limitations of most existing sasrrraindd-oiimiiatto----eeeeee approaches and motivates the development of CCPO based on conditional variational inference. Importantly, CCPO can generalize diverse unseen constraint thresholds without retraining the policy.
2. We introduce two key techniques, VVE and CVI, for safe and versatile policy learning. To the best of our knowledge, our method is the first successful online safe RL approach capable of achieving zero-shot adaptation for unseen thresholds while preserving safety. Our theoretical analysis further provides insights into our approach's 's data efficiency and safety guarantees.
We conduct comprehensive evaluations of our method on various safe RL tasks. The results demonstrate that CCPO outperforms baseline methods in terms of both safety and task performance for varying constraint conditions. The performance gap is notably larger in tasks with the high- dimensional state and action space, wherein all baseline methods fail to realize safe adaptation.
2 Related Work
Safe RL has been approached through various methods. Some techniques leverage domain knowledge of the target problem to enhance the safety of an RL. agent 18 26 Another line of work employs constrained optimization techniques to learn constraint-satiifaationn policy 27
28 such as the Lagrangian-base approach 29 [1), where the Lagrange multipliers can be optimized via gradient Alternatively, other works approximate the non- convex constrained optimization problem with low-order Taylor expansions [5] or through variational inference K+ , then solve for the dual variable using convex optimization 35 38 However, most existing approaches consider a fixed constraint threshold during training. which can hardly be deployed for different threshold conditions after training.
descent along with the policy parameters
33
33 12].
Transfer learning in RL. The concept of transfer learning, also recognized as knowledge transfer, denotes sophisticated technique that exploits external knowledge harnessed from various domains to enhance the learning trajectory of a specified target task [39]. Transfer learning in RL can be
categorized from multiple perspectives, such skill composition for novel tasks 20 45), parameter
transfer [44] and feature representation transfer 45 Among these, the methodologies leveraging Successor Features (SFs) 48 17
are particularly relevant to this work. These methodologies operate under the assumption that the reward function can be deconstructed into a linear combination of features, and they further extend the successor representation to decouple environmental dynamics from rewards [T 49). However, most existing works using SFs consider transfer learning problems among tasks with different reward functions but not with different task conditions.
3 Problem Formulation
3.1 Safe RL with Constrained Markov Decision Process
Constrained Markov Decision Process (CMDP) M is defined by the tuple rr,. 500
where S is the state space, A is the action space, P : S x A x S + [0, 1] is the transition function, S x A S - IR is the reward function, and pg : S > [0. 1] is the initial state distribution. CMDP augments MDP with an additional element e: : the constraint. Note that in this work use a single constraint for case of demonstration.
x - IR+ to characterize the cost of violating

---

Versatile Critic Learning
Versatile Actor Training
Replay buffer
VVE: versatile value
CVT: conditional variational Inference
,,s,,a,r,c,&
function estimation
Conditioned E-step
(convex optimization)
(6. ((s,a) I(())
q"(: 16, Y,, E
s. x 16) (RD)
Versatile M-step (supervised learning)
V v (e)
Versatile safe RL x(( le)
Figure I: Proposed framework
A safe RL problem is specified by CMDP and constraint threshold [0. +oc). Let π S x A + [0. 1] denote the policy and т (1..) denote the trajectory. We use shorthand fr f(s,, at 8t+1),f
cr for simplicity. The value function is V/" (No) [r,c), which is the expectation of discounted return under the pol- icy и and the mitial state distribution μoo The goal safe RL is find the policy that maximizes the reward return while limiting the cost return under the pre-delined threshold
f. ?
π arg max V (μO), 8.1. (io S.
3.2 Versatile Safe RL beyond a Single Threshold
We consider the versatile safe RL problem beyond single pre-defined constraint threshold. Specifically, consider set of thresholds
and constraint-conditiones policy space:
T(-Ie) S x A & + [o, 1]. We can then formulate the versatile safe RL problem as finding the optimal versatile policy x² (-le) that maximizes the reward while sticking the corresponding threshold condition on a range of constraint thresholds e E E E,, i.e.
(+le) arg max V.- (HO); s.t. (iT Se,
EE
(2)
The generated action is subsequently based the state and threshold: T(sec. A successful versatile training algorithm should satisfy as least two core properties:
(1) Thresholds Sampling Efficiency: The training dataset D = U.N, D, is collected through a
limited set of thresholds D; ~ (-l6,)) Ve; E, with c E, E = N, where N denotes the number of behavior policies with pre-specified constraint conditions during training.
(2) Safety for Varying Thresholds: Given that the agent is trained under a restricted range of threshold conditions, ensuring safety when adapting unseen thresholds is of significant importance.
We identify two key challenges to meet these requirements: (1) Q function estimation unseen threshold conditions with limited behavior policy data and (2) encoding arbitrary safety constraint conditions in the versatile policy training. To address these challenges, we propose the Constraint- Conditioned Policy Optimization (CCPO) method as follows.
Method
As illustrated in Figure. D CCPO is comprised of two parts. The first versatile critic learning part involves data collection via several behavior agents, each under their respective target constraint thresholds. The goal is to learn feature representations for both the state-action pair feature and the target thresholds, facilitating linear disentanglement of Q values and hence enabling generalization to unseen thresholds. This step is inspired by the concept of successor feature in RL transfer learning [6], and we term it Versatile Value Estimation (VVE).
In the second versatile agent training part, we train the policy to be responsive to range of unseen thresholds based on the well-trained value functions. Our key insight to adopt the variational inference safe RL framework [4] which allows us to compute the optimal con-aarmettric policy

---

distributions for various thresholds through convex optimization, and, subsequently, use them to train the constrait-coonditonne" policy via supervised learning. With this Conditional Variational Inference (CVI) step, the policy can achieve zero-shot adaptation to unseen thresholds without the necessity for behavior agents to collect data under corresponding conditions. We introduce each step as follows.
4.1 Versatile value estimation
It has been shown that inherent trade-offs exist between the reward and cost values for the optimal policy under varying cost thresholds [K]. Typically, stricter safery requirements tend to be accompa- nied by reduced rewards. Hence, the estimation of Q-value functions becomes increasingly crucial when dealing with unseen thresholds that are not encountered by the behavior agents. To this end, we propose the versatile critics learning module, which disentangles observations and target thresholds within a latent feature space. The assumption regarding the decomposition is as follows.
Assumption 1 (Crities linear decomposition), The versatile Q functions Qi with respect to the optimal versatile policy nt can be represented as:
QF(s, alc) ((((s.))=(((e) fe [r.c).
(3)
where [/bbe(s, alo I. Ke is the feature for the function of the state-action pair (s. a). Kt are constants, and ziz the optimal constraint-conditionel policy feature, which only depends on the policy condition for specified task. The dimension of shr(s, a) and zi(e) are both M.
Note that Assumption T is reasonable and widely accepted in the RL transfer learning literature with successor features [7], it is reasonable to find nigh-dimensional feature space to decompose the Q functions into the product of feature functions wr(ss, a) and the latent vectors zr(c) [5I]. As the
shown in Theorem by learning ( a) and zr(c) jointly and adding norm constraints feature function (s, a> I Kr, we can efficiently encode the threshold information into the Q functions and achieve accurate estimations for unseen thresholds. This is the basis for method's data-efficient training. To further facilitate theoretical analysis, we assume that the optimal constraint-conddiionee policy feature zÿ (e) can be approximated by polynomial functions:
Assumption 2 (Polynomial feature space). The optimal oonstraint-condition
policy feature zē can be approximated by zi(e) = P)ly(E, + e, meaning each element of zł is a p-degree polynomial of e, and e is the remainder. Each component for e fooloow Ej ~ NN, ),) = i,..., M, and denote maxj T-.
Note that the degree p corresponds to the z(e) model representation capability, Based on the above assumptions, we can derive the Q function estimation error bound follows.
Theorem 1 (Bounded estimation error). Denote e. and en are the lower and upper bound of the target threshold interval for E. Suppose the threshold conditions {E:]s=1,2..... for behavior policies are selected to divide the interval (e,,, cH) evenly, then with confidence level 1 the estimation error of versatile Q functions conditioned on arbitrary e E [EL,EH] can be bounded by:
2p/2B(p) N
VORGGM
(4)
where B(p) and B(p) are both functions of the polynomial degree p. and 2//2 is the upper alpha quantile for the standard Gaussian distribution. The proof and detailed discussion of Theorem [ and functions (pp), (pp) are shown in Appendix [. It is worth noting that we normalize the threshold conditions E [Er.,
the interval [0, 1] by e (e - ct.)/En l for numerical stability. Theorem ! establishes that by decomposing the Q function into the product of vb(s, a) and z(s) and jointly learning these components, we can guarantee bounded estimation error for Q functions under unseen threshold conditions. Furthermore, we can derive the bounded cost violation for unseen thresholds and e-sample complexity analysis based on the theorem, both of which are discussed in proposition We also provide empirical verification of Q function estimation in Appendix D.I
and remark I
4.2 Conditioned variational inference
Given well-trained versatile Q functions in the previous VVE module, we aim to encode arbitrary threshold constraints during policy learning in the versatile policy learning part. We utilize the safe

---

RL as inference framework to achieve this goal, as it decomposes safe RL to convex optimization followed by supervised learning, both stages readily accommodating varying target thresholds. In contrast to the classical view of safe RI. aiming to find the most-rewarding actions while satisfying the constraints, the probabilistic inference perspective finds the feasible (constraint-aatifyinng actions most likely to have been taken given future success in maximizing task rewards ke
we consider an infinite discounted reward formu- lation. In the condition that the constraint threshold is é;, we denote O = [(s, a) as the optimality variable of state-action pair (s, a), which indicates the reward- -anamdmixxiiiingg (optimal) event by choosing an action a at state Then for given trajectory the likelihood of being optimal is pro- portional to the exponential of the discounted cumulative reward: p(O 1|7) x exp(E. 2'r:/ where a is a temperature parameter. Since the probability of getting a trajectory T under the con- ditioned policy x (+1E2) can be expressed as Pa(-)ea) (7) p(*o) Ileoo (8+++ 8e, at) (a) [ 6)), the lower bound for the log-likelihood of optimality given the conditioned policy (-Ies) is:
Following the RL inference literature 22
p(0 (p(())
log Pr(()e)) (O =) log E ok 4))
(T(())
pl0 = 1/T) (17))=((())()))
E le) log
(5)
g(T(())
α E T (ea)
r'd a (Dka
T(g mle),,
t=0
where the inequality follows Jensen's 's incquality, and q( (e) is an auxiliary trajectory- variational
distribution conditioned on 4, [(g x/<x) in equation (5) is the evidence lower bound (ELBO) to reach
the reward optimality under condition eG. Since ((T}c, = p(50) Ie20 P( [s,, a [g(0//( 44), we
have the following ELBO over the state and constraint conditioned action distribution g(a)s, e:):
[lg,0kc) =B
Σ Y' - (inn- + logr(()) t=0
(6)
where (x((() is the stationary state distribution induced by (-118, ()) and Po- 0 refers to the parame-
ters for policy æ, and p(0) is prior distribution over the parameters. Note we overload q by using both in g(als, e.) and ((T)e:).
We utilize the J[g, x (4) in terms of (((())) and (T) to improve the likelihood of optimality [54, 35 34]
Dnpectation-Maximizatee
(EM)-based RL algorithms, which alternate improve
Denote the feasible distribution family for the conditioned variational distribution q(-18, [8, e3) as:
I = {q(a|s, e:): Er-g(-la))
Σ t=0
ral S a S..
(7)
which is of all the state-conditioned action distributions that satisfy the safety constraint specified by e;. Then the E-step optimizes q(T(es) to maximize the reward return within the trust region of the old policy and within IIg. while the M-step aims to minimize the KL divergence between Pr(-lez)) (T) and ((Tle:) by updating the parametrized policy in supervised learning fashion. Since Off Off-policy deep RI. techniques can be used during training, the EM updating steps are more data efficient.
The key strength of using the variational inference framework lies in its ability to encode arbitrary threshold conditions during policy learning, as shown in 7 a feat that is challenging for other methods, such as those based on primal-dual algorithms. Optimizing the factorized lower bound J(q, 8|c;) w.r.t g-l|c) within the feasible distribution family and the policy parameter 0 iteratively with one single threshold 6 via EM yields the CVPO method [4) which is the basis of our method. We introduce the modified constraint-conditiona E-step and M-step as follows.
Constraint-Conditionee E-step: The conditioned E-step aims to find the optimal variational distribution q(-les) E HT that maximizes the reward return while satisfying the safety condition defined by eq. At the j-th iteration, We can write the ELBO objective w.r.t g constrained
5

---

optimization problem (see Appendix
for proofs):
max E Pe g(als,e))
q (als, Qr" (s,
/
(8)
s.t. EE
(s, da S 6;
Ep, [DLL (1 (a)s, ei) IITE, (-lei))
S K;
where Qr(-lez) is the versatile Q functions introduced in section 4.1 the first inequality constraint represents the constraint defined in 7 and the last term in the constraint is the trust region with the old policy defined by KL distance k. Inspired by [54) we use the solution of the optimal variational distribution of = 9, (als, Ei) for arbitrary safety constraint which has the closed form:
T8, (+|Es) Z(s,
(
-; 3 (+(c)
q.
exp
(9)
7
where (ss, (2) is a normalizer make sure o.' a valid distribution, and the dual variables ny and \ are the solutions of the following convex optimization problem (see Appendix B for details):
( - x.(c(1)))))
min g(T, As) = A5G + MEKRE A.0:20
log (())))))
exp
(10)
Then we can encode arbitrary safety constraints by calculating the optimal distribution of (als, ei) regarding the corresponding condition e, efficiently with (9). The term of (a(8,6,) means when conditioned on E,, the probability taking a at s for the optimal feasible policy.
E-step, we obtain a set of optimal feasible variational distribution of 07 (-|s, for each constraint threshold q; In the versatile M-step, we aim to improve the ELBO w.r.t the policy parameter 6 for e; E
Versatile M-step: After the constraint-condition
e(0 Ep, a.Eg [log Te(aa s, <i]] [t]] + log(ple;)
(11)
Using Gaussian prior for each threshold-condittoned policy, this problem be further converted to the following supervised-lcariing problem with KL-divergence constraints [54] 55]:
max 1P..
Σ Eut [log (a(ass, e1)] s.t. Ep, [DKL (Te, (als, (.)]| 4()) S Vi, (12)
where& E is the set for all the sampled versatile policy conditions {e:} in fine-tuning stage of training.
The constraint in (12) is regularizer to stabilize the policy update. The proposed CCPO for versatile
safe RL and implementation details are summarized Appendix C
4.3 Theoretical analysis
Proposition (Bounded safety violation). With the threshold conditions E; E E for behavior policies selected to divide the target condition interval (eL, (s) evenly, and with confidence level 1 a, the cost violation of versatile policy under arbitrary threshold condition [EL, EH) is bounded as:
B(p) N8(p)
Vo(polc) -
Vo²K?M,
(13)
The proof shown in Appendix B. Proposition T eesurees that the cost violation of the versatile safe RL agent unseen thresholds can be bounded if the selected behavior policy conditions divide the interval eH] evenly. We can observe that the bound (13 is proportional to / /KFM. Since larger K₂ and M correspond to a wider range of threshold conditions, this safety violation bound is related to the interval range. Also, we provide the complexity analysis for the c-sample, i.e., the estimation error corresponding to the number of behavior policies N as shown remark [
Remark 1 (c-sample complexity analysis). The estimation error for Q functions and safety violation bound decreases the number of behavior policies N increases. The decreasing rate proportional to N where the exponent of N is related to p, which is the representation capabilities of the model to represent the construint-conditonn as shown the Appendix B. which means when the model capability high, the proposed method significantly becomes more data- -ata-efficiens.
policy feature z(e). When p increases, B also increases

---

The functions (p)) (p) are shown in Appendix B.1 Since CCPO is under the "RL as inference" framework, we also enjoy many benefits as revealed in previous works, such as the optimality guarantees and training robustness [4]. see Appendix A for details.
5
Experiment
We aim to answer five major questions the experiment section: (1) Can we achieve versatile safe RL by simply applying linear combination of single-threshold policies (2) Can we combine safe RI. algorithms with constraint-conditioned actor to achieve versatile safe RI.? (3) What is the performance of our proposed CCPO method in versatile safe RL tasks in terms of constraint violation and reward? (4) How e-efficient CCPO is compared to exhaustively training the safe RL agents? (5) What is the contribution of each component in CCPO contribute to the overall performance? We adopt the following experiment setting to address these questions.
Task. The simulation environments are from publicly available benchmark [56]. We consider two tasks (Run and Circle) and four robots (Ball, Car, Drone, and Ant) which have been used in many previous works as the testing ground [3 For the Run task, the agents are rewarded for running fast between two boundaries and are given constraint violation cost if they run across the boundaries or exceed an agent-specific velocity threshold. For the Circle task, the agents are rewarded for running in a circle but are constrained within safe region smaller than the target circle's radius. We name the tasks Ball-Circle, Car-Circla
Drone-Rui Run, and
Constraint-condittonee Baselines. We design these baselines by directly integrating the threshold as a part of the state in the CMDP tuple, ø = [s;c]. c]. The policy is optimized with behavior policy conditions only. We adopt commonly used off- off-policy safe RL algorithms, SAC-Lag and DDPG-Lag. and name the proposed baselines as V-SAC- and V-DDPG-Lag.
Policy linear combination baselines. We also compare our method with single-threshon policy combinations. Denote as an unseen target threshold for adaptation, and 61, 62 as two behavior policy conditions closest Then the policy for is the combination of (11()),((-(())))
(-Ic)- = wi (-1G) + (22(- W1 = (c2 - c)/(c2 - -),, w/ = (c- (1/022)/22------11/))) (2)
(14)
This method is designed for both threshold interpolation and extrapolation, i.e., the coefficients wi or w2 can be negative. This baseline draws inspiration from the safe control theory, which suggests the safe input component is proportional to the conservativeness level [57]. To this end, we use two strong on-policy methods PPO-Lag and TRPO-Lag to train the single-the single-hrresoodd agents and named the corresponding basclines as C C-PPO-Lag and TRPO- More baselines and results can be found in the Appendix.
Metrics: We compare the methods in terms of episodic reward (the higher, the better) and episodic constraint violation cost (the lower, the better) on each evaluated threshold condition, which have been used in many related works 14 58 For all the results shown in section 5.1 and 5.3 the behavior policy conditions are & = (20, 40, 60) and the threshold conditions for evaluation are set to be (10, 15, ..., 70). We take the average of the episodic reward (Avg. R) and constraint violation (Avg. CV) as the main comparison metrics. The constraint violation for threshold e is defined as:
CV max Σ,q - c)
(15)
As mentioned in previous works [88] the general evaluation criteria in safe RL are: (1) method A is better than method B if A achieves better safety performance than B. (2) If both and B satisfy constraints, the one with the higher reward is better. We also report the average performance solely on unseen thresholds (Avg. R-G and Avg. CV-G) to characterize the adaptation capability, More experiments can be found in Appendix D
5.1 Main Results and Analysis
The evaluation results are shown in Figure. lowest averaged cost violation values.
2 and Table I We shade the two safest agents with the
First, we can observe that our method outperforms the modified versatile safe RL baselines V-SAC-Lag and V-DDPG-Lag that directly concatenate the threshold condition into the state. Al- though V-SAC-Lag or V-DDPG-Lag gets higher rewards on simple-dynamics tasks Ball-Circle,
7

---

Table 1: Evaluation results of proposed CCPO method and the proposed versatile safe RL baselines. the higher reward, better. 4: lower constraint violation (minimal 0), better. The models are evaluated on a series of threshold conditions and we report the averaged reward and constraint violation values on all evaluation thresholds and generalized thresholds. Each value is reported as mean standard deviation for 50 episodes and 5 seeds. We shade the two safest agents with the lowest averaged cost violation values.
Constraint-conditionedd
Lincar combination
Task
Stats
CCPO (ours)
V-SAC-Lag
V-DDPG-Lag
C-PPO-Lag
C-TRPO-Lag
Avg. R
710.86+20:47
174214+
762.61=58.65
637.85+14003
699.38 1851.99 94
Avg. CV
0.59+0.31
5,32+5.00
2.81+1.12
3.11+1.64
4.50=0.08
Ball-Cirele
Avg. R-G f
699. 999.04=2048
766.52+22.59
756.67=58.48
667.89
699.14 14=205
Avg. CV-G
0.8310442
6.29=5.72
3.53:1.26
3.4021.75
5.5920.25
Avg. R 1
406.06+6..0
448.82 15.8.2
440.01+2.59
461.14=1.39
Avg. CV 4
1.60#0.91
12.1844.65
14.48+8. 14
9.09+1.52
7.84+1.71
Car-Circle
Avg. R-G1
401.53a5.59
331.19&11.00
445.32a17.42
438.3143.03
460.72a1.15
Avg. CV-G
1.4940.38
12.74x4.32
14.63+8.69
11.07=1.58
9. 9.1422.01
Avg. R
630.55:40.03
693.69+22.37
734.58=49.66
192.64a23.13
380.77+18.22
Avg. CV
0.32+0.38
13.2448.80
19.62+11.15
0,45+0.38
6.55+1.95
Drone-Circle
Avg. R-G
625.51a40.12
699, 1924,14
430.29&48.43
342.77a19.06
291.87
Avg. CV-G
0.47a0.55
14.97=10.10
19.44x10.36
0.21
7,23+2.03
Avg. R
458. 69.12.98
355.61+35.44
244.60=48.29
398.88 88:2152
461.70=491
Avg. CV
0,23+0.25
8.66+4430
11,3349.63
9,46+5.63
47.97+3.49
Drone-Run
Avg. R-G
455.64211.8
354.61+33.34
236.61=43:49
386.77430.09
464.07=6.61
Avg. CV-G
0.33a0.37
9.9624.54
12.72a9.91
11.1827.46
60.3924.32
Avg. R f
660.88+4.88 82
615.73+91.99
194,75+172.35
636.06+6,78
329.83=7.5 84
Avg. CV
3.13+1.67
8.47=3.55
23.69+30.42
5.16&1.59
0,22+0.1 17
Ant-Run
Avg. R-G f
660.07+5.20 26
626.27284.61
592.50=173.01
620. 660.4649.90 9.99
605.07
Avg. CV-G
3.25+1.48
7.76x11.83
22.9019.39
6.73a2.32
0.0320.06
V-DDPG-Lag
C-TRPO-Lag
Threshold condition
Ball-CCirlee
Car-Cirkk
Drone-Cirole
Drome Rum
Rum
Reward
Cost
Threshold condition
Figure 2: Results of zero-shot adaption to different cost returns. Each column is a task. The x-axis is the threshold condition. The first row shows the evaluated reward, and the second row shows the evaluated cost under different target costs. All plots are averaged among 5 random seeds and 50 trajectories for each seed. The solid line is the mean value, and the light shade represents the area within standard deviation. We train the versatile agent with behavior policy conditions E {20, 40, 60}, and evaluate it on E {10, 15.... 70).
and Car-Circle, their cost violation values are significantly larger than the proposed CCPO cost. In the Prone-Circle Prone-Rume and Ant-Rum tasks characterized by highly- dynamics, all the baseline methods exhibit poor generalization when being exposed to different thresholds, thus leading to high cost violation values. This limitation arises due to the inadequacy of utilizing only a limited number of behavior policies for versatile policy training high-dimensional observation and action spaces. Insufficient conditions prevent the actor from effectively distinguishing between different constraint conditions, thus resulting in large performance variance even for behavior policies, and large cost violations unseen thresholds. The poor safety
robot
tasks with

---

performance when generalizing to different threshold conditions indicates the necessity of studying versatile safe RL methods.
Second, although the linear combination of single-thresoold policies performs well on Ball-Circle and Car-Circle with simple rohot dynamics, it can hardly handle Drone-Circll Drone-Run, and Ant-Run tasks the interpolation method has a significant reward drop at unseen thresholds. It is because, for Ball-Circle and Car-Circle task, the action dimensionality is low and the dynamics are simple, thus directly applying linear combination may work in these settings. However, for Drone-Circle and Drone-Run, the high nonlinearity in agent dynamics and the large action space will make this naive approach fail to perform well. These results show that the concepts from the control theory that the safety-critical control component is proportional to the conservativeness level can not be directly used in versatile safe RL with high-dimensional settings, which further indicates the necessity of the versatile safe RI. method.
Finally, from Table | we can clearly see that our proposed CCPO method learns a versatile safe RL policy that can generalize well to unseen thresholds with low cost-violations and high rewards. Also, from Figure. 2 we can observe the performance of the CCPO method has a smooth relation with respect threshold conditions, which indicates it can efficiently encode the threshold conditions into the versatile safe RL agent.
5.2 Evaluation of e-sampling efficiency
The proposed algorithm is s-saplling-- efficient as satisfies the first requirement thresholds sampling efficiency mentioned in section 3.2 it is able to train the versatile safe RL agent with limited behavior policies for data collection. In this experiment, we aim to answer the question: How e-fficientt CCPO is compared to exhaustively training the safe RL agents? We compare our method with C-TRPO, which is the strongest baseline method as shown in Table. with different behavior policy set
E = (20, 40, 60} and E' = {20, 30, 40, 50, 60, 70), where | 2 The algorithms are evaluated on threshold conditions& E = (10, 15, .... 70), and the averaged performance is reported in Table.
We observe that CCPO exhibits significant e-efficiency over C-TRPO-Lag: CCPO outperforms C-TRPO-Lag significantly in terms of both safety and reward, even with a smaller set of behavior policy conditions. This comparison showcases the remarkable e-efficiency of our approach. In fact, our method demonstrates at least 2 times greater e-sampling efficiency compared to exhaustively training safe RL agents using C- G-PPPL-La.
Table 2: e-sampling efficiency evaluation. t: the higher reward, the better. 4: the lower constraint violation (minimal 0), the better. The models are evaluated on a series of threshold conditions and we report the averaged reward and constraint violation values on all evaluation thresholds and generalized thresholds. Each value is reported as + standard deviation for 50 episodes and 5 seeds. We shade the safest agent with the lowest averaged cost violation value.
Algorithm
Stats
Ball-Circle
Car-Circle
Drone-Circle
Drone-Ruse Run
Averaged Score
CCPO with E
Avg. R1
710.86:20.47
406.06#6.30
630.55=40.03
458 12.98
551.54
Avg. CV 4
0.59+0.31
1.60#0.91
0.32+0.38
0.23+0.25
0.69
C-TRPO
Avg. R
699.38+1.99
461. 1,33
18..77
461.70+4.91
500.75
with E
Avg. CV
4.50+0.08
7.84
6.55+1.95
47.9 47.97+3.49
16.72
C-TRPO
Avg. R
682.9448.00
158.1342.22
411 111.9148.95
472.8942.65
506.47
with E'
Avg. CV
2.66+0.37
11.90+2.12
5.20+0.81
30,20 477
12,49
5.3 Ablation Study
To study the influence of VVE, and CVI components CCPO introduced in the section 4.1 and 4.2 we conduct an ablation study by removing each component from the full CCPO algorithm. The ablation experiment results are shown in Table. We shade the safest agent with the lowest averaged cost violation value. We can also observe significant safety performance (cost violation) degradation and task performance (reward) drop if we remove the VVE module (versatile critic learning) or CVI module (versatile actor learning) since they help us learn versatile Q functions more accurately and

---

improve the generalizability of learned actors. Removing them will result in the bad estimation of for state-action pair value function and lead to poor policy generalization capability.
Table 3: Ablation study of removing the versatile value function estimation (VVE), and the conditioned
variational inference (CVT). f: the higher reward, the better. 4: the lower constraint violation (minimal 0), the hetter. Each value is reported as mean + standard deviation for 50 episodes and seeds. Fach value is reported as mean standard deviation.
Algorithm
Stats
Ball-Circle
Car-Circle
Drone-Cirele
Drone-Run
Ant-Run
Avg. R
710.86+20.47
406.06+6.30
630.55+40.03
458.69+12.98
660.88s4.82
Avg. CV
0.5910.31
1.6010.91
0.32=0.38
0,2340.25
3.1321.67
CCPO (Full)
Avg. R-G f
699 04420.48
401 .5345.99
625.51+40.12
455.64+11.83
660.07+5.26
Avg. CV-G 4
0.83+0.42
1.4920.38
0.47:0.55
0.33+0.37
3.25+1.48
Avg.
674.55+17.81
370.42+14.38
426.47=49.30
417.84x8.24
428.59488, 39
Avg. CV
0.6010.41
6,4220.85
8.67=1.45
3.28&2.86
10.66211.81
CCPO w/o VVE
Avg. R-G f
670.61+14.18
364.5e14.51
416.83e47,46
413.28+9.04
434.59+83.89
Avg. CV-G
0.73a0.36
5,6420.92
7.74s1.36
3.3343.16
12.01
Avg. R
6411.33+4014
387.31
520.70s42.18
386.8 1 44
165.80&31.78
Avg. CV
1.44x0.72
1.6620.79
2.3622.67
0,8120.76
3.51
CCPO w/o CVI
Avg. R-G f
623.17241.42
383.2426.30
519.05236.31
388.69235.
465.36232.2
Avg. CV-G 4
1.78+0.70
2.17+1.09
2.73+3.03
1.15+1.08
3.96+1.01
Conclusion
In this study, we pioneered the concept of versatile safe reinforcement learning (RL), presenting the CunstraittCoonditonne Policy Optimization (CCPO) algorithm. This approach adapts efficiently to different and unseen cost thresholds, offering promising solution to safe RL beyond pre-defined constraint thresholds. With its core components, Versatile Value Estimation (VVE) and Conditioned Variational Inference (CVT), CCPO facilitates zero-shot generalization for constraint thresholds. Our theoretical analysis further offers insights into the constraint violation bounds for unseen thresholds and the sampling efficiency of the employed behavior policies. The extensive experimental results reconfirm that CCPO effectively adapts unseen threshold conditions and is much safer and more data-efficient than baseline methods. The limitations include that our method would be more computationally expensive than the primal-dual-bart based safe RL. approaches due the convex optimization problems in the constraint-conditione E-step. One potential negative social impact is that the misuse of this work in safety-critical scenarios can cause unexpected damage. We hope our findings can inspire more research in studying the generalization capability in safe RL.
7 Acknowledgment
The work is partially supported by Google Deepmind with an unrestricted grant. The authors also want acknowledge the support from the National Science Foundation under grants CNS-204745A
References
[1] Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang. and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 10330, 2022.
[2] David Isele, Alireza Nakhaei. and Kikuo Fujimura. Safe reinforcement learning on autonomous vehicles. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1-6. IEEE, 2018.
[3] Hongyuan Ding, Yan Xu, Benjamin Chew Si Hao, Qiaoqiao Li, and Antonis Lentzakis. A safe reinforcement learning approach for multi-energy management of smart home. Electric Power Systems Research, 210:108122 120, 2022.
10

---

[4] Zengyi Qin, Kaiqing Zhang, Yuxiao Chen, Jingkai Chen, and Chuchu Fan. Learning safe multi- agent control with decentralized neural barrier certificates. arXiv preprint arXiv:2101.05436 2021.
[5] Tsung-Yen Yang, Tingnan Zhang. Linda Luu, Sehoon Ha, Jie Tan, and Wenhao Yu. Safe reinforcement learning for legged locomotion, arXiv preprint arXiv:2203. 0.63800 2022.
[6] Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang. Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6(3):4915-492 2021.
[7] Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety crities for exploration. arXiv preprint arXiv:2010.14497, 2020.
[8] Vanshaj Khattar, Yuhao Ding, Bilgehan Sel, Javad Lavaei, and Ming Jin. cmdp-within-online framework for meta-safe reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
[9] Yuhao Ding and Javad Lavaei. Provably efficient primal-dual reinforcement learning for emdps with non-stationary objectives and constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7396-74660 04066,7744247 2023.
[10] Honghao Wei, Arnob Ghosh, Ness Shroff, Lei Ying, and Xingyu Zhou. Provably efficient model-free algorithms for non-stationary cmdps. In International Conference on Artificial Intelligence and Statistics, pages 6527-6570. PMLR, 2023.
[11] Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou. Jacopo Panerati, and Pchooelig Safe learning in robotics: From learning- based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5, 2021.
[12] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep rein- forcement learning. arXiv preprint arXiv:1910001708, 7, 2019.
[13] Zuxin Liu, Zijian Guo. Zhepeng Cen, Huan Zhang, Jie Tan, Bo Li, and Ding Zhao. On the robustness of safe reinforcement learning under observational perturbations. arXiv preprint 1rX1v 2022.
[14] Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. arXiv preprini arXiv:2302.07351, 2023.
[15] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning, pages 22-31. PMLR, 2017.
[16] André Barreto, Will Dabney, Rémi Munos, Jonathan Hunt, Tom Schaul, Hado van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
[17] Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pages 501-510. PMLR, 2018.
[18] Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.00 08757, 2018.
[19] William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention. arXiv preprint arXiv:1707.05173, 2017.
[20] Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. Thirty-Second AAAI Conference on Artificial Intelligence, 2018.

---

[21] Haonan Yu, Wei Xu, and Haichao Zhang. Towards safe reinforcement learning with a safety editor policy, arXiv preprint arXiv:2201.12427, 2022.
[22] Zuxin Liu, Hongyi Zhou, Baiming Chen, Sicheng Zhong, Martial Hebert, and Ding Zhao Safe model-based reinforcement learning with robust cross-entropy method. arXiv preprini arXiv:2010.07968, 2020.
[23] Yuping Luo and Tengyu Ma. Learning barrier certificates: Towards safe reinforcement learning with zero training-time violations. Advances Neural Information Processing Systems, 34, 2021.
[24] Aivar Sootla, Alexander Cowen- Taher Jafferjee, Ziyan Wang, David Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning, pages 10423-20443 PMLR, 2022.
[25] Baiming Chen, Zuxin Liu, Jiacheng Zhu, Mengdi Xu, Wenhao Ding, Liang Li, and Ding Zhao Context-aware safe reinforcement learning for non-stationary environments. In 202 IEEE International Conference Robotics and Automation (ICRA), pages 10689-10695. IEEE, 2021.
[26] Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Yihang Yao, Hanjiang Hu, and Ding Zhao. Towards robust and safe reinforcement learning with benign off- off-po data. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 21586-21610, PMLR, 23-29 Jul 2023.
[27] Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
(28) Yannis Flet-Berliac and Debabrota Basu. Saac: Safe reinforcement learning as an adversarial game of actor-critics. arXiv preprini arXiv:2204.09424, 2022.
[29] Shalabh Bhatnagar and K Lakshmanan. An online actor-critic algorithm with function ap- proximation for constrained markov decision processes. Journal of Optimization Theory and Applications, 153(3):688-700) 2012.
[30] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk ciss-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070-6 6120, 2017.
[31] Yarden As, Inura Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy opti- mization via bayesian world models. arXiv preprint arXiv:2201.09800 2022.
[32] Qingkai Liang, Fanyu Que, and Eytan Modiano. Accelerated primal-dual policy optimization for safe reinforcement learning. arXiv preprint 0tti::/802.06440 2018.
[33] Chen Tessler, Daniel Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint 1 1004, 2018
[34] Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference Machine Learning, pages 13644-13668. PMLR, 2022.
[35] Ming Yu. Zhuoran Yang, Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning. arXiv preprint arXiv: /2/56, 2019.
[36] Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. arXiv preprint arXiv:2010.03/55 2020.
[37] Shangding Gu, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang, Zheng Tian, Jun Wang, Alois Knoll, and Yaodong Yang. Multi-agent constrained policy optimisation. arXiv preprint 21Xiv:2110.02793, 2021.
12

---

[38] Dohyeong Kim and Songhwai Oh. Efficient off off-policy safe reinforcement learning using trust region conditional value at risk. IEEE Robotics and Automation Letters, 7(3)7644-7651, 2022.
[39] Zhuangdi Zhu, Kaixiang Lin, Anil K Jain, and Jiayu Zhou. Transfer learning in deep reinforce- ment learning: A survey. arXiv preprint arXiv:2009.00 200008800,, 2020.
[40] Pashootan Vaezipoor, Andrew C Li, Rodrigo A Toro Icarte, and Sheila A Mcilraith. Ltl2action: Generalizing Itl instructions for multi-task, rl. In International Conference Machine Learning, pages 10497-10508. PMLR, 2021.
[41] Brandon Araki, Xiao Li, Kiran Vodrahalli, Jonathan DeCastro, Micah Fry, and Daniela Rus. The logical options framework. In International Conference on Machine Learning, pages 307-317. PMLR, 2021.
[42] Emanuel Todorov. Compositionality of optimal control laws. Advances in neural information processing systems, 22, 2009.
[43] Geraud Nangue Tasse, Steven James, and Benjamin Rosman. boolean task algebra for reinforcement learning. Advances in Neural Information Processing Systems, 33:9497-9507, 2020.
[44] Janarthanan Rajendran, Aravind Srinivas, Mitesh M Khapra, P Prasanna, and Balaraman Ravindran. Attend, adapt and transfer: Attentive deep architecture for adaptive transfer from multiple sources in the same domain. arXiv preprint arXiv:1510.02879, 2015.
[45] Chen Ma, Dylan R Ashley, Junfeng Wen, and Yoshua Bengio. Universal successor features for transfer reinforcement learning. arXiv preprint arXiv:2001.04022 2020.
[46] Jackyeom Kim, Seohong Park, and Gunhee Kim. Constrained gpi for zero-shot transfer in reinforcement learning. Advances in Neural Information Processing Systems, 155:485-4599) 2022.
[47] Mark Nemecek and Ronald Parr. Policy caches with successor features. International Conference Machine Learning, pages 8025-8033. PMLR, 2021.
[48] Peter Dayan. Improving generalization for temporal difference learning: The successor repre- sentation. Neural computation, 5(4):613-624, 1993.
[49] Ida Momennejad, Evan M Russek, Jin H Cheong, Matthew M Botvinick, Nathaniel Douglass Daw, and Samuel J Gershman. The successor representation in human reinforcement learning. Nature human behaviour, 19)):680-692, 2017.
[50] Eitan Altman. Constrained Markov decision processes: stochastic modeling. Routledge, 1999.
[51] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798- 1828, 2013.
[52] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv: /rXiv:/805.00009, 2018
[53] Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Yihang Yao, Hanjiang Hu, and Ding Zhao. Towards robust and safe reinforcement learning with benign off-policy data. 2023.
[54] Abbas Abdolmaleki, Jost Tobias Springenherg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint 2018.
[55] Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Heess, and Martin Riedmiller. Relative entropy regularized policy iteration. arXiv preprint 1rXiv:18/2.02255. 2018.
[56] Sven Gronauer. Bullet-safety- gym: Aframework for constrained reinforcement learning. 2022.
13

---

[57] Tianhao Wei and Changliu Liu. Safe control algorithms using energy functions: A uni ed framework, benchmark, and new directions. In 2019 IEEE 58th Conference on Decision and Control (CDC), pages 238-243. IEFE, 2019.
[58] Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.0930) 2023.
[59] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (75.009977 2015.
[60] Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. Advances in Neural Information Processing Systems, 30, 2017.
[61] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, al. Con- ditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
[62] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI Conference Artificial Intellignnce volume 32, 2018.
[63] Aviral Kumar, Xue Bin Peng, and Sergey Levine. Rewarr--conitionee policies. arXiv preprint 19X1v:191..13465, 2019.
14


Reviews:
[{"review_id": "zFoPp9cxCK", "reviewer": "Reviewer_R5zk", "summary": "The study investigated a versatile, safe reinforcement learning problem and proposed the Conditioned Constrained Policy Optimization (CCPO) algorithm. CCPO may be helpful in ensuring safety under the settings of various constraint limits. Moreover, the experiment results demonstrate that CCPO works well.", "questions": "1. How can we handle non-convex optimization while ensuring safety?\n2. If the cost limits are linear, what will happen?\n3. Could you compare it with CPO-based methods?\n", "limitations": "1. The balance between reward and cost is not handled well.\n2. Although the Lagrangian-based methods can not perform well on various cost-limit settings, their reward performance is better than CCPO.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The problem is interesting.\n2. The method may be useful for safe RL research.\n", "weaknesses": "1. The code the study provided is not convenient to run; we have to download the code file by file. Could you provide the whole code? Such that we can examine the algorithm better.\n2. Please check the whole paper, and improve the writing quality, e.g., on page 3, line 101, please check the sentence, and revise “z(1)” as “(1)”.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time and valuable feedback. We address the concerns as follows.\n\n> W1: The code the study provided is not convenient to run; we have to download the code file by file. Could you provide the whole code? Such that we can examine the algorithm better.\n\nWe apologize if the code of the current version brings any inconvenience. We have submitted a new version to AC based on the review guideline.\n\n\n> W2: Please check the whole paper, and improve the writing quality, e.g., on page 3, line 101, please check the sentence, and revise “z(1)” as “(1)”.\n\nWe sincerely appreciate your detailed review of our work. We will correct the typos and further improve the writing quality in the revision.\n\n> Q1: How can we handle non-convex optimization while ensuring safety?\n\nThank you for your question. We decompose the general non-convex safe RL problems into two steps (E-M steps), where the first step handling safety constraints is a convex optimization problem. Here are the details: In the Constraint-Conditioned E-step, where safety is encoded, the policy optimization process is formulated as Eq. (8). We calculate the optimal variational distribution, denoted as $q(\\cdot | \\epsilon)$, by solving its dual form (10). Subsequently, we can obtain the closed-form solution for Eq. (8). In Appendix B.3, we demonstrate that the dual problem (10) is convex, and under certain mild conditions, we can ensure its strong convexity. Thus, various off-the-shelf convex optimization solvers can be utilized to find the optimal solution.\n\n> Q2: If the cost limits are linear, what will happen?\n\nThank you for your question. In our experiments, all the behavior policy cost conditions exhibit a linear relationship with respect to the behavior policy index, indicating that the condition set forms an arithmetic progression. \n\nIn terms of our Critics linear decomposition (starting from line 132): the linearity is describing the operational relationship between $\\psi(s, a)$ and $z(\\epsilon)$, while $\\psi(s, a)$ and $z(\\epsilon)$ are both nonlinear functions. We also provide some verification results for Q functions decomposition/estimation in Appendix D.1.\n\nWe welcome any additional clarification or comments you may have on this problem.\n\n> Q3: Could you compare it with CPO-based methods?\n\nThank you for your valuable feedback. We have made modifications to CPO, creating the Versatile CPO (via policy linear combination), and conducted experiments in all five tasks. The following table presents the comparison results between Versatile CPO and our proposed CCPO. It is evident that the proposed CCPO method outperforms Versatile CPO in terms of both constraint satisfaction and reward efficiency.\n\n| Tasks | stats | BC | CC | DC | DR | AR | Average |\n| :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | \n|  Versatile CPO | Avg. R $\\uparrow$ | 686.08 ± 2.58 | 455.87 ± 1.51 | 565.98 ± 5.71 | 304.09 ± 32.17 | 602.05 ± 1.66 | 522.81\n|  | Avg. CV $\\downarrow$ | 1.77 ± 0.67 | 4.50 ± 2.08 | 6.32 ± 2.32 | 11.82 ± 5.85 | 2.19 ± 0.63 | 6.60\n|  | Avg. R-G $\\uparrow$ | 676.27 ± 2.32 | 454.62 ± 1.97 | 559.71 ± 7.32 | 300.85 ± 32.41 | 592.89 ± 2.68 | 516.87\n|  | Avg. CV-G $\\downarrow$ | 2.09 ± 0.59 | 4.94 ± 2.37 | 6.75 ± 2.47 | 13.07 ± 5.44 | 2.28 ± 0.73 | 5.83\n| CCPO | Avg. R $\\uparrow$ | 710.86 ± 20.47 | 406.06 ± 6.30 | 630.55 ± 40.03 | 458.69 ± 12.98 | 660.88 ± 4.82 | **573.41**\n|  | Avg. CV $\\downarrow$ | 0.59 ± 0.31 | 1.60 ± 0.91 | 0.32 ± 0.38 | 0.23 ± 0.25 | 3.13 ± 1.67 | **1.17**\n|  | Avg. R-G $\\uparrow$ | 699.04 ± 20.48 | 401.53 ± 5.59 | 625.51 ± 40.12 | 455.64 ± 11.83 | 660.07 ± 5.26 | **568.36**\n|  | Avg. CV-G $\\downarrow$ | 0.83 ± 0.42 | 1.49 ± 0.38 | 0.47 ± 0.55 | 0.33 ± 0.37 | 3.25 ± 1.48 | **1.27**\n\n> Limitation: The balance between reward and cost is not handled well; Although the Lagrangian-based methods can not perform well on various cost-limit settings, their reward performance is better than CCPO.\n\nThank you for raising this point. It is true that the rewards of baselines in some tasks are slightly higher than our method's. However, we would like to gently point out that this is a common trade-off between task utility and safety performance in the safe RL literature -- the **optimal** policy under safety constraints might not be the most rewarding one[R-1, R-2, R-3]. \nTo be more concrete, the evaluation criteria in safe RL are:\n1) method A is better than method B if A achieves better safety performance than B.\n2) If both A and B satisfy constraints, the one with the higher reward is better.\n\nWith the above metrics, we can conclude that CCPO outperforms baselines in most experiments. We will make the evaluation criteria clear in the revision to avoid confusion.\n\n**Reference** \n\n[R-1] Yu, Haonan, Wei Xu, and Haichao Zhang. \"Towards safe reinforcement learning with a safety editor policy.\" Advances in Neural Information Processing Systems 35 (2022): 2608-2621.\n\n[R-2] Stooke, Adam, Joshua Achiam, and Pieter Abbeel. \"Responsive safety in reinforcement learning by pid lagrangian methods.\" International Conference on Machine Learning. PMLR, 2020.\n\n[R-3] Liu, Zuxin, et al. \"On the Robustness of Safe Reinforcement Learning under Observational Perturbations.\" The Eleventh International Conference on Learning Representations. 2022."}, {"review_id": "FppSwsu6UE", "reviewer": "Reviewer_hfso", "summary": "The authors introduce the versatile safe RL problem, in which we want to maximize the expected reward, subject to an expected constraint, but for a range of constraint thresholds $\\epsilon\\in \\mathcal E$ (compared to the traditional safe RL problem, which only aims to find a solution for one $\\epsilon$). This is an interesting problem setting, with good motivation and potential real world application in learning a policy that works under many circumstances. The authors also provide an algorithm CVPO,  that nicely combines the control as inference framework with the EM framework to improve the likelihood of optimality while trying to stay within a feasible set.", "questions": "\nOn Page 4, Line 134 - what is $K_f$? is it a constant for $f \\in \\{r, c\\}$? Similarly, on line 146, I understand that $Poly(\\epsilon, p)$ is a $p$-degree polynomial of $\\epsilon$, but shouldn't it have a vector output since $z_f^*$ is a vector? Also, on line 158, why is it okay to normalize epsilon to [0, 1] since realistically constraint thresholds can be arbitrary?\n\nOn Page 5, Line 197, the authors say - \"The key strength of using the variational inference framework lies in its ability to encode arbitrary threshold conditions during policy learning, as shown in (7), a feat that is challenging for other methods, such as those based on primal-dual algorithms.\" Could you elaborate why this is challenging for other methods?\n\nFor testing generalization to unseen thresholds, the authors train on data from $\\epsilon$=20,40,60 and test with $\\epsilon$=10,20,30,40,50,60,70. This is relatively close to the thresholds in the training data. It would be nice to see this algorithm trained with thresholds in [20, 30, 40] and tested with thresholds in [50, 60, 70]. Ideally, the algorithm may be able to provide feasible policies, but it is also important to test if the reward achieved is highest possible. If it is able to handle thresholds well outside the training set, that would be a good thing and it will show that the algorithm is quite versatile. If it is unable to do so, then, it should be discussed a bit further i.e. what unseen thresholds can we realistically expect good performance on?\n\nSuggestions related to language and typos:\n1. On Page 5 - \"We utilize the safe RL as inference framework ...\" - since this is proposed by this work, it would be better to say \"we introduce the ...\".\n\nEDIT (16 Aug 2023): Updated score from 6->7.", "limitations": " ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": " ", "weaknesses": " ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time and valuable feedback. We address the concerns as follows.\n\n> Q1: On Page 4, Line 134 - what is $K_f$ is it a constant for $f \\in r, c$ Similarly, on line 146, I understand that $Poly(\\epsilon, p)$ is a $p$-degree polynomial of $\\epsilon$, but shouldn't it have a vector output since $z_f$ is a vector? Also, on line 158, why is it okay to normalize $\\epsilon$ to [0, 1] since realistically constraint thresholds can be arbitrary? \n\nThank you for your questions. Yes, $K_f$ is a constant for $f \\in r, c$, and each element of $z_f$ is a $p$-degree polynomial of $\\epsilon$. In this work, we consider finite-step trajectories and bounded costs for each step. Consequently, we have a maximum cost value for each trajectory. To simplify the process, we can normalize $\\epsilon$ by dividing it by the upper bound for trajectory cost. All the experiment tasks, as well as many common safe RL benchmarks such as safety-gym [R-1], comply with these assumptions (finite-step, bounded cost value). We apologize if the formulation has caused any confusion, and we will further refine it in the revision.\n\n> Q2: On Page 5, Line 197, the authors say - \"The key strength of using the variational inference framework lies in its ability to encode arbitrary threshold conditions during policy learning, as shown in (7), a feat that is challenging for other methods, such as those based on primal-dual algorithms.\" Could you elaborate why this is challenging for other methods?\n\nThank you for your insightful question. Taking Lagrangian-based safe RL algorithms as an example, the key component of such algorithms is to find a proper $\\lambda$ balancing the trade-off between reward and cost satisfaction, i.e., $\\lambda$ for $V_r - \\lambda V_c$ in the policy loss. The multiplier $\\lambda$ is a function of the policy $\\pi$. One of the most common approaches is using the PID method [R-2] to update $\\lambda$ with the cost difference $(\\Sigma c_t-\\epsilon)$. Such methods present a major challenge for versatile safe RL: **Lagrangian multiplier update for unseen thresholds.** Let $\\epsilon_1$ represent one behavior policy condition, and $\\epsilon_2 < \\epsilon_1$ represents another unseen threshold for adaptation. Let $\\lambda_1$ and $\\lambda_2$ be the corresponding Lagrangian multipliers. Suppose we update $\\lambda_2$ using trajectory data from the behavior policy $\\pi(\\cdot | \\epsilon_1)$; since $\\Sigma c_t$ converges to $\\epsilon_1$, then $\\Sigma c_t - \\epsilon_2$ will never converge to $0$, which makes $\\lambda_2$ overestimated. This could make the policy $\\pi(\\cdot|\\epsilon_2)$ overly conservative.\n\nWith the CCPO method, we do not encounter such problems since we encode the threshold condition in the constraint-conditioned E-step by setting constraints in an off-policy fashion while calculating the optimal variational distribution $q^*(\\cdot | \\epsilon)$. This constraint is flexible for arbitrary $\\epsilon$ as long as we have obtained a good estimation of the Q-function $Q_c(\\cdot | \\epsilon)$, which is accomplished in the **VVE** module of this work.\n\n> Q3: For testing generalization to unseen thresholds...\n\nThank you for your valuable feedback. The experiment results for CCPO with behavior policy condition $[20, 30, 40]$ and evaluation on $[10, 15, ..., 70]$ are shown in Figure R-1 and the following table. Each value is reported as mean ± standard deviation for 50 episodes and 5 seeds. \n\n|  Tasks | BC | CC | DC | \n|  :------: | :------: | :------: | :------: | \n| Avg. R $\\uparrow$ | 542.71 ± 9.63 | 369.76 ± 2.92 | 559.11 ± 46.05 | \n| Avg. CV $\\downarrow$ | 0 ± 0 | 0.68 ± 0.46 | 0.57 ± 0.22 | \n| Avg. R-G $\\uparrow$ | 545.50 ± 11.99 | 369.14 ± 2.16 | 545.85 ± 54.97 | \n| Avg. CV-G $\\downarrow$ | 0 ± 0 | 0.78 ± 0.51 | 0.75 ± 0.28 | \n\nFrom the experiments, we can observe the following:\n\n**(1) The performance around the behavior policy conditions $[20, 30, 40]$ is stable and good in terms of both reward and constraint satisfaction.** Within the threshold condition range from $10$ to $50$, the versatile agent can adapt to unseen thresholds with reasonable task reward and near-zero cost violation.\n\n**(2) The selection of behavior policy conditions is crucial for the performance over the target condition interval.** The rewards concerning threshold conditions from $50$ to $70$ do not achieve the highest possible value. This sub-optimality may arise from challenges in estimating $z(\\epsilon)$ in $Q(s,a | \\epsilon) = \\psi(s, a)^T z(\\epsilon)$, when $\\epsilon$ is too far away from the behavior policy conditions. In Theorem $1$, the bounded estimation error is derived based on the assumption that behavior policy conditions divide the target threshold condition interval equally. How to address the proposed problem is an interesting question and can be a significant area for future work. For example, we may introduce some additional regularizer into the $z(\\epsilon)$ model to resolve this.\n\nWe will add related discussions in the revision.\n\n\n**Reference:**\n\n[R-1] A. Ray, J. Achiam, and D. Amodei, “Benchmarking Safe Exploration in Deep Reinforcement Learning,” 2019.\n\n[R-2] Stooke, Adam, Joshua Achiam, and Pieter Abbeel. \"Responsive safety in reinforcement learning by pid lagrangian methods.\" International Conference on Machine Learning. PMLR, 2020."}, {"review_id": "NmUXYY2W2D", "reviewer": "Reviewer_5QUs", "summary": "Safe reinforcement learning approaches have so far focused on the case where\ncost thresholds are static and known a priori. In contrast, this work addresses\nRL environments where cost constraints may change over time. To do this, the\nproposed algorithm learns a policy which is conditioned on the given cost\nthreshold, allowing the policy to adapt to new cost thresholds at test time.\nThe policy is derived from a pair of learned Q functions, one for rewards and\none for costs. The Q functions are decomposed into a feature for state-action\npairs and a feature for the safety threshold. The safety threshold feature uses\na relatively low-dimensional model in order to provide better generalization\nwithout needing to see a large set of thresholds at training time, making\ntraining more efficient. In experiments, the proposed algorithm outperforms\nLagrangian approaches to safety in a suite of relatively high-dimensional\nbenchmarks.", "questions": "Do you have any intiution about why TRPO-Lag achieves better safety on Ant-Run\nthan CCPO? Is there some feature of the benchmark or the algorithm which makes\nthis a particularly challening case for CCPO?", "limitations": "The authors have adequately addressed some limitations and potential societal\nimpacts in the paper.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The problem of versatlie safe RL is interesting, potentially important, and (to\nmy knowledge) novel. The threshold in CMDP-based safe RL work is often\narbitrary, so it seems useful to be able to modify that threshold without\nretraining the system. Moreover, there may be some environments where the cost\nthreshold changes over time.\n\nThe proposed high-level approach is a very elegant solution to this problem. I\nparticularly like the idea of separating out the effect of $\\epsilon$ into a\nlower-parameter model which should generalize better than a full neural model.\nThe way this allows for efficient training without sacrificing generalizability\nto new safety thresholds is quite impressive. Similarly, separating out Q-value\nlearning from policy construction is an interesting way to generalize while\nmaintaining the power of neural learning.\n\nThe experiments are fairly extensive and show impressive results. In\nparticular, CPPO demonstrates much better safety than other approaches on most\nbenchmarks, while maintaining comparable performance. I think Figure 2 is a\ngood illustration of the advantages of CPPO, showing how the number of observed\ncost closely tracks the allowed cost. This suggests that CPPO is using all of\nits available safety budget to achieve high rewards.", "weaknesses": "I think the problem statement could do with a bit more motivation. While I do\nbelieve the versatile safe RL problem is interesting, it's not clear to me how\noften these kinds of problems actually arise. The given example of a car\noperating in highway vs. urban environments provides some intuition, but I'm\nnot clear on what the cost signal is that would be changing in that case. An\nextra example or more explanation might help clarify this.\n\nI found the development in Section 4 to be difficult to follow. I'm not very\nfamiliar with the work on RL as inference, so maybe this explanation is\nsufficient for more expert readers. For less expert readers, I think it would\nhelp to add a bit more intuition or maybe some examples if feasible.\n\nFor the experiments, I think it would help to include some baselines from the\nsafe RL literature beyond basic Lagrangian approaches. For example, you might\ncompare with CPO or Conservative Safety Critics (among others). These\napproaches are generally tailored to a single threshold, but a comparison could\nbe made by training them with a strict threshold then using a more relaxed\nthreshold at test time. In principle, CPPO should achieve a higher reward since\nit is able to take advantage of the increased safety budget.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We gratefully thank the reviewer for your time, valuable feedback, and acknowledgment of our contribution. We address your concerns as follows.\n\n> W1: The given example of a car operating in highway vs. urban environments provides some intuition, but I'm not clear on what the cost signal is that would be changing in that case. An extra example or more explanation might help clarify this.\n\nWe sincerely appreciate your interest in our proposed versatile safe RL settings. Here is an extra example: for a legged delivery robot, we can define the reward as the walking speed and the cost as the shaking amplitude. Intuitively, stricter constraints on cost typically lead to more conservative behavior (low-speed motion) and lower task rewards. In this case, the robot can adapt to a larger cost threshold for carrying non-fragile than carrying fragile cargo to maximize working efficiency.\n\n> W2: ... I think it would help to add a bit more intuition or maybe some examples in Section 4 if feasible.\n\nThank you for your great suggestion. We fully agree that it would be desirable to provide some additional explanations. Here, we present a toy-like example for **Section 4.2. Conditional Variational Inference**. Consider the task illustrated in Figure R-2, where the action space is discrete and has a dimension of 2 (representing the acceleration and angular velocity of the unicycle). The key idea of this section is to first calculate the optimal distribution $q^*(a | s, \\epsilon_i)$ for a state $s$ and sampled actions $a$ in the **Constraint-Conditioned E-step**, and subsequently optimize the policy $\\pi(\\cdot|\\epsilon_i)$ based on $q^*(a | s, \\epsilon_i)$ using supervised learning in the **Versatile M-step**.\n\n\nAs shown in the Figure. R-3, for a given state $s$, we sample all possible actions and calculate the corresponding optimal distribution $q^*$ within the feasible distribution $q$ set. The optimization variable $q$ represents the probabilities of taking each sampled action. In the **Constraint-Conditioned E-step**, when dealing with different thresholds $\\epsilon_1 = 10$ and $\\epsilon_2 = 20$, the feasible $q$ set calculated based on $Q_c(\\cdot| \\epsilon_i)$ may differ, as illustrated in Figure R-3 $(a)$ and $(c)$. Consequently, the optimal distribution $q(\\cdot|\\epsilon_i)$ will also vary - in the looser condition $\\epsilon_2=20$, the agent can take more aggressive actions to achieve higher rewards, as illustrated in Figure R-3 $(b)$ and $(d)$. After obtaining the optimal distribution $q^*(\\cdot|\\epsilon_i)$, in the **Versatile M-step**, we can improve the Evidence Lower Bound (ELBO) of $\\pi(\\cdot|\\epsilon_i)$ with respect to $q^*(\\cdot|\\epsilon_i)$ for policy optimization, as shown in Eq. (12) in the main text.\n\n\n\nWe welcome any additional comments you may have on this example. Thank you for your feedback!\n\n> W3: More baselines...\n\nThank you for your valuable feedback. We have made modifications to CPO (via policy linear combination) creating the Versatile CPO, and conducted experiments in all five tasks. The following table presents the comparison results between Versatile CPO and our proposed CCPO. It is evident that the proposed CCPO method outperforms Versatile CPO in terms of both constraint satisfaction and reward efficiency.\n\n| Tasks | stats | BC | CC | DC | DR | AR | Average |\n| :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | \n|  Versatile CPO | Avg. R $\\uparrow$ | 686.08 ± 2.58 | 455.87 ± 1.51 | 565.98 ± 5.71 | 304.09 ± 32.17 | 602.05 ± 1.66 | 522.81\n|  | Avg. CV $\\downarrow$ | 1.77 ± 0.67 | 4.50 ± 2.08 | 6.32 ± 2.32 | 11.82 ± 5.85 | 2.19 ± 0.63 | 6.60\n|  | Avg. R-G $\\uparrow$ | 676.27 ± 2.32 | 454.62 ± 1.97 | 559.71 ± 7.32 | 300.85 ± 32.41 | 592.89 ± 2.68 | 516.87\n|  | Avg. CV-G $\\downarrow$ | 2.09 ± 0.59 | 4.94 ± 2.37 | 6.75 ± 2.47 | 13.07 ± 5.44 | 2.28 ± 0.73 | 5.83\n| CCPO | Avg. R $\\uparrow$ | 710.86 ± 20.47 | 406.06 ± 6.30 | 630.55 ± 40.03 | 458.69 ± 12.98 | 660.88 ± 4.82 | **573.41**\n|  | Avg. CV $\\downarrow$ | 0.59 ± 0.31 | 1.60 ± 0.91 | 0.32 ± 0.38 | 0.23 ± 0.25 | 3.13 ± 1.67 | **1.17**\n|  | Avg. R-G $\\uparrow$ | 699.04 ± 20.48 | 401.53 ± 5.59 | 625.51 ± 40.12 | 455.64 ± 11.83 | 660.07 ± 5.26 | **568.36**\n|  | Avg. CV-G $\\downarrow$ | 0.83 ± 0.42 | 1.49 ± 0.38 | 0.47 ± 0.55 | 0.33 ± 0.37 | 3.25 ± 1.48 | **1.27**\n\n> Q1: Intuition about why TRPO-Lag achieves better safety on Ant-Run than CCPO.\n\nThank you for your valuable feedback. Let's briefly introduce the Ant-Run task and analyze the results. In Ant-Run, the reward is defined by the distance moved along one direction, and the cost signal is sparse (i.e., 0-1 cost), where the agent incurs a cost once it exceeds the speed limit or goes beyond the boundary. The dynamics of the Ant are complicated, with the observation space dimension being $33$, containing the ego position, joint angles, etc. The action space dimension is $8$, encompassing the torque of each joint. \n\n\nConsidering the TRPO-Lag baseline, we observe in Figure 2 of the main text that there are significant reward performance drops at some unseen thresholds (e.g., 10, 15, 25, 30, 35, 65, ...). These drops indicate that under these conditions, the walking ability of Ant is poor. Consequently, the risk of the Ant going beyond the speed limit is also low, which explains why TRPO-Lag achieves higher safety performance in Table 1. On the other hand, our proposed CCPO algorithm exhibits stable and smooth reward performance with respect to different threshold conditions. Moreover, the cost returns in CCPO consistently adhere to the target thresholds on nearly every unseen threshold with low coefficient of variation (CV) and variance, showcasing CCPO's more reliable and stable performance."}, {"review_id": "r7EyJ1aI9N", "reviewer": "Reviewer_TpqY", "summary": "The paper addresses Safe Reinforcement Learning with varying safety constraint requirements (Versatile Safe Reinforcement Learning). The paper proposes to learn a versatile Q function, which is decomposed as the product of feature functions of state and action,  and the latent features of the constraint thresholds. The Q function will be used in a variational EM algorithm to find the constraint-satisfying policy that maximizes the return. The proposed method shows good safety and decent returns in experimental robotic tasks such as Run and Circle, outperforming simple constraint-conditioned and linear combination baselines.", "questions": "See Weakness.", "limitations": "The method is complicated with different stages of optimization. Can it work in higher dimensional domains such as visual state space?", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The target problem is interesting and seems novel in the setting of Safe RL\n- The paper has relevant theoretical analysis to back up the method", "weaknesses": "- The writing is unclear, especially in the pretraining phase. It is better to have an algorithm in the main text to summarize the process and connect the components in the method. It is hard to connect Fig. 1 to the method section. For example, where are the two bottom boxes in the pretraining phase described in the method section?\n- The experiment result overall is not really good. Although the safety criteria show good performance, the reward counterpart is not. In many tasks, the proposed method underperforms the baselines regarding the rewards. How to ensure that the trade-off is acceptable and the behaviour is desirable in Safe RL setting? Overall, more tasks should be tested to confirm if the lower performance is a consistent issue in the approach. Furthermore, there are claims on sample efficiency, yet no report on it can be found in the main text. It should be noted that running time should be considered as well because as admitted in the discussion, the method can be slow due to complicated optimization steps.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time and valuable feedback. We address the concerns as follows.\n\n> W1: The writing is unclear, especially in the pretraining phase...\n\nThank you for your valuable suggestions. While we placed the detailed algorithm diagram in the appendix due to page constraints, we wholeheartedly concur that having an algorithm diagram in the main text for summarizing the process and interconnecting method components is advantageous. To cater to this, we offer a concise algorithm diagram in the attached PDF. In this diagram, lines 1-2 correspond to the VVE module, while lines 3-15 correspond to the CVI procedure. Specifically, during the pre-training phase, the selection of threshold conditions for training is denoted as the behavior policy condition set (line 5). In the fine-tuning stage, the choice shifts to a larger condition set (line 5). In Figure 1, \"Versatile Safe RL Agent\" signifies the policy incorporating thresholds as conditions, whereas \"Constraint-Conditioned Behavior Policy Learning\" encompasses the CVI process with behavior policy conditions (lines 3-15 in the new algorithm diagram). We will add the algorithm diagram in the revision.\n\n\n> W2.1: ... Although the safety criteria show good performance, the reward counterpart is not...\n\nThank you for your valuable feedback. We understand the reviewer's confusion that the rewards of baselines in some tasks are slightly higher than our method's. However, we would like to gently point out that this is a common trade-off between task utility and safety performance in the safe RL literature -- the **optimal** policy under safety constraints might not be the most rewarding one [R-1, R-2, R-3]. \nTo be more concrete, the evaluation criteria in safe RL are:\n1) method A is better than method B if A achieves better safety performance than B.\n2) If both A and B satisfy constraints, the one with the higher reward is better.\n\nTherefore, we can conclude that CCPO outperforms baselines in most experiments. We will make the evaluation criteria clear in the revision to avoid confusion.\n\n> W2.2: Claims on sample efficiency...\n\nThank you for your valuable feedback. Due to the page limit, we initially placed the discussion about $\\epsilon$-sampling efficiency in Appendix D.4. However, we acknowledge the inconvenience this may cause and will address this by incorporating the discussion into the main context in the revision. Below are the evaluation results.\n\nThe proposed algorithm effectively demonstrates $\\epsilon$-sampling efficiency, as it meets the first requirement outlined in **Section 3.2: Thresholds Sampling Efficiency**: enabling training of a versatile safe RL agent with a limited set of behavior policies for data collection. We compare our approach with C-TRPO, the strongest baseline method, as highlighted in Table 1. We evaluate algorithms using different behavior policy sets, specifically $\\tilde{\\mathcal{E}} = \\{20, 40, 60\\}$ and $\\tilde{\\mathcal{E}}^{\\prime} = \\{20, 30, 40, 50, 60, 70\\}$, where $|\\tilde{\\mathcal{E}}^{\\prime}| = 2 \\times |\\tilde{\\mathcal{E}}|$. The evaluation encompasses threshold conditions $\\mathcal{E} = \\{10, 15, ..., 70\\}$, and we report the averaged performance outcomes in the subsequent table.\n\n\n| Tasks | stats | BC | CC |  DC | DR | Average |\n| :------: | :------: | :------: | :------: | :------: | :------: | :------: |\n| CCPO with $\\tilde{\\mathcal{E}}$ | Avg. R $\\uparrow$ | 710.86 | 406.06 | 630.55 | 458.69 | **551.54** |\n| | Avg. CV $\\downarrow$ | 0.59 | 1.60 | 0.32 | 0.23 | **0.69** |\n| C-TRPO with $\\tilde{\\mathcal{E}}$ | Avg. R $\\uparrow$ | 699.38 | 461.14 | 380.77 | 461.70 | 500.75 |\n| | Avg. CV $\\downarrow$ | 4.50 | 7.84 | 6.55 | 47.97 | 16.72 |\n| C-TRPO with $\\tilde{\\mathcal{E}}^{\\prime}$ | Avg. R $\\uparrow$ | 682.94 | 458.13 | 411.91 | 472.89 | 506.47 |\n| | Avg. CV $\\downarrow$ | 2.66 | 11.90 | 5.20 | 30.20 | 12.49 |\n\nWe observe that **CCPO Exhibits Significant $\\epsilon$-Efficiency Over C-TRPO:** CCPO outperforms C-TRPO significantly in terms of both safety and reward, even with a smaller set of behavior policy conditions. This comparison showcases the remarkable $\\epsilon$-efficiency of our approach. In fact, our method demonstrates at least 2 times greater $\\epsilon$-sampling efficiency compared to exhaustively training safe RL agents using C-TRPO.\n\n> W2.3: It should be noted that running time should be considered as well because as admitted in the discussion, the method can be slow due to complicated optimization steps.\n\nThank you for your valuable feedback and detailed discussion. The running time is presented below.\n\n**Running time:** All the experiments are run on the AMD EPYC 7713 64-Core Processor without GPU acceleration. We report the averaged running time based on the experiment logs in the following table. All the experiments can be done within 24h, and the training can be further accelerated if GPUs are available.\n\n| Tasks | BC | CC | DC | DR | AR |\n| :------: | :------: | :------: | :------: | :------: | :------: |\n| CCPO | 10h | 10h | 22h | 15h | 23h |\n| V-DDPG | 6h | 7h | 17h | 14h | 13h |\n\n> L1: The method is complicated...\n\nThank you for your feedback. The proposed CCPO method has the potential to extend to visual-input tasks. However, state feature extracting may be necessary since it can help in the versatile Q function learning and the policy optimization process. We leave it as a future work.\n\n**Reference** \n\n[R-1] Yu, Haonan, Wei Xu, and Haichao Zhang. \"Towards safe reinforcement learning with a safety editor policy.\" Advances in Neural Information Processing Systems 35 (2022): 2608-2621.\n\n[R-2] Stooke, Adam, Joshua Achiam, and Pieter Abbeel. \"Responsive safety in reinforcement learning by pid lagrangian methods.\" International Conference on Machine Learning. PMLR, 2020.\n\n[R-3] Liu, Zuxin, et al. \"On the Robustness of Safe Reinforcement Learning under Observational Perturbations.\" The Eleventh International Conference on Learning Representations. 2022."}, {"review_id": "GDMe0iA0xQ", "reviewer": "Reviewer_tMdC", "summary": "1. This paper introduces the Conditioned Constrained Policy Optimization (CCPO) framework, consisting of two modules: (1) Versatile Value Estimation (VVE) for approximating value functions under unseen threshold conditions, and (2) Conditioned Policy Inference (CPI) for encoding arbitrary constraint thresholds during policy optimization.\n2. The proposed method can enable learning versatile safe policies that can adapt to varying safety constraint requirements during deployment without retraining.", "questions": "1. How reasonable/strong is assumption 2? Especially if latent vectors are highly dimensional. In experiments, what model do you use to estimate the latent vector z(\\epsilon)? \n2. What is the main technical challenge in extending CVPO [30] to the constraint-conditional scenarios?\n3. How does this method scale when there are multiple constraints?\n4. How does this work compare with the following missing literature with similar topics:\nMeta CMDP: \nKhattar, V., Ding, Y., Sel, B., Lavaei, J., & Jin, M. (2022, September). A CMDP-within-online framework for Meta-Safe Reinforcement Learning. In The Eleventh International Conference on Learning Representations.\nNon-stationary CMDP:\nDing, Y., & Lavaei, J. (2023, June). Provably efficient primal-dual reinforcement learning for CMDPs with non-stationary objectives and constraints. In Proceedings of the AAAI Conference on Artificial Intelligence.\nWei, H., Ghosh, A., Shroff, N., Ying, L., & Zhou, X. (2023, April). Provably Efficient Model-Free Algorithms for Non-stationary CMDPs. In International Conference on Artificial Intelligence and Statistics (pp. 6527-6570). PMLR.\n", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. training efficiency and zero-shot adaptation capability: the trained policy can adapt to varying safety constraint requirements during deployment without retraining.\n2. The theoretical analysis for the q-function estimation and constraint violations.\n3. The extensive experiments are implemented to demonstrate the effectiveness of the method. ", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time, valuable feedback, and acknowledgment of our contribution. We address the concerns as follows.\n\n> Q1: How reasonable/strong is assumption 2? Especially if latent vectors are highly dimensional. In experiments, what model do you use to estimate the latent vector $z(\\epsilon)$?\n\nThank you for your question. Assumption $2$ is considered mild. We provide some verification and visualization results of $Q(\\cdot | \\epsilon)$ estimation in Appendix D.1, which support the reasonableness of using this assumption for training. In practice, we employ a small Neural-Network model, as shown in the illustrations in Appendix C.1.\n\n> Q2: What is the main technical challenge in extending CVPO [30] to the constraint-conditional scenarios?\n\nThank you for your question. The main technical challenge for CVPO is the estimation of the Q function $Q(\\cdot|\\epsilon)$ for arbitrary threshold $\\epsilon$, which has been addressed in the **VVE** module of this work.\n\n> Q3: How does this method scale when there are multiple constraints?\n\nThank you for your insightful question. We will leave the multi-constraint versatile safe RL in the future work as challenges arise from two factors: (1) Higher dimension in $z_f(\\epsilon)$: Since $\\epsilon$ is now a vector, learning $z_f$ becomes more challenging. (2) Multi-constraint safe RL itself is more complex than the single-constraint scenario.\n\n\n> Q4: How does this work compare with the following missing literature with similar topics...\n\nThank you for your feedback. The comparison is as follows.\n\n**The Meta CMDP paper:** Khattar, V., Ding, Y., Sel, B., Lavaei, J., & Jin, M. (2022, September). A CMDP-within-online framework for Meta-Safe Reinforcement Learning. In The Eleventh International Conference on Learning Representations. This work focuses on Multi-task Safe RL, involving tasks with different initialization conditions, and dynamic environments, rather than varying constraint thresholds.\n\n**The Non-stationary CMDP papers:** Ding, Y., & Lavaei, J. (2023, June). Provably efficient primal-dual reinforcement learning for CMDPs with non-stationary objectives and constraints. In Proceedings of the AAAI Conference on Artificial Intelligence; and Wei, H., Ghosh, A., Shroff, N., Ying, L., & Zhou, X. (2023, April). Provably Efficient Model-Free Algorithms for Non-stationary CMDPs. In International Conference on Artificial Intelligence and Statistics (pp. 6527-6570). PMLR. These works address safe RL problems in non-stationary environments, where cost/reward functions change during an episode. In contrast, our work focuses on settings with fixed reward/cost functions and varying cost threshold conditions. Exploring versatile safe agent learning in non-stationary environments could be an interesting direction for future research.\n\n\nWe will add the additional discussion in the revision."}]

Author rebuttal:
Dear reviewers,

We sincerely appreciate your valuable and constructive feedback on our paper. In addition to addressing each reviewer's comments, we'd like to highlight the new examples and experiments we added during the rebuttal phase. You can find the figures we reference in the attached PDF file.

### 1. Additional Illustration and Examples:
We are pleased to hear that you found our problem formulation interesting (R5zk, hsfo, 5QUs, TpqY), novel (5QUs), well-motivated (hsfo), and our method useful (R5zk), impressive and elegantly designed (5QUs). In response to your feedback, we have included more motivating examples for the problem settings, explanatory illustrations for the methodology, and a (shorter) algorithm diagram to provide clearer visual context.

### 2. New Experiments:
We are grateful for your positive feedback on our extensive (tMdc, 5QUs) and impressive experiments (5QUs). As a result of your valuable insights, we have incorporated the following new experiments to further enhance our work:

- (Reviewer R5zk, 5QUs) **CPO-Based Baseline**: To showcase the superiority of our approach, we modified CPO to create the versatile CPO and performed comprehensive experiments. The results demonstrate that CCPO consistently outperforms the versatile CPO, reinforcing the effectiveness of our method.
- (Reviewer hsfo) **Exploring More Behavior Policy Conditions**: In response to your suggestion, we conducted additional experiments with varied behavior policy conditions. These experiments highlight CCPO's ability to maintain safety for unseen threshold conditions, and we also offer further ideas to address this challenge.

We genuinely appreciate your time, attention, and valuable feedback.

Best regards,

The Authors

Meta-review: The authors provided extra code. The reviewers reach a consensus that this paper should be accepted. It would be better to improve the experimental part by comparing to existing Safe Policy Optimization methods.
Decision: Accept (poster)