SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\min  _ {\lambda_ {i}, \eta_ {i} \geq 0} g \left(\eta_ {i}, \lambda_ {i}\right) = \lambda_ {i} \epsilon_ {i} + \eta_ {i} \kappa \mathbb {E} _ {\rho_ {q}} \left[ \log \mathbb {E} _ {\pi \left(\cdot \mid \epsilon_ {i}\right)} \left[ \exp \left(\frac {\hat {Q} _ {r} (\cdot \mid \epsilon_ {i}) - \lambda_ {i} \hat {Q} _ {c} (\cdot \mid \epsilon_ {i})}{\eta_ {i}}\right) \right] \right]. \tag {10}
$$
Then we can encode arbitrary safety constraints by calculating the optimal distribution  $q_{i}^{*}(a|s,\epsilon_{i})$  regarding the corresponding condition  $\epsilon_{i}$  efficiently with (9). The term  $q_{i}^{*}(a|s,\epsilon_{i})$  means when conditioned on  $\epsilon_{i}$ , the probability of taking  $a$  at  $s$  for the optimal feasible policy.
Versatile M-step: After the constraint-conditioned E-step, we obtain a set of optimal feasible variational distribution  $q_{i}^{*} = q_{i}^{*}(\cdot |s,\epsilon_{i})$  for each constraint threshold  $\epsilon_{i}$ . In the versatile M-step, we aim to improve the ELBO (5) w.r.t the policy parameter  $\theta$  for  $\epsilon_{i}\in \mathcal{E}$ .
$$
\mathcal {J} (\theta | \epsilon_ {i}) = \mathbb {E} _ {\rho_ {q}} \left[ \alpha \mathbb {E} _ {q _ {i} ^ {*}} \left[ \log \pi_ {\theta} (a | s, \epsilon_ {i}) \right] \right] + \log (p | \epsilon_ {i}) \tag {11}
$$
Using a Gaussian prior for each threshold-conditioned policy, this problem can be further converted to the following supervised-learning problem with KL-divergence constraints [54, 55]:
$$
\max  _ {\theta} \mathbb {E} _ {\rho_ {q}} \left[ \sum_ {i = 1} ^ {| \mathcal {E} |} \mathbb {E} _ {q _ {i} ^ {*}} \left[ \log \pi_ {\theta} (a | s, \epsilon_ {i}) \right] / | \mathcal {E} | \right] s. t. \mathbb {E} _ {\rho_ {q}} \left[ D _ {\mathrm {K L}} \left(\pi_ {\theta_ {j}} (a | s, \epsilon_ {i}) \| \pi_ {\theta} (a | s, \epsilon_ {i})\right) \right] \leq \gamma \forall i, \tag {12}
$$
where  $\mathcal{E}$  is the set for all the sampled versatile policy conditions  $\{\epsilon_i\}$  in fine-tuning stage of training. The constraint in (12) is a regularizer to stabilize the policy update. The proposed CCPO for versatile safe RL and implementation details are summarized in Appendix C.
4.3 Theoretical analysis
Proposition 1 (Bounded safety violation). With the threshold conditions  $\tilde{\epsilon}_i\in \tilde{\mathcal{E}}$  for behavior policies selected to divide the target condition interval  $[\epsilon_L,\epsilon_H]$  evenly, and with confidence level  $1 - \alpha$ , the cost violation of versatile policy under arbitrary threshold condition  $\epsilon \in [\epsilon_L,\epsilon_H]$  is bounded as:
$$
V _ {c} ^ {\pi (\mu_ {0} | \epsilon)} - \epsilon \leq \frac {z _ {\alpha / 2} B (p)}{N ^ {\beta (p)}} \sqrt {\sigma^ {2} K _ {c} ^ {2} M}, \tag {13}
$$
The proof is shown in Appendix B.1. Proposition 1 ensures that the cost violation of the versatile safe RL agent on unseen thresholds can be bounded if the selected behavior policy conditions divide the interval  $[\epsilon_L, \epsilon_H]$  evenly. We can observe that the bound (13) is proportional to  $\sqrt{K_c^2 M}$ . Since larger  $K_c$  and  $M$  correspond to a wider range of threshold conditions, this safety violation bound is related to the interval range. Also, we provide the complexity analysis for the  $\epsilon$ -sample, i.e., the estimation error corresponding to the number of behavior policies  $N$  as shown in remark 1.
Remark 1 (ε-sample complexity analysis). The estimation error for  $Q$  functions and safety violation bound decreases as the number of behavior policies  $N$  increases. The decreasing rate is proportional to  $\frac{1}{N^{\beta(p)}}$ , where the exponent of  $N$  is related to  $p$ , which is the representation capabilities of the model to represent the constraint-conditioned policy feature  $z(\epsilon)$ . When  $p$  increases,  $\beta$  also increases as shown in the Appendix B.1 which means when the model capability is high, the proposed method significantly becomes more data-efficient.
The functions  $\beta(p), B(p)$  are shown in Appendix B.1. Since CCPO is under the "RL as inference" framework, we also enjoy many benefits as revealed in previous works, such as the optimality guarantees and training robustness [34], see Appendix A for details.
We aim to answer five major questions in the experiment section: (1) Can we achieve versatile safe RL by simply applying a linear combination of single-threshold policies (2) Can we combine safe RL algorithms with a constraint-conditioned actor to achieve versatile safe RL? (3) What is the performance of our proposed CCPO method in versatile safe RL tasks in terms of constraint violation and reward? (4) How  $\epsilon$ -efficient CCPO is compared to exhaustively training the safe RL agents? (5) What is the contribution of each component in CCPO contribute to the overall performance? We adopt the following experiment setting to address these questions.
Task. The simulation environments are from a publicly available benchmark [56]. We consider two tasks (Run and Circle) and four robots (Ball, Car, Drone, and Ant) which have been used in many previous works as the testing ground [13-15]. For the Run task, the agents are rewarded for running fast between two boundaries and are given constraint violation cost if they run across the boundaries or exceed an agent-specific velocity threshold. For the Circle task, the agents are rewarded for running in a circle but are constrained within a safe region smaller than the target circle's radius. We name the tasks as Ball-Circle, Car-Circle, Drone-Circle, Drone-Run, and Ant-Run.
Constraint-conditioned Baselines. We design these baselines by directly integrating the threshold as a part of the state in the CMDP tuple,  $\bar{s} = [s;\epsilon]$ . The policy is optimized with behavior policy conditions only. We adopt commonly used off-policy safe RL algorithms, SAC-Lag and DDPG-Lag, and name the proposed baselines as V-SAC-Lag and V-DDPG-Lag.
Policy linear combination baselines. We also compare our method with single-threshold policy combinations. Denote  $\epsilon$  as an unseen target threshold for adaptation, and  $\epsilon_{1},\epsilon_{2}$  as two behavior policy conditions closest to  $\epsilon$ . Then the policy for  $\epsilon$  is the combination of  $\pi (\cdot |\epsilon_1)$ ,  $\pi (\cdot |\epsilon_2)$ :
$$
\pi (\cdot | \epsilon) = w _ {1} \pi (\cdot | \epsilon_ {1}) + w _ {2} \pi (\cdot | \epsilon_ {2}); \quad w _ {1} = \left(\epsilon_ {2} - \epsilon\right) / \left(\epsilon_ {2} - \epsilon_ {1}\right), w _ {2} = \left(\epsilon - \epsilon_ {1}\right) / \left(\epsilon_ {2} - \epsilon_ {1}\right) \tag {14}
$$
This method is designed for both threshold interpolation and extrapolation, i.e., the coefficients  $w_{1}$  or  $w_{2}$  can be negative. This baseline draws inspiration from the safe control theory, which suggests the safe input component is proportional to the conservativeness level [57]. To this end, we use two strong on-policy methods PPO-Lag and TRPO-Lag to train the single-threshold agents and named the corresponding baselines as C-PP0-Lag and C-TRPO-Lag. More baselines and results can be found in the Appendix.
Metrics: We compare the methods in terms of episodic reward (the higher, the better) and episodic constraint violation cost (the lower, the better) on each evaluated threshold condition, which have been used in many related works [14, 58]. For all the results shown in section 5.1 and 5.3, the behavior policy conditions are  $\tilde{\mathcal{E}} = \{20, 40, 60\}$  and the threshold conditions for evaluation are set to be  $\{10, 15, \dots, 70\}$ . We take the average of the episodic reward (Avg. R) and constraint violation (Avg. CV) as the main comparison metrics. The constraint violation for threshold  $\epsilon$  is defined as:
$$
\mathbf {C V} = \max  \{0, \Sigma_ {\tau} c _ {t} - \epsilon \} \tag {15}
$$
As mentioned in previous works [58], the general evaluation criteria in safe RL are: (1) method A is better than method B if A achieves better safety performance than B. (2) If both A and B satisfy constraints, the one with the higher reward is better. We also report the average performance solely on unseen thresholds (Avg. R-G and Avg. CV-G) to characterize the adaptation capability. More experiments can be found in Appendix D.
5.1 Main Results and Analysis
The evaluation results are shown in Figure. 2 and Table 1. We shade the two safest agents with the lowest averaged cost violation values.
First, we can observe that our method outperforms the modified versatile safe RL baselines V-SAC-Lag and V-DDPG-Lag that directly concatenate the threshold condition into the state. Although V-SAC-Lag or V-DDPG-Lag gets higher rewards on simple-dynamics tasks Ball-Circle,
Table 1: Evaluation results of proposed CCPO method and the proposed versatile safe RL baselines.  $\uparrow$  : the higher reward, the better.  $\downarrow$  : the lower constraint violation (minimal 0), the better. The models are evaluated on a series of threshold conditions and we report the averaged reward and constraint violation values on all evaluation thresholds and generalized thresholds. Each value is reported as mean  $\pm$  standard deviation for 50 episodes and 5 seeds. We shade the two safest agents with the lowest averaged cost violation values.
Task	Stats	CCPO (ours)	Constraint-conditioned	Linear combination
V-SAC-Lag	V-DDPG-Lag	C-PPO-Lag	C-TRPO-Lag
Ball-Circle	Avg. R ↑	710.86±20.47	774.16±20.34	762.61±58.65	637.85±14.03	699.38±1.94
Avg. CV ↓	0.59±0.31	5.32±5.00	2.81±1.12	3.11±1.64	4.50±0.08
Avg. R-G ↑	699.04±20.48	766.52±22.59	756.67±58.48	667.89±12.17	699.14±2.05
Avg. CV-G ↓	0.83±0.42	6.29±5.72	3.53±1.26	3.40±1.75	5.59±0.25
Car-Circle	Avg. R ↑	406.06±6.30	331.80±11.57	448.82±18.65	440.01±2.59	461.14±1.39
Avg. CV ↓	1.60±0.91	12.18±4.65	14.48±8.14	9.09±1.52	7.84±1.71
Avg. R-G ↑	401.53±5.59	331.19±11.00	445.32±17.42	438.31±3.03	460.72±1.15
Avg. CV-G ↓	1.49±0.38	12.74±4.32	14.63±8.69	11.07±1.58	9.14±2.01
Drone-Circle	Avg. R ↑	630.55±40.03	693.69±22.37	734.58±49.69	392.64±23.13	380.77±18.62
Avg. CV ↓	0.32±0.38	13.24±8.80	19.62±11.15	0.45±0.38	6.55±1.95
Avg. R-G ↑	625.51±40.12	699.14±24.88	730.29±48.43	342.77±19.06	291.87±19.88
Avg. CV-G ↓	0.47±0.55	14.97±10.10	19.44±10.36	0.21±0.09	7.23±2.03
Drone-Run	Avg. R ↑	458.69±12.98	355.61±35.44	244.60±48.29	398.88±21.53	461.70±4.91
Avg. CV ↓	0.23±0.25	8.66±4.30	11.33±9.63	9.46±5.63	47.97±3.49
Avg. R-G ↑	455.64±11.83	354.61±33.34	236.61±43.49	386.77±30.09	464.07±6.61
Avg. CV-G ↓	0.33±0.37	9.96±4.54	12.72±9.91	11.18±7.46	60.39±4.32
Ant-Run	Avg. R ↑	660.88±4.82	615.73±91.99	594.75±172.35	636.06±6.78	629.83±7.84
Avg. CV ↓	3.13±1.67	8.47±3.55	23.69±30.42	5.16±1.59	0.22±0.17
Avg. R-G ↑	660.07±5.26	626.27±84.61	592.50±173.01	620.46±9.99	605.07±10.63
Avg. CV-G ↓	3.25±1.48	7.76±11.83	22.90±9.39	6.73±2.32	0.03±0.06
Figure 2: Results of zero-shot adaption to different cost returns. Each column is a task. The x-axis is the threshold condition. The first row shows the evaluated reward, and the second row shows the evaluated cost under different target costs. All plots are averaged among 5 random seeds and 50 trajectories for each seed. The solid line is the mean value, and the light shade represents the area within one standard deviation. We train the versatile agent with behavior policy conditions  $\tilde{\mathcal{E}} = \{20,40,60\}$ , and evaluate it on  $\mathcal{E} = \{10,15,\dots,70\}$ .
and Car-Circle, their cost violation values are significantly larger than the proposed CCPO cost. In the Drone-Circle, Drone-Run, and Ant-Run tasks characterized by highly-nonlinear robot dynamics, all the baseline methods exhibit poor generalization when being exposed to different thresholds, thus leading to high cost violation values. This limitation arises due to the inadequacy of utilizing only a limited number of behavior policies for versatile policy training in tasks with high-dimensional observation and action spaces. Insufficient conditions prevent the actor from effectively distinguishing between different constraint conditions, thus resulting in large performance variance even for behavior policies, and large cost violations on unseen thresholds. The poor safety
performance when generalizing to different threshold conditions indicates the necessity of studying versatile safe RL methods.
Second, although the linear combination of single-threshold policies performs well on Ball-Circle and Car-Circle with simple robot dynamics, it can hardly handle Drone-Circle, Drone-Run, and Ant-Run tasks – the interpolation method has a significant reward drop at unseen thresholds. It is because, for Ball-Circle and Car-Circle task, the action dimensionality is low and the dynamics are simple, thus directly applying linear combination may work in these settings. However, for Drone-Circle and Drone-Run, the high nonlinearity in agent dynamics and the large action space will make this naive approach fail to perform well. These results show that the concepts from the control theory that the safety-critical control component is proportional to the conservativeness level can not be directly used in versatile safe RL with high-dimensional settings, which further indicates the necessity of the versatile safe RL method.
Finally, from Table 1, we can clearly see that our proposed CCPO method learns a versatile safe RL policy that can generalize well to unseen thresholds with low cost-violations and high rewards. Also, from Figure 2, we can observe the performance of the CCPO method has a smooth relation with respect to threshold conditions, which indicates it can efficiently encode the threshold conditions into the versatile safe RL agent.
5.2 Evaluation of  $\epsilon$ -sampling efficiency
The proposed algorithm is  $\epsilon$ -sampling-efficient as it satisfies the first requirement thresholds sampling efficiency mentioned in section 3.2; it is able to train the versatile safe RL agent with limited behavior policies for data collection. In this experiment, we aim to answer the question: How  $\epsilon$ -efficient CCP0 is compared to exhaustively training the safe RL agents? We compare our method with C-TRP0, which is the strongest baseline method as shown in Table. [1] with different behavior policy set  $\tilde{\mathcal{E}} = \{20, 40, 60\}$  and  $\tilde{\mathcal{E}}' = \{20, 30, 40, 50, 60, 70\}$ , where  $|\tilde{\mathcal{E}}'| = 2|\tilde{\mathcal{E}}|$ . The algorithms are evaluated on threshold conditions  $\mathcal{E} = \{10, 15, \dots, 70\}$ , and the averaged performance is reported in Table. [2].
We observe that CCPO exhibits significant  $\epsilon$ -efficiency over C-TRPO-Lag: CCPO outperforms C-TRPO-Lag significantly in terms of both safety and reward, even with a smaller set of behavior policy conditions. This comparison showcases the remarkable  $\epsilon$ -efficiency of our approach. In fact, our method demonstrates at least 2 times greater  $\epsilon$ -sampling efficiency compared to exhaustively training safe RL agents using C-TRPO-Lag.
Table 2:  $\epsilon$ -sampling efficiency evaluation.  $\uparrow$ : the higher reward, the better.  $\downarrow$ : the lower constraint violation (minimal 0), the better. The models are evaluated on a series of threshold conditions and we report the averaged reward and constraint violation values on all evaluation thresholds and generalized thresholds. Each value is reported as mean  $\pm$  standard deviation for 50 episodes and 5 seeds. We shade the safest agent with the lowest averaged cost violation value.
Algorithm	Stats	Ball-Circle	Car-Circle	Drone-Circle	Drone-Run	Averaged Score
CCPO with Ε	Avg. R ↑	710.86±20.47	406.06±6.30	630.55±40.03	458.69±12.98	551.54
Avg. CV ↓	0.59±0.31	1.60±0.91	0.32±0.38	0.23±0.25	0.69
C-TRPO with Ε	Avg. R ↑	699.38±1.94	461.14±1.39	380.77±18.62	461.70±4.91	500.75
Avg. CV ↓	4.50±0.08	7.84±1.71	6.55±1.95	47.97±3.49	16.72
C-TRPO with Ε'	Avg. R ↑	682.94±8.08	458.13±2.22	411.91±8.95	472.89±2.65	506.47
Avg. CV ↓	2.66±0.37	11.90±2.12	5.20±0.81	30.20±2.47	12.49
To study the influence of VVE, and CVI components of CCPO introduced in the section 4.1 and 4.2 we conduct an ablation study by removing each component from the full CCPO algorithm. The ablation experiment results are shown in Table 3. We shade the safest agent with the lowest averaged cost violation value. We can also observe significant safety performance (cost violation) degradation and task performance (reward) drop if we remove the VVE module (versatile critic learning) or CVI module (versatile actor learning) since they help us learn versatile Q functions more accurately and
improve the generalizability of learned actors. Removing them will result in the bad estimation of for state-action pair value function and lead to poor policy generalization capability.
Table 3: Ablation study of removing the versatile value function estimation (VVE), and the conditioned variational inference (CVI).  $\uparrow$ : the higher reward, the better.  $\downarrow$ : the lower constraint violation (minimal 0), the better. Each value is reported as mean  $\pm$  standard deviation for 50 episodes and 5 seeds. Each value is reported as mean  $\pm$  standard deviation.
Algorithm	Stats	Ball-Circle	Car-Circle	Drone-Circle	Drone-Run	Ant-Run
CCPO (Full)	Avg. R ↑	710.86±20.47	406.06±6.30	630.55±40.03	458.69±12.98	660.88±4.82
Avg. CV ↓	0.59±0.31	1.60±0.91	0.32±0.38	0.23±0.25	3.13±1.67
Avg. R-G ↑	699.04±20.48	401.53±5.59	625.51±40.12	455.64±11.83	660.07±5.26
Avg. CV-G ↓	0.83±0.42	1.49±0.38	0.47±0.55	0.33±0.37	3.25±1.48
CCPO w/o VVE	Avg. R ↑	674.55±17.81	370.42±14.38	426.47±49.30	417.84±8.24	428.59±88.39
Avg. CV ↓	0.60±0.41	6.42±0.85	8.67±1.45	3.28±2.86	10.66±11.81
Avg. R-G ↑	670.61±14.18	364.5±14.51	416.83±47.46	413.28±9.04	434.59±83.89
Avg. CV-G ↓	0.73±0.36	5.64±0.92	7.74±1.36	3.33±3.16	12.01±10.08
CCPO w/o CVI	Avg. R ↑	641.33±40.14	387.31±5.76	520.70±42.18	386.81±39.44	465.80±31.78
Avg. CV ↓	1.44±0.72	1.66±0.79	2.36±2.67	0.81±0.76	3.51±0.93
Avg. R-G ↑	623.17±41.42	383.24±6.30	519.05±36.31	388.69±35.35	465.36±32.20
Avg. CV-G ↓	1.78±0.70	2.17±1.09	2.73±3.03	1.15±1.08	3.96±1.01
In this study, we pioneered the concept of versatile safe reinforcement learning (RL), presenting the Constraint-Conditioned Policy Optimization (CCPO) algorithm. This approach adapts efficiently to different and unseen cost thresholds, offering a promising solution to safe RL beyond pre-defined constraint thresholds. With its core components, Versatile Value Estimation (VVE) and Conditioned Variational Inference (CVI), CCPO facilitates zero-shot generalization for constraint thresholds. Our theoretical analysis further offers insights into the constraint violation bounds for unseen thresholds and the sampling efficiency of the employed behavior policies. The extensive experimental results reconfirm that CCPO effectively adapts to unseen threshold conditions and is much safer and more data-efficient than baseline methods. The limitations include that our method would be more computationally expensive than the primal-dual-based safe RL approaches due to the convex optimization problems in the constraint-conditioned E-step. One potential negative social impact is that the misuse of this work in safety-critical scenarios can cause unexpected damage. We hope our findings can inspire more research in studying the generalization capability in safe RL.
7 Acknowledgment
The work is partially supported by Google Deepmind with an unrestricted grant. The authors also want to acknowledge the support from the National Science Foundation under grants CNS-2047454.
[1] Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330, 2022.
[2] David Isele, Alireza Nakhaei, and Kikuo Fujimura. Safe reinforcement learning on autonomous vehicles. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1-6. IEEE, 2018.
[3] Hongyuan Ding, Yan Xu, Benjamin Chew Si Hao, Qiaoqiao Li, and Antonis Lentzakis. A safe reinforcement learning approach for multi-energy management of smart home. Electric Power Systems Research, 210:108120, 2022.
[57] Tianhao Wei and Changliu Liu. Safe control algorithms using energy functions: A uni ed framework, benchmark, and new directions. In 2019 IEEE 58th Conference on Decision and Control (CDC), pages 238-243. IEEE, 2019.
[58] Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09303, 2023.
[59] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
[60] Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. Advances in Neural Information Processing Systems, 30, 2017.
[61] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelCNN decoders. Advances in neural information processing systems, 29, 2016.
[62] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[63] Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. arXiv preprint arXiv:1912.13465, 2019.