Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation
Authors: Shangding Gu
Laixi Shi
Yuhao Ding
Alois Knoll
Costas Spanos
Adam Wierman
Ming Jin
Venue: NeurIPS 2024 poster (2024)
Keywords: Safe Reinforcement Learning
Sample Manipulation
Efficient Learning
Primary area: safety_in_machine_learning

Abstract: Safe reinforcement learning (RL) is crucial for deploying RL agents in real-world applications, as it aims to maximize long-term rewards while satisfying safety constraints. However, safe RL often suffers from sample inefficiency, requiring extensive interactions with the environment to learn a safe policy. We propose Efficient Safe Policy Optimization (ESPO), a novel approach that enhances the efficiency of safe RL through sample manipulation. ESPO employs an optimization framework with three modes: maximizing rewards, minimizing costs, and balancing the trade-off between the two. By dynamically adjusting the sampling process based on the observed conflict between reward and safety gradients, ESPO theoretically guarantees convergence, optimization stability, and improved sample complexity bounds. Experiments on the Safety-MuJoCo and Omnisafe benchmarks demonstrate that ESPO significantly outperforms existing primal-based and primal-dual-based baselines in terms of reward maximization and constraint satisfaction. Moreover, ESPO achieves substantial gains in sample efficiency, requiring 25--29\% fewer samples than baselines, and reduces training time by 21--38\%.

Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation
Shangding Gu 1,3 Laixi Shi Yuhao Ding', Alois Knoll Costas Spanos', Adam Wierman',
Gu
Ming Jin
University of California, Berkeley, USA ?California Institute of Technology, USA Tecchiiall University of Munich, Germany 4 Virginia Tech, USA
Abstract
Safe reinforcement learning (RL) is crucial for deploying RL agents real-world applications, it aims to maximize long-term rewards while satisfying safety constraints. However, safe RL often suffers from sample inefficiency, requiring extensive interactions with the environment to learn safe policy. We propose Efficient Safe Policy Optimization (ESPO). a novel approach that enhances the efficiency of safe RL through sample manipulation. ESPO employs optimiza- tion framework with three modes: maximizing rewards, minimizing costs, and balancing the trade-off between the two. By dynamically adjusting the sampling process based on the observed conflict between reward and safety gradients, ESPO theoretically guarantees convergence, optimization stability, and improved sample complexity bounds. Experiments on the Safety-MuJoCo and Omnisafe benchmarks demonstrate that ESPO significantly outperforms existing primal-bases and primal- dual-based baselines in terms of reward maximization and constraint satisfaction. Moreover, ESPO achieves substantial gains in sample efficiency, requiring 25-29% fewer samples than baselines, and reduces training time by 21-38%.
Introduction
Reinforcement learning (RL) [49] has demonstrated powerful capabilities in various domains [25, 46. 11]. However, ensuring safety RL, particularly in real-world applications such autonomous driving, robotics, and power grids, is crucial [6, 12, 31, 35, 38, 39, 41, 48. 55, 57, 17, 59]. Safe RL aims maximize long-term cumulative rewards while adhering to additional safety cost constraints.
Most state-of-the--ee art (SOTA) safe RL methods, including primal-based baselines (e.g.. CRPO [53], PCRPO [30]) and primal-dual-based methods (e.g., CUP [54], PPOLag [32]). optimize cost and reward with a predetermined sample size for each iteration. However, this approach may lead to sample inefficiency due two main reasons:
Wasted samples and computational resources in simple scenarios, where the (computa- tional/physical) cost of obtaining these samples may outweigh their learning benefits. Insufficient exploration in complex cases with high uncertainty or conflicting objectives, potentially hindering the learning of a safe and optimal policy.
key insight from optimization literature suggests that the selection of sample size worthwhile but delicate issue, as it may vary depending the optimization stage and landscape [13, 28, 51]. However, this insight remains largely unexplored in the context of safe RL, where the consideration
Equual contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

We provide a of safety adds complexity. The presence of safety constraints can create regions with high conflict between reward and safety objectives, requiring careful balancing and potentially more samples to resolve. Therefore, an unresolved question in safe RI. is: Can we enhance sample efficiency by dynamically adapting the sample size, while simultaneously improving reward performance and guaranteeing safety?
To address this question, we focus on primal-based ap- proaches, which do not require fine-tuning of dual pa- rameters or heavily rely on initialization, unlike primal- dual- based optimization [30, 53]. The key to effectively enhancing sample efficiency is to establish reliable cri- teria for determining sample size requirements. In- spired by insights from multi-objective optimization/RI. [40, 37, 30], we use gradient conflict befween rewards and costs as an effective signal for adjusting sample size in each iteration. Intuitively, when gradient conflict occurs, balancing reward and safety optimization with uniform sample size becomes challenging; conversely, when there is gradient aligmment, optimizing with fewer samples is more straightforward. This motivates to adopt three-mode optimization framework: 1) optimizing cost exclusively upon safety violation; 2) simultancously optimizing both reward and cost during a soft constraint violation; 3) optimizing only the reward when violations present. This allows tailored sample size adjustment based on the optimization regime. We increase the sample size in situations of gradient conflict incorporate more informative samples and reduce it in cases of gradient alignment to prevent unnecessary costs and training time. This sampling adjustment is effective in each policy learning mode (cost only, simultaneous reward and cost, and reward only), enabling the search for improved policies that prioritize safety, rewards, or a balance of both.
Reward only
Beward-cnt balance
Cost only Constraints
--- ESPO [ours] ---- Existing methoes
paint
Figure I: Oscillation analysis compared our method with existing safe RL methods in three modes of optimization.
This study makes three key contributions emphasizing sample manipulation for safe RL:
① We propose Efficient Safe Policy Optimization (ESPO), an algorithm that depart from prior arts byincorporating sample manipulation by leveraging gradient conflict signals as criteria to enhance sample efficiency and reduce unnecessary interactions with the environments, comprehensive theoretical analysis of ESPO, including convergence rates, the advantages of reducing optimization oscillation, and provable sample efficiency. The theoretical results inspire ESPO's sample manipulation approach and could be of independent interest for broad RL applicability. @ We evaluate ESPO through comparative and ablation experiments on two benchmarks: Safety- MuJoCo[30] and Omnisafe[32]. The results demonstrate that ESPO improves reward performance and safety compared to SOTA primal-based and primal-dual-bant baselines. Notably, ESPO significantly reduces the number of samples used during policy learning and minimizes training costs while ensuring safety and achieving superior reward performance.
2 Related Works
Various methodologies have been developed to enhance safety in RL [12, 31], including constrained optimization-basee based methods, Among these, constrained optimization-basel methods have gained notable popularity due to their ease of use and reduced dependency on external knowledge [31].
methods [18, 19, 33, 29], and formal methods [43].
Constrained optimization-based methods can be categorized into primal-dual (e.g., CPO [1], PCPO [56], CUP [54]) and primal approaches. Primal-dual methods face challenges in tuning dual multipli- ers, ensuring feasible initialization, and sensitivity to learning rates [53, 30J. Primal methods offer a distinct advantage by climinating the need for dual multipliers. A prominent primal-based method is CRPO [53], which focuses on directly optimizing the primal problem. When safety violations occur, CRPO exclusively improves the violated constraints. However, it encounters significant challenges with conflicting gradients between optimizing rewards and constraints, which can impact ensuring both performance and ongoing safety compliance. PCRPO [30] addresses this issue by balaneing the trade-offs between reward and safety performance through strategic gradient manipulation. How- ever, lacks comprehensive convergence and sample complexity analysis and faces computational challenges due to the need to compute reward and safety gradients in each gradient handling step.

---

Several efficient safe RL methods have been recently proposed [16, 21, 22, 23, 24, 36, 42, 47, 50]. including offline [47] and off-policy settings [36, 42]. Our model-free, on-policy approach is distinguished by its dynamic calibration of sampling based on the interplay between reward maximization and safety assurance. Closely related works are [21] and [24]. [21] employs symbolic reasoning for safety but relies on external knowledge, potentially limiting applicability. [24] proposes a non-stationary safe RL approach with regret bounds using linear function approximation but may struggle with complex tasks and inherits issues common in primal-dual safe RL [22, 23]. Our
primal-based method circumvents these drawbacks.
Adaptive sampling methods in optimization can be categorized into prescribed (e.g.. geometric)
sample size increase [13, 26] [13, 8]. gradient approximation test [14, 7, 13, 8, 15, 10, 5], and derivative-fres free [45, 9] and simulation- based methods [44] (see [20] for a review). These methods focus on controlling the variance of gradient approximations or function evaluations (e.g., through inner product [8] or norm tests [14, 15]) to balance computational efficiency and sample complexity. Adaptive sampling methods have also been applied to constrained stochastic optimization problems with convex feasible sets [4, 52]. A recent work [60] extends adaptive sampling to multi-objective setting, but their criteria are still based on variance. Our research introduces a novel perspective by focusing on conflict-aware updates based safety and performance gradients in safe RL, making the first adaptive sampling method for this important domain.
Problem Formulation
A Constrained Markov Decision Process (CMDP) [3] is often used to model safe RL problems. A CMDP is denoted as AS,A, P. P.r. c, b, 7), where S is the state space, A is the action space, P x A x S + [0, 1] is the transition probability function, r S x A
R is the reward function,
and y is the discount factor. To encode safety, == (cI,..., Cx) )= S x A - IR" is the cost function assigning costs state-action pairs, with higher costs indicating higher risks, b = (b1....,bb, b,,) TR" contains safety thresholds for each constraint. This CMDP framework searches for a safe policy n in the stochastic Markov policy set 11, balancing rewards and safety constraints.
The expected cumulative reward values are defined (.T E - 7'r (st: a.) 80 and
@7 a) = o 7 ax) n, = s, for states and state-action pairs, respectively. Simi- larly, safety is guantified using the cost state values V- (s) and cost state-attion values Q a). The primary objective in safe RL is to maximize the accumulative reward while ensuring safety, under an initial state distribution p:
max V. (p) = EsNp [V. (s)], s.t. (C' = = V. (s) I b. Tell
(1)
However, conflicts often arise in safe RL between the reward gradient gr = VV, (p) and negative cost gradient g. = -VV (p). These conflicts can lead to unstable policy updates that cause experiences violating safety constraints, forcing reversion to prior policies and wasting samples. Such unstable dynamics further impede efficient exploration, risking premature convergence and squandering of computational resources. This study aims to efficiently search for safe policy by manipulating samples to reduce waste and improve safe RL efficiency.
Algorithm Design and Analysis
4.1 Three-Mode Optimization
To improve learning efficiency and mitigate oscillations, we leverage PCRPO [30] and categorize performance optimization into three distinct strategies: focusing on reward, on both reward and
cost simultaneously, or solely on cost. Two essential parameters are introduced to construct a soft constraint region h- the lower side and h+ the upper side. With h hh in hand, [30] divides the optimization process into three modes as below. Throughout the paper, we parameterize the policy π by w.
1) Safety Violations. When the cost values (p) > (I+ + b), we apply (2) to update the policy parameter w. with learning rate 17. In such mode, since the constraints are violated, prioritize
3

---

safety and choose to minimize the cost objective to achieve compliance with safety standards.
W+++1 = W. + 7gc
+ b). we leverage (3) and (4) for simultaneous optimization of reward and safety performance. Specifically, when within the soft constraint region, the conftict between the reward and cost gradients is characterized by the angle 0, between the reward gradient g, and the cost gradient ge. When By.c > 90", indicates the directions that optimize the reward and the safety performance are in conflict, and the update rule is (3).
2) Soft Constraint Violations. When V (p) E 1A-- +b,
]
gr
ge
ge gr
Wr++1
Wt + 17
33 8r
g
+
ge
1
8r
(3)
We + 77 [ g. + ge
(4)
where r;; 25 2 0 and xf + 2 = for all t E T. It employs gradient projection techniques [30, 58), projecting reward and cost gradients onto their normal planes and ensuring that the policy adjustment balances the conflicting objectives of maximizing rewards and minimizing costs. In contrast, when Or. S 90", namely, the directions for maximizing rewards and minimizing costs are aligned or do not significantly oppose each other, we use the update rule (4). In this scenario, the gradient for the update is computed based on the weight of the reward and cost gradients. This method leverages the synergistic potential between reward maximization and cost minimization, aiming for policy update that harmoniously improves both aspects.
3) No Violations. When V. (p) < (h- + b), the update rule in (5) is applied to optimize the policy: Wh+11 = W/ + 7gr.
(5)
In other words, given that the policy adheres to all specified constraints, only the reward objective is considered.
4.2 Sample Size Manipulation
As introduced above, PCRPO [30] allows for adaptive optimization updates based on different conditions. However, PCRPO and other existing safe RL methods usually apply an identical sample size during the learning process, resulting in potentially unnecessary computation cost for simpler tasks and inadequate exploration for more complex tasks. Furthermore, there is no existing theoretical analysis for PCRPO, leaving the performance guarantces of somewhat uncharted. To address propose method called ESPO based on crucial sample manipulation
the above challenges, approach that will be introduced momentarily. A comprehensive theoretical analysis of ESPO is provided in Section 4.4.
Throughout the framework of three-mode optimization, our proposed method dynamically adjusts the number of samples utilized at each iteration based on the criteria of gradient conflict, to meet specific demands of reducing unnecessary samples simpler scenarios and increasing exploration in more complex situations. Specifically,, we consider the three-mode optimization classified by the gradient-conffict criteria respectively. (a) Soft Constraint Violations with Gradient Conflict, where Or,c > 90º (cf. (6)): the cases with slight safe constraint violation and gradient conflict between reward and safety objectives. In this scenario, adjusting the sample size becomes crucial for sufficiently exploring the environments to identify careful balancer udpate direction. We increase the sample size in (6) to enhance the likelihood of achieving a near-optimal balance between the reward and cost objectives. 2)(b)Soft Constraint Violations without Gradient Conflict, where 8r. & 90 (cf. (7)): the cases with slight safe constraint violation and gradient alignment between reward and safet objectives. Considering it is easier search for a update direction that benefits the aligned reward and cost objectives, we reduce the sample size in (7) to achieve efficient learning. 1) and 3) Safety Violations and No Violations: only reward or cost objective is considered. It indicates that there is no gradient conflict since only one objective is targeted, where we also employ the update rule in (7).
For more details, we dynamically adjust the sample size X, (X denote a default fixed sample size), with e and & representing some sample size adjustment parameters.
X x if 0re 90"
(6)
X
x xx+ if One S 90".
(7)

---

This gradient-conflict based sample manipulation is a crucial feature of our proposed method, which enables adaptively sample size tailored the specific nature of the joint reward-safety objective landscape at each update iteration.
4.3 Efficient Safe Policy Optimization (ESPO)
Building upon the above two modules three-mode optimization and sample size manipulation, we have formulated a practical algorithm. The details of this algorithm are summarized in Algorithm in Appendix B. This algorithm encompasses a strategic approach sample size adjustment and policy updates under various conditions: 1) Safety Violations: When a safety violation occurs, we adjust the sample size X, using Equation (7). Simultaneously, the policy Tw. updated to ensure safety, as dictated by Equation (2). (a) Soft Constraint Violations with Gradient Angle S 90%: In modes of soft region violation where the angle fr.c between gradients gr and ge less than or equal to 90%, we adjust the sample size X₂ using Equation (7). The policy Tw, is then updated in accordance with Equation (3). ()) Soft Constraint Violations with Gradient Angle 90% Conversely, the soft region violation occurs with gradient angle Or.c exceeding 90, the sample size X. is adjusted via Equation (6). Policy updates are made using Equation (4). 3) No Violations: In the absence of any violations, the sample size X, is altered using Equation (7). The policy Tw, is then updated to maximize the reward V (p). following Equation (5). This practical algorithm reflects an insightful analysis of the interplay between reward maximization and safety assurance in safe RL, tailoring the learning process to the specific demands cach scenario.
4.4 Theoretical analysis of ESPO
In this section, we provide theoretical guarantees for the proposed ESPO, including the convergence rate guarantee and provable optimization stability and sample complexity advancements.
Tabular setting with softmax policy class. In this paper, we focus on fundamental tabular setting with finite state and action space. We consider the class of policies with the softmax parameterization which is complete including all stochastic policies. Specifically, policy Tu associated with w e
RISIA is defined as
exp(w(s,a)
(s, a) E ESXA=
The (als)
(8)
En' exp(w(s,a'))
Before proceeding, introduce some useful notations. When executing ESPO (cf. Algorithm 1), let B, Bsott and Be denote the set of iterations using Safety Violation Response (mode 1), Soft Constrains Violation Response (mode 2), and No Violation Response (mode 3) in Section 4.3, respectively.
Provable convergence of ESPO. First, we present the convergence rate of our proposed ESPO in terms of both the optimal reward and the constraint requirements in the following theorem; the proof is given in Appendix A.3.
Theorem 4.1. Consider tabular setting with policy class defined in (8), and any ö E (). For Algorithmm 1, applying Tp: = Ō ( log( [S|[AA)
(1->)3 s Al) iterations for each policy evaluation step, set tolerance
h+ =Ö(- - 2/ISIIA
(1-))L8TT and the learning rate of NPG update l (1 (1-7)1.5 / ATT Then, the output के of Algorithm satisfies that with probability at least 1 - 5,
(V1-11777
|S|IA1
B[V ()) -V (p) 5 (V |S||AI
V. (p) BVV (
(1 (1-7)3T
Here, the expectation is taken with respect to the randomness of the output म. which is randomly selected from { JisiST with a certain probability distribution (specified in Appendix (30)).
Theorem 4.1 demonstrates that taking the output policy मे as random one selected from TT hiser following some distribution, the proposed ESPO algorithm achieves convergence to a globally
Throughout this paper, the standard notation Ö(-) indicates the order of a function with all constant terms hidden.
5

---

optimal policy x² within the feasible safe set, following the convergence rate of Ō ((=)eer)
The convergence rate for constraint violations towards 0 is also Ō
While note that the implementation of Algorithm practice only need to output the final π Twr for simplicity. The
SA (1->)²
randomized procedure is only used for theoretical analysis.
We observe that ESPO enjoys the same convergence rate as the well-known primal safe RL algorithm CRPO [53]. In addition, Theorem (4.1) directly indicates the same convergence rate guarantee for PCRPO [30] the three-mode optimization framework that our ESPO refer to, which closes the gap between practice and theoretical guarantees for PCRPO [30]. Technically, to handle the variation in ESPO's 's update rules across a three-mode optimization process compared to CRPO, deriving the results necessitates to overcome additional challenges by tailoring new distribution probability for the algorithm that is used to randomly select policies from (Tw. h 1SiTT-
Besides the efficient convergence, in the following, we present two advantages of ESPO in terms of both optimization benefits and sample efficiency; the proof are provided in Appendix A.4 and A.5 respectively.
I: Efficient optimization with reduced oscillation. Shown qualitatively in Figure compared to other primal safe RL algorithms (such as CRPO), our proposed ESPO can significantly increase the ratios of iterations for maximizing the reward objective within the (relaxed) soft safe region by reducing oscillation across the safe region boundary. We provide a rigorous quantitative analysis for such advancement as below:
Proposition 4.2. Suppose CRPO [53] and ESPO (ours) are initialized
identical point wy
RI Denote the set of iterations that CRPO updates according to the reward objective as 1 ICCRPO
Then by adaptively choosing the parameters (2), æii of Algorithm 1, if there exist iteration tin T such that E B, L Baott one has
Whnn SIST: 8 B, Booft
(9a)
B, + |Bott = T Li 2 B CRPO
(9b)
In words, (9a) shows that as long as ESPO (cf. Algorithm 1) enters the safe region that the constraint violated most h+, it will stay and always (at least partially) optimizes the reward objective without oscillation across the safe region boundary. In addition, (9b) indicates that the proposed ESPO enables more iterations to maximize the reward objective inside the safe region with comparison to CRPO, accelerating the optimization towards the global optimal policy. These two theoretical guarantees are further corroborated by the phenomena in practice (shown in Table 3): ESPO spends more iterations (99.4% steps) optimizing the reward objective inside the safe region compared to CRPO 335.6% steps), while only a few on solely cost objective.
III: Sample efficiency with sample size manipulation. Besides the efficient optimization of ESPO, the following proposition presents the provable sample efficiency ESPO.
Proposition 4.3. Consider any 0 S 62 -
To meet the following goals of performance gaps
VF' EVV (p) S e1, (p)) (p) 82, ESPO (Algorithm 1) needs fewer number of samples than that without the sample manipulation in Section 4.2.
(10)
The result demonstrates that, considering the accuracy level/constraint violation requirements, the sample manipulation module contributes to a more sample-efficien- algorithm ESPO (Algorithm 1). Additionally, the conflict between reward and cost gradients emerges as an effective metric for determining sample size requirements.
Experiments and Evaluation
To evaluate the effectiveness of our algorithm, we compare it with two key paradigms in safe RL frameworks. The first paradigm is based on the primal framework, including PCRPO [30] and CRPO [53] the representative baselines. The second paradigm includes methods that leverage

---

the primal-dua framework, with PCPO [56], CUP [54], and PPOLag [32] serving as representative methodologies. Our algorithm is developed within the primal framework, thereby highlighting the importance of comparing it against these paradigmatic safe RI. algorithms to clearly demonstrate its performance. Experiments are conducted using both primal and primal-dual benchmarks. The Om- nisafe" [32] benchmark is leveraged for primal-dual based methods, where representative techniques such as PCPO [56], CUP [54]. and PPOLag [32] generally exhibit stronger performance compared to existing primal methods like CRPO [53]. finding discussed in [27]. Additionally, we use the methods. This benchmark, developed in 2024, is relatively and primarily supports primal-basee based methods due to the specific implementation efforts involved. The detailed experimental settings are provided in Appendix D. Furthermore, to thoroughly evaluate the effectiveness of our method, we conduct series of ablation experiments regarding different cost limits and sample manipulation techniques. In particular, we provide performance update analysis in terms of constraint violations. These experiments specifieally designed to dissect and understand the impact of various factors integral to our approach.
Safety-MuJooo [30] benchmark for primal-base
5.1 Experiments of Comparison with Primal-Baa Frimal-Based Methods
We deploy our algorithm on the Safety-MuJoCo benchmark and carry out experiments compared with representative primal algorithms, PCRPO [30] and CRPO [53]. Specifically, we conduct experiments on a set of challenging tasks, namely, SaferyReacher.v4, SafetyWalker-v4, SaferyyHummoodStanduup v4.
Safetyfleachetsp
Sefety/leachee-ot
SufetyfBeachense
-92
2500
ESPO Dhard
ESPO (0ung
-102
Avenage Epiodee
2000
Trining Time Jmte)
Averggetpoohh
1500
Coeti Inst-40
152
1000
30-
-232
500
16
222
Sumple Steps
Number of Sample Steps
6
ESPOrOUrs CATO
PCAPO
(a)
(b)
(c)
SafetyWaa kerzd V4
Safetyybiikerdd wi
SafetyWaikerdd vi
Ave Ieergge Episote Cost
2SP0 10url
71
cinng
20
Avarage Epizads
2200
200
20
Number Sample Steps
Number of Sample Steps
ESPOEOUnn)
C&PO
PCAPO
(d)
Figure 2: Compare our algorithm (ESPO) with PCRPO [30] and CRPO [53] on the Safety-MuloCo benchmark. Our algorithm consistently and remarkably outperforms the SOTA baseline across multi- ple performance metrics, including reward maximization, safety assurance, and learning efficiency.
Algorithm
ESPO (Ours)
CRPO
PCRPO
Task
SaferyReacher-v4
5.7 M
8 8M
M
Safety Walker-v4 SafetyHumanoidSeende
6.2 M
8 M
8 M
5.1 M
8 M
8 M
Table Comparison of sampling steps with primal-based methods (The lower, the better). denotes one million.
In the experiments conducted on the SafetyReucher-v4 task, as depicted in Figures 2(a)-(c), our method demonstrates superior performance compared to SOTA primal baselines, CRPO and PCRPO. For instance, our method achieves better reward performance than CRPO and PCRPO. Another notable aspect of ESPO's s performance is its training efficiency, which largely attributed to sample manipulation. Specifically, depicted in Table I, while CRPO and PCRPO utilize 8 million
/nctpe://gichub..cm// cwt.oou-ocouuuoe- www.//KK-AALiggmmettommntttttei 'nttps://githur wow/Safeeleeeeeeee
7

---

samples for the SaferyReacher-v4 task, our method requires only 5.7 million samples for the same task. Crucially, our method improves reward and efficiency performance without sacrificing safety. However, CRPO and PCRPO are struggling to ensure safety during policy learning. Ensuring safety pivotal aspect of RL in safety-critical environments. The experiment results indicate that our method's ability to balance safety with other performance metrics is a significant improvement. As illustrated in Figures (()),(f), our comparison experiments on the challenging Safety Walker-v4 task, yielding findings consistent with those observed in SafetyReacher-vt tasks. Due to space limits, additional experiments on Safery/lumanoidSuuuluuy are postponed to Appendix D.
5.2 Experiments of Comparison with Hlimal-Dual-Base Methods
The Omnisafe Benchmark is a popular platform for evaluating the performance of safe RI. algorithms. To further examine the effectivvneess of our method, we have implemented our algorithm within the Omnisafe framework and conducted an extensive series of experiments compared with SOTA primal-dual-basee baselines, e.g., PPOLag [32], CUP [54] and PCPO [56], focusing mainly on challenging tasks such as SafetyHopperVelociyy
and SaferyAntVelocity-vz
Saretyllspervelccctyv
Satetylloppeeeeeccyy
Satetylooperveeccyyy
Averaqe Eqiaada Reward
2200
Aeerage Episode Cost
126
PCPO
315
313
1300
46-
Co-
Limbe22
Tainting Tome
500
40-
:50
155
FCPO
20
CUP
PPOLag
Number of Sample Stops
Number of Sample Stops
(a)
SafetyArtKeloctyyaa
SafesyAntteelschyuu
Average Reward
2502
Epicods Cost
2.2
ESPO Ooril
inaiiing Timms (miin)
250
315
312
2002
APOLag
Cosl Inchon
1902
2.3
502
10h0
O.P
-b02
Sample Steps
1.5
2.0
02
0.6
Sample Steps
2.6
1.0
21P01Ourst PCPO
CUP
(d)
(e)
(I)
Figure 3: Compare our algorithm (ESPO) with PCPO [56], CUP [54] and PPOLag [32] on the Omnisafe benchmark. Our algorithm performs significantly better than the SOTA baselines regarding reward, safety, and efficiency performance.
Algorihm
Task
ESPO (Ours)
PCPO
CUP
PPOLag
MadetyHoppeeVellcityyy SaferyAntVelociieex
7.3
10
10
10M
7.6
10
1
Table 2: Comparison of sampling steps with primal-dee dual based methods (The lower, the better). M denotes one million samples.
The efficacy of our algorithm, ESPO, is demonstrated in Figures 3(a)-(c), where it is benchmarked against SOTA baselines on the peteey/toopeeellciyyuve tasks. Firstly, ESPO is remarkably able to achieve better reward performance than the SOTA primal-dual-basee aspect of algorithm is its capability ensure safety. It is particularly significant considering that some of the compared baselines, such as CUP [54] and PPOLag [32], struggle to maintain safety within the same task parameters. Thirdly, an outstanding feature of ESPO is its efficiency, as evidenced by approximately half the training time required compared to the SOTA baselines like CUP and PPOLag. This efficiency in training time demonstrates ESPO's s practicality for use in various applications, especially where computational resources and time are constraints. Moreover, while PCPO [56] manages to ensure safety, its reward performance inferior to ESPO's. PCPO also requires more training time than ESPO, underscoring our algorithm's 's reward, safety performance, and training efficiency advantages. Particularly, as illustrated Table 2, across the entire training period, all the benchmark baselines, including PCPO, CUP, and PPOLag, utilized 10 million samples for tasks on
baselines. Secondly, a critical
In contrast, our method required only 7.3 million samples for the SafetyHooppeVelocittyy task. The trends observed in the performance of our algorithm on the

---

SaferyHopperVelociiyvv task are similarly reflected in the results presented in Figures ((d)-(f, about the SaferyAntVelociny-v task. These findings further prove the effectiveness of ESPO in various tasks. Note that the reduction in samples may not equate to corresponding reduction in training time, as this can vary depending on the characteristies of the benchmarks and the algorithms applied to different tasks. Factors such as the action space of the task and the settings of parallel processing supported by the benchmark can influence the overall training time.
These results on Omnisafe tasks further highlight the strengths of ESPO in improving reward performance with safety assurance while maintaining greater efficiency training. The ability of ESPO validates its potential as an effective solution for further exploration and application in real-world environments.
5.3 Ablation Experiments
We conducted ablation studies focusing various cost limits, sample sizes, learning rates, gradient weights, and update styles to further assess our method's effectiveness. These studies are crucial for gaining deeper insights into method, highlighting its strengths, and identifying potential areas for improvement. Through this evaluation, we aim to demonstrate the adaptability of our method, confirming its applicability and efficacy across broad spectrum of safe RL scenarios. Due space limits, details of the ablation studies are provided in Appendix C.
6 Conclusion
In the study, we improved the efficiency of safe RL through a three-mode optimization scheme employing sample manipulation. We provide an in-depth theoretical analysis of convergence, stability, and sample complexity. These theoretical insights inform practical algorithm for safety-critical control. Extensive experiments on two major benchmarks, Safety- Safety-MuJoCo and Omnisafe, indicate that our method not only surpasses the SOTA baselines in terms of efficiency but also achieves higher reward performance while maintaining safety. Moving forward, we plan to assess our method's capabilities in real world control applications further expand its influential reach into safety-critical domains. Impact and limitation statements are provided in Appendix E.
Acknowledgement
The work of L. Shi is supported in part by the Resnick Institute and Computing, Data, and Society Postdoctoral Fellowship California Institute of Technology. The work A. Wierman is supported in part from CNS-21 14, (P3-2136197, CNS-2106403, and NGSDI-2105648. M. Jin acknowledges The work of S. Gu supported by funds from the Prof.
support from NSF Spanos' Andrew S. Grove Endowed Chair.
References
[1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
[2] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. Optimality and ap- proximation with policy gradient methods in Markov decision processes. arXiv preprint arXiv: 2019.
[3] Eitan Altman. Constrained Markov Decision Processes, volume 7. CRC Press, 1999.
[4] Florian Beiser, Brendan Keith, Simon Urbainczyk, and Barbara Wohlmuth. Adaptive sampling strategies for risk-averse stochastic optimization with constraints. IMA Journal of Numerical Analysis, 33(6):372)33765. 2023.
[5] Albert S Berahas, Liyuan Cao, and Katya Scheinberg. Global convergence rate analysis of generic line search algorithm with noise. SIAM Journal on Optimization, 31(2):1499-1518 2021.

---

[6] Felix Berkenkamp. Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems, 30, 2017.
[7] Dimitri Bertsekas, Angelia Nedic, and Asuman Ozdaglar. Convex analysis and optimization, volume 1. Athena Scientific, 2003.
[8] Raghu Bollapragada, Richard Byrd, and Jorge Nocedal. Adaptive sampling strategies for stochastic optimization. SIAM Journal on Optimization, 28(4):3312-3343, 2018.
(9] Raghu Bollapragada, Cem Karamanli, and Stefan M Wild. Derivative-free optimization via adaptive sampling strategies. arXiv preprint 1rXiv:2404.11883, 2024.
[10] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60(2):223-31 2018.
[11] Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365(6456))8885-899)) 2019.
[12] Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning robotics: From learning- control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5:41 1-44,, 2022.
[13] Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127-155, 2012.
[14) Richard G Carter. On the global convergence of trust region algorithms using inexact gradient information, SIAM Journal on Numerical Analysis, 28(1):251-28) 225, 1991.
[15] Coralia Cartis and Katya Scheinberg. Global convergence rate analysis of unconstrained optimization methods based on probabilistic models. Mathematical Programming, 169:337- 375, 2018
[16] Hongyi Chen and Changliu Liu. Safe and sample-efficient reinforcement learning for clustered dynamic environments. IEEE Control Systems Letters, 6:1928-1955 1933, 2021.
[17] Xin Chen, Guannan Qu, Yujie Tang, Steven Low, and Na Li. Reinforcement learning for selective key applications power systems: Recent advances and future challenges. IEEE Transactions on Smart Grid, 13(4):2935-2958, 2022.
[18) Yinlam Chow, Ofrr Nachum, Edgar Duenez-Guzman and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. Advances in neural information processing systems, 31, 2018.
[19] Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapumov- based safe policy optimization for continuous control. arXiv preprint arXiv: 10031, 2019.
[20] Frank E Curtis and Katya Scheinberg. Adaptive stochastic optimization: A framework for analyzing stochastic optimization algorithms. IEEE Signal Processing Magazine, 37(():32 42, 2020.
[21] Floris Den Hengst, Vincent François-Laves Mark Hoogendoomn, and Frank van Harmelen. Planning for potential: efficient safe reinforcement learning. Machine Learning, 1 111(6))2255- 2274, 2022.
[22] Dongsheng Ding, Xiaohan Wei, Zhuoran Yang. Zhaoran Wang, and Mihailo Jovanovic, Provably efficient safe exploration via primal-dual policy optimization. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 3304-3312. PMLR, 13-15 Apr 2021.
[23] Dongsheng Ding, Kaiqing Zhang, Jiali Duan, Tamer Başar, and Mihailo R Jovanović. Conver- gence and sample complexity of natural policy gradient primal-dual methods for constrained mdps. arXiv preprint arXiv:2206.02346 2022.
10

---

[24] Yuhao Ding and Javad Lavaei. Provably efficient primal-dual reinforcement learning for cmdps with non-stationary objectives and constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7396-7404, 2023..
[25] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pages 1329-1338. PMLR, 2016.
[26] Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochaaee methods for data fitting. SIAM Journal on Scientific Computing, (1880))138--11000 2012.
[27] Milan Ganai, Zheng Gong, Chenning Yu, Sylvia Herbert, and Sicun Gao. Iterative reachability estimation for safe reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[28] Zhan Gao, Alec Koppel, and Alejandro Ribeiro. Balancing rates and variance via adaptive batch-size for stochastic optimization problems. IEEE Transactions on Signal Processing. 70:3693-3708, 2022.
[29] Fangda Gu, He Yin, Laurent El Ghaoui, Murat Arcak, Peter Seiler, and Ming Jin. Recurrent neural network controllers synthesis with stability guarantees for partially observed systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 5385-5394, 2022.
[30] Shangding Gu, Bilgehan Sel, Yuhao Ding, Lu Wang, Qingwei Lin, Ming Jin, and Alois Knoll. Balance reward and safety optimization for safe reinforcement learning: A perspective of gradient manipulation. In AAAI, 2024.
[31] Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang. and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 10330, 2022.
[32] Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang. Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. arXiv preprint MeXii:2305.09300 2023.
[33] Ming Jin and Javad Lavaci. Stability-cerüfiedd reinforcement learning: A control-theoretic perspective. IEEE Access, 8:229086-229100 2020.
[34] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proc. International Conference on Machine Learning (ICML), volume 2, pages 267-274, 2002.
[35] Dohyeong Kim, Kyungjae Lee, and Songhwai Oh. Trust region-based safe distributional reinforcement learning for multiple constraints. Advances in neural information processing systems, 36, 2024.
[36] Dohyeong Kim and Songhwai Oh. Efficient off ofolpc safe reinforcement learning using trust region conditional value at risk. IEEE Robotics and Automation Letters, 733)7644-7651, 2022.
[37] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878-18890, 2021.
[38] Puze Liu, Haitham Bou-Ammar, Jan Peters, and Davide Tateo. Safe reinforcement learning on the constraint manifold: Theory and applications. arXiv preprint arXiv:2404. 09080, 2024.
[39] Zuxin Liu, Zijian Guo. Zhepeng Cen, Huan Zhang, Jie Tan, Bo Li, and Ding Zhao. On the robustness of safe reinforcement learning under observational perturbations. In The Eleventh International Conference on Learning Representations, 2022.
[40] Debabrata Mahapatra and Vaibhav Rajan. Multi-task learning with user preferences: Gradient descent with controlled ascent in pareto optimization. In International Conference on Machine Learning, pages 6597-6607. PMLR, 2020.

---

[41] Rohan Mitta, Hosein Hasanbeig, Jun Wang, Daniel Kroening, Yiannis Kantaros, and Alessandro Abate. Safeguarded progress in reinforcement learning: Safe bayesian exploration for control policy synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21412-21419, 2024.
[42] Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems, 29, 2016.
[43] Anitha Murugesan, Mohammad Moghadamfalahi, and Arunabh Chattopadhyay. Formal meth- ods assisted training of safe reinforcement learning agents. In NASA Formal Methods: 11th International Symposium, NFM 2019, Houston, TX, USA, May 7-9, 2019, Proceedings 11,
pages 333-340. Springer, 2019.
[44] Raghu Pasupathy, Peter Glynn, Soumyadip Ghosh, and Fatemeh Hashemi. On sampling rates in dimulation-basee recursions. SIAM Journal on Optimization, 28(1):45-73, 2018.
[45] Sara Shashaani, Fatemeh Hashemi, and Raghu Pasupathy. Astro-df: A class of adaptive sampling trust-region algorithms for derivative-free stochastic optimization. SIAM Journal on Optimization, 28(4):3 145-3176, 2018.
[46] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484-48) 2016.
[47] Dylan Z Slack, Yinlam Chow, Bo Dai, and Nevan Wichers. Safer: Data-efficient and safe reinforcement learning via skill acquisition. Decision Awareness Reinforcement Learning Workshop at ICML 2022, 2022.
[48] Yanan Sui, Alkis Gotovos, Joel Burdick, and Andreas Krause. Safe exploration for optimization with gaussian processes. In International conference machine learning, pages 997-1005. PMLR, 2015.
[49] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[50] Daniel Tabas and Baosen Zhang. Computationally efficient safe reinforcement learning for power systems. 2022 American Control Conference (ACC), pages 3303-3310. IEEE, 2022.
[51] Tim Tiz-Kit Lau, Han Liu, and Mladen Kolar. Adadagrad: Adaptive batch size schemes for adaptive gradient methods. arXiv e-prints, pages arXiv-2402, 2024.
[52] Yuchen Xie, Raghu Bollapragada, Richard Byrd, and Jorge Nocedal. Constrained and composite optimization via adaptive sampling methods. IMA Journal of Numerical Analysis, 44(2):680- 709, 2024.
[53] Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, pages 11480-1 1491. PMLR, 2021.
[54] Long Yang. Jiaming Ji, Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization. Advances in Neural Information Processing Systems, 35:9111-9122 2022.
[55] Qisong Yang, Thiago D Simão, Simon Tindemans, and Matthijs TJ Spaan. Wesae: Worst-case soft actor critic for safety-constrained reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10639-10646, 2021.
[56] Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Projection-basee constrained policy optimization. In International Conference on Learning Representations, 2020.
12

---

[57] Ming Yu. Zhuoran Yang, Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[58] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning, Advances in Neural Information Processing Systems, 33:5824-588 5836, 2020.
[59] Weiye Zhao, Tairan He, Rui Chen, Tianhao Wei, and Changliu Liu. State-wise safe reinforce- ment learning: survey. In Proceedings of the Thirty-Scond International Joint Conference Artificial Intelligence, pages 6814-6822, 2023.
[60] Yong Zhao, Wang Chen, and Xinmin Yang. Adaptive sampling stochastic multigradient algorithm for stochastic multiobjective optimization. Journal of Optimization Theory and Applications, (00(1))215-244) 2024.
13

---

Appendix
A Proof of the theoretical analysis
Inspired by [53], the theoretical results in this section are established by tailoring to our algorithm ESPO to ensure the key recursion relation still hold for the proposed complex update rules differennt update rules in three different modes.
A.1 Preliminaries
To proceed, we first introduce some notations and invoke several key facts and results that have been derived by prior arts.
Notation. We recall and introduce some useful notation throughout this section.
Qi Qf: this two function represent the policy evaluation results from Algoriathm 1, namely, the estimates of true Q-functions , Qt
t. the learning rate of the NPG update rule in Algoriathm 1.
Bott Beonf, we denote the set of iterations when Algorithm executes (4) (resp. (3)) as gost (resp. Bgcon)).
(xy, æf): when the iteration & g (no conflict between the gradients of reward and cost objectives), =t (resp. x55) represents the weight of the gradient w.r.t. the reward objective (resp. the cost function). So is easily verified that S x7, og and xy + x =
(vi wf): when the iteration & E Beonf (the gradients of reward and cost objectives are conflict with each other), y/ (resp. yf) represents the weight of the gradient w.r.t. the reward objective (resp. the cost function). So it is easily verified that Vi yi 2 O.
Umax: without loss of generality, we assume r(s, a) E [0, Umax] and cs a) E [0, Umax] for all 1 Sis n..
h+,, lt,hh h- for simplicity, we let h hit,t, J,t,11,, ,,; - for all SLST.
Lemma A.1 (Performance difference lemma [34]) ). For any policies n, and initial distribution one has
1 Vi [c,r]: VT - VT' (p) 1-7 Band, [ (s, (s,a))
(11)
where (p) and dp denote the accumulated reward (cost) function and state-action visitation distribution under policy π when the initial state distribution p. Here, AT (s. Q (s.a) V. (s) the advantage function of policy π over state-action pair (8, a).
Lemma A.2. Considering the approximated NPG update rule and Algorithm I in the tabular setting. the NPG update in four possible diverse modes take the form:
74 (c.a)
W++1 = Wt 1- and
Tmita Twiti(a|s) (als) Tw., (als)
exp
(1-9)
ift E B,
(((e)
Wh+1 = W: + (a
exp
and T (als) = Tw, (als)
ift E Buo
1-7
2 (8)
((uE (f(aaa))
W1+1 = We +
n, Q +1
1 1-7
and Twe (als) = Twr, (a|s)
exp
2:.0i2 (x)
(1-7)
ift E Boonf
W441 W. + I-
Qi and Twees (als) Twn, (als)
exp
(1 (1-))
ift E B. (12)
(7(s)
14

---

where
(+)) = Σ
Twn (a)s) exp
1 1-7
aEd
(x) +2
(s) = Σ
Two (a]s) exp
(1 71
aed
(
(s)s)
Σ
Tw, (als) exp
1-7
aEA
n (
Z, (s)
Tw, (a|s) exp
(13)
(1-7)
acA
Proof. The first line of (12) has been verified by [Lemma 5.6. [2]]. Following the same proof pipeline for the update rules of Algorithm different modes completes the proof.
A.2 Key lemmas
The proof of Theorem 4.1 heavily count several key lemmas in the following.
First, introduce the performance improvement bound for the update rules of Algorithm 1 in different modes, which is a fundamental result for its convergence; the proof is postponed to Appendix A.
Lemma A.3 (Performance improvement bound for approximated NPG). Consider any initial state distribution p and the iterate Tw, generated by Algorithm at time step L. One has when iteration L E Br:
Vr TWEl (p) - V
(14)
7 Eomp
log (s)
17
W (s) +
Tw. (a|s) (T,s,a) -Q," Twr (s,
17
1 1-9
aEA
Bandd
Tey (a|s) (8,
Q7 Twe (s,a))
aEA
Band
Twiss (a|s) (7 - O, Tur (s, a) = diff.-
(15)
aEA
Similarly, we have
vt E Be
Ve Twr+1 (p) - -. (p) 2 diff,
(16)
and then
xt Vr (i (p) V.= (p)
xz Ve -w (p) V.T (0)
2 2 diff; + x;diff;
ift Bot
vi V (p) V,==a( (p) 1/ w (p) Vo (0) ya diff; of if E geonf
(17)
Armed with above lemma, now we can control the performance gap between the current poliy Twx and the optimal policy n² in the following lemma; the proof is postponed to Appendix A.6.2. Lemma A.4 (Suboptimality gap bound for update rules of Algorithm 1), Consider the approximated
NPG updates in (12). When iteration t B., denoting the visitation distribution under the optimal policy as d², we have
V. (p)
V, V, (p)
Egand- (DKL ( DKa(TT (m (1 + An|S||Alus (1 --7)
3(1 + 17mmax)
11
-
1)
(1 7)2
gap?
(18)
15

---

Similarly, we have
Vt E Bc: Ve (p) - Ve v (0) S gap..
(19)
In addition, for other iterations: ift E B we have
z) V (0) V1, (0)) + 2 Ve (p) Ve Ve (p)
$ Bignd (DxL(T [[TTm]]))== (Tx((Tr(((trtt)) 2 |8|AA|
||
(1 (1-
+ 3(1 + 1/Uinax))
(ot 11, (s. ()(ss,))12 + +z
11 ()(s (20)
(1 - 77)2
and ift Beont we have
yi VT (p) (p)) + of Ve" (p) Vo-ui -vo=((p))
27po (1% + 5/)|51144 ||A]
$ Bond- (DeL(TT (TT]] ) -xx((T
[[Twe())) ([T||[
17
(1 (1-9) 7)3
+ 3(1 + 1)Umax)
[o 1T (s, (2(s, + 2 11@e m (s, - (s(ss)))
(21)
(1 7)
Now we are ready to develop a key lemma that is associated with the expectation of the performance gap directly. The proof is provided in Appendix A.6.3
Lemma A.S. In the tabular setting, consider any < < and suppose the iterations of policy evaluation obey Tpi
Ō T log( (811A]
With probability at least - 6, applying Algorithm I leads
Σ (V. (p) Vrr a (p)) + 1 Σ
(V, (p) V.* (p)) + Σ
(V, " (p) -V, (p))
IEB,
BE
+ nhh + |Bel n/h
Ft ryh
Ur
1EB
LE Besan
+ 211a 2
S |/A
3 + 170mmax)
< Eo DKL(T=1|TT
maxx
(T- 1Bsoof )) + Σ
(ui + +i)
7)3
(1 )
Boouf
(22)
S Ex ([TT]))) + u u (S] (AAT
31(1 + 7) V SAAA
(23)
(1 - 733
(1 7)1.5
where
Q - - Qi + Σ Q m - 2
Epi =
EEB,
1EB
+
(xi11Q7 0i112 + x/il 1t -21l2)
E Bre
(1 Qi1 + v2l Qi1l2)
(24)
EE
Finally, we introduce the following lemma which indicates the number of iterations that optimize the reward objective is in the order of I as long as h+ and h are chosen properly. The proof provided in Appendix A.6.4
Lemma A.6 (The frequency of optimizing reward objective). Consider any 8 < 1 and h- Suppose
1 nh >ph+T2 T Eardd ( (TT)) + 2 1 3n(1 + 170max)
VISTAAT
(25)
(1 7)3
(1 1.5
then with probability least 1 - 8, the following fact holds
16

---

1. B, U Buoft * 0.
2. Either of the following claims holds:
(a) [B, U Byottl 2 T/2:
(b) The weighted performance gap is non-positive:
Σ [V, (p) - V₂ w (p)) + Σ
(V (p) - -V;" (p))
EEB
sct
Σ VV (p) Vr (p)) 5.
(26)
B soft
A.3
Proof of Theorem 4.1
Now are ready to provide the proof for Theorem 4.1.
Recall the goal to prove
SA (1-7)
(p) E (p)] ②ō
(V
(27)
where the expectation is taken with respect weighted average over all {Tw, higigr We still consider the modes when the policy evaluation results are accurate such that
Epi S -
(28)
which combined with Lemma A.5 yields
(0) - V (0)) +1 Σ
(V² (p) V,, (p)) + x=(VV=" (0) - VoT (0))
REB
AE Iseo
Σ VV (p) - - " (p)) + VV= (p) Ve.T" (p))
+
+E Essomi
+ 1/4 Bel - 1//2 Σ
xe - 11h1 Σ si
BE B
Issent
2m²² 2 Umax
|AT
31 + mmmax) ) / SAAT
S Bn DKL ( ||
+
(29)
7}-
(1 11-3- 1..
The probability distribution associated with the expectation. distribution) to be proportion to
Here, we let the weighs (probability
if E B,
x if / E Bno soft De if E B Bronf
(30)
if E Be,
which will be normalized by
Ty weighted =
|B, +
(31)
LE
Boon
Then we introduce an important fact for yf and yf. Recall that when t E Beorf keeping the weights xff and xt as the same as the mode E goo for the reward and cost, the gradient is constructed
g,
g. gr
gi x
ge
ge
+
x
Ig
gr
cos AC
ge
cos 0r
gr
xx
1
+
gr + x
1
ge,
(32)
lgr
ge
17

---

which indicates
Vt Boont
Vi 2 1+
cos f+
gel
x
and
yf =1 (1+
cos 0t
gr
2 z/
gr
(33)
since cos of 2 0 as E sontt The above fact directly gives that letting at 2 1/2
T
T 2 weighted 2 4'
If [B, U BLott| 2
(34)
The reward objective. We first consider the performance gap w.r.t. the reward. Armed with above facts, we can see if (26) holds, with the weights in (30) then we directly have
V (0) - E (0) S.
(35)
Otherwise, applying Lemma A.6 gives
(V7 (p) (p1))
Σ (V. (p) V, (p)) +n Σ
(V (p) - V (p))
11
1E
rEB
+ n Σ (V, (p) - (p)
S Esnd' DKL (TL(TT|| [1 + |S|AA[
3n(1 + m)x VISIAAT
(36)
7)3
(1 (1-7)1.5
which indicates
V (p) (p) S 2//
(1 (1- (.-5)1.5TT (Renw DKI. (T + 40 + 6Umax)
(37)
Here, the last inequality hols by letting the learning rate rj = (1- (1-7))"/
Constraint violation. Now we move on to the cost objective. Taking the probability distribution of the expectation in (30) as well, we have
EIV. (p)] - b
1
Σ IEB,
Ve (p) + Σ
V (p)
Σ
V ( - b
Tr. wnighted
I. Bro
BE Broont
V Ture (p) -b) + Σ 3r ( (0) (0)-))) -b) 1
V Tul (p) - -b)+ +
Σ
Fr.
T weighted
IEB,
DE Bio
BEB
1 weighted
Tise (p) VeTue (p)
x
V c
(p) Vc 1 (p)
T
EEB
Σ
vr V.tu (p) v. (p)
TE w0Rt
< htt
Σ
Qtte Di + Σ
x
Q
Tue - -Q,
+ Σ
v Q - 0i
TT
weighted
EEB.
Ee
LE Boond
< +
4
Q Two - Qi + Σ
Qr
Tue
+ Σ
(38)
T
BEB
LE koft
EE gooon
18

---

where the last inequality holds by (34). Finally, also considering the mode when the policy evaluation error in (28), we have
Σ EEB,
2o T Qi
+
Σ
2 Q0
Σ ve -0-00010
S Epii S (1--))
B
ME 3oft
(39)
Then without loss of generality, taking the tolerance level h 0 and
ht 2/|S/A
(Eond DKL( (TTT) + 4u + 60max)
(40)
(1 (1-))1 1.55 /T
complete the proof by showing
B (0) - - bss 1+t + T
Σ
Qc The Qi + Σ
Qc Tue Qi + Σ
3 QT Qi
LEB,
LE saft
AE Bronn
(41)
S 2 |
4 4v88/
(1 (1-27 71-5))1533 (Resned DEL (TT" (1 + 4u + 600max)
+ (1 (1-7)1.5VT
(42)
A.4 Proof of proposition 4.2
We consider the ideal mode when the number of iterations of policy evaluation Tμ + oo such that the ground truth cost function V.=" V
First, we will focus on verifying the fact in (9a). Recall that there exists an iteration in < T such that consider two different modes separately.
lin B, U Buott So for the next step Iin
When tin E B, In this mode, we directly have
Ve
Jwitin (p) V S b ht
(43)
Then we know that for the next step = lin + 1,
Ve T (0) S Ve Kattn (0) + n/ V. Nwtn (p)ll2 S b +
20
S bb+h"*
(44)
1-7
where the penultimate inequality holds by the bound of the policy gradient established in [53, Lemma 5], and the last inequality holds by when the learning rate 17 is small enough such that
20max]] 2/18
(Eond- DKI.(T") (T' + 4v muxx 6Umax) ht.
(45)
(1 (1-7)L55T
The observation in (44) shows that the next time step tin E B, U Booft.
When li Boot. One has
Tuttin (p) = tn S b Ve
(46)
Then we can adaptively choose the weights for the reward and cost function xf,, of. Invoking Lemma A.3, we have
x
Twe+1 (p) - V,"Tar (p) + x Vo.== (p) - Ve Jwrt1 (p)
2
0 if E B
Twit3 (p) - w/ (p) + y, =. (p) "w (p)
20 if
Boonf E soft
(47)
Then observing that when t = tim in the mode with x.f
= 1, we have
Ve Twtn (p)
V.T (p)
2 0 if E goft
(48)
Awinn (p) Vea Tue (p)
2 0 if E Boont
19

---

which implies that
Vo-u (p) S Ve
wiin (p) bb+
(49)
So we have the next time step 1= in + E U, Bsoft- This implies that long 2z chosen properly ensuring (48) holds, achieve tin + E B, U Boot-
Summing up the two modes and applying them recursively, complete the proof of (9a).
Finally, to verify (9b), we suppose ESPO and CRPO are initialized at same point. Then observing that ESPO and CRPO execute the same update rule until the iteration lin E B, U B Then applying (9a), we know that
BB U Boott] T tin-
(50)
While CRPO may has some iterations later such that falls into Be. So we have the number of iterations when CRPO update according the reward objective BCRPO < T -in- We complete the proof.
A.5 Proof of proposition 4.3
Recall the goal of the algorithm is to achieve
Ve (p) EV, (p) s1. EIV (p) V. (p) S
(51)
with as few samples as possible
We start from considering V." E (p)] E1 We observe that if (26) holds, taking the expectation w.r.. the probability distribution in (30), we directly have
(p) - E|V (p)) < <0S
(52)
Otherwise, applying Lemma A.6 and (22) gives
(p) - EVV, (p)]
Σ (V, (p) v, vp))
Σ
*V (p) -V, (p))+
Σ (V, (p) V," (p)
IEB,
IEB
1 < Expar DKL. (Ir (T=T) +
211u2 max
SHATT 3(1 + 17Umax)
(1 - 7)² Epi-
(53)
3)
The first two terms are independent the sample size. So we focus on control 3(1+))mux) the goal, namely, we need to achieve
(1->)² Epl to meet
Epi Σ |1eT
@ill+
(xi11QT - Qill + =fllQ. Q71l2)
tEB,
BE BE
+ Σ
w - - vfll Qi1l2) s S
(54)
for some E E1.
To continue, without loss of generality, we let x. 1, 29 0, and |B,| 0 (in this mode, the sampling approach is fixed), we have
Σ ||QT
Qil12
Σ
Vl -2
Epi
Bn mh
AG Boont
cos 0r
gall
|1
Qill2
Σ
1 +
|1 T.
Q
gr
LE solt
= Σ
5, +
Σ
cos 0c
ga
8t
(55)
gr
EE sR
20

---

where the penultimate inequality holds by the relation between yi, 27 in (33), and the last inequality follows from denoting ||Qr - -illz= or.
Now ready to show the advantages of using different batch size for different modes when t E goft or E Bgcon soft We make the following assumption about the relation between 6, and sample size (the number of iterations for the policy evaluation of Algorithm 1), which is qualitatively consistent with the policy evaluation bound in [53, Lemma 2).
Assumption A.7. Suppose for any t Booft- when the sample size varies around some basic size, the possible feasible & is in the range such that & = Y as$ such that Y is some small constant and of is the sample size used for policy evaluation at /-th iteration.
With the above assumption in hand, (55) can be written as
cos 0r Bal
'pi
Σ
Y
B + as
Σ
1+
(Y -asss = E
(56)
gg-
DE 2
BE
If there is no adaptive sampling, then we have sp. = sp for any t, t' E Booft which leads to the total number of samples as
E |Boft
Nall Sbatch Boott|
(57)
soft
Σ
(1+
cos 0!., (o.lllll
lgg.
where Sbatch is the number of iterations in this mode.
Our proposed algorithm ESPO will increase the sample size when t Beont and decrease the sample size when t soft-
So long there exists at least one iteration t B with
(1
cos cost".ll 1 ||xtel
> we can increase the stt by Sextra <
Sbatch (1
cos o [|o.l] and decrease
g
l|g.
any of by Sextra
g.ll at time & E B Consequently, the total number of samples are smaller and (56) still holds. So we complete the proof.
(1 +
|
A.6 Proof of auxiliary results
A.6.1 Proof of Lemma A.3
To begin with, note that the first two statements (15) and (16) has already been established in [53, Lemma 6]. So the remainder of the proof will focus on (17), which recall here
x
Vi," (e (p) - V,"T v,T=((p) (p) + x (Vo "=1+1 (p) - V.Tw: (p) 2 xfdiff; + xfdiff;
if
t E B
Vr -== (p) - V/= (p) + V/ Ve Tms+1 (p) - v. (p) 2 yf diff; + yfdiff;
t E Boonf
(58)
Towards this, the left hand side of the first line can be written out as
xf
Vr Fwi
(p)
V.Tu (0)
1 Ve Twei (p) Ve (p)
1 Band
Σ
T1 (als) u1 (8, a) + x Eand, Σ
Twiss (als)A T. (s,
aEd
aEd
E Eand, Σ aEA
Twiss (a|s) (2, "M () + " (s)
1
E Eond, (IiV," (s)
Exwdp Σ
Twizi (a|s) [ (s, t ()))))))
1 ?
aEA
E Eand, (IiV," (s) + VVV.
21

---

+ 1 1-7
E Bondd Σ
Twres (a|s) [at (s, a) - a)) + t (Qe" (s,a) [))]
?
aEA
(1)
Bynd
Twiet (als) log
Twres Tw. Tw.(als) (als)
1-7 Badd (x/ (s) + =iVo
17
aEd
1
+
Eand
E
Twets (als) [ot (Qr Fue (s, - ()) + of (Q (s, (i(s,a))]
1 ?
aEA
1
1 1-- Band, (,:V - (8) + xiVo" (s))
- Byndy (TI.(TTTLT (imetallTarn + - Band, log (x
17
9
E Egndp
Twets
(a|s) [x] (QF =M (s. a) a)) + . (Q (s, a) Qi a))
1
aEA
(59)
where (i) follows from the update rule in Lemma A.2. To continue, invoking the basic fact (xL 2 0, we have
of Vr mwe+s (p) - -vTT((()) + x (Ve"'<((p) Ve (o))
Bg Bs
log log22ice1 (s)
"1 Bond, (xiV, (5) + =iVe (s))
1/
1-7
+
17
Two (a|s)
Em - -(((
a) + ワ Q r (s,a) (i(s,a)]] )
1
acA
1
Exnd,
(s,a) Tw, (als) [u]]] |0, TM 1F - (i(s, a)) + क | TM (s,u) Qi(s, [)]]
1
aed
1 E
Eandp Σ
Twis1 (als) soy 10 ( - Qi(s, + + Q T (s,a) - as)
1 --1 γ
aEA
?
Esup
log (s) 1-7 Eand, (x m (s) + =2V.."1 (s))
17
+ 1
1)
Tase (a|s) [of]] 10 (s,a) - 01 a) + x Q k- (s,a) @i a]]l
aeA
1
Eandd
E
Ten Tw, (a|s) [x]]] |Q TH (x, - a)(, + TM as - Qi(s.a))
1
aEA
1 E Ex-dp
Twiti (als) 10 tM (s, - Qi(s, a) + 2 00 - (s, (s(s.
1
aEA
= =iidiff; + xidiff;
(60)
where the penultimate inequality holds by the fact p 2 1 - and the following claim which will be proved momentarily:
log Z, 2. (s)
1 1- Eand, (0 (s) + =iVe" (s))
")
Σ Tw, (a|s)
[a] Q7 (s,a) Q'(s, a) + x (T=(((s(a) Qi(s, a]]] 2 0. (61)
aEA
So the rest of the proof is to verify (61). To do so, applying the definition of 2, in (13), we observe that
log Z, an (s)
17
Eand, (ITV, (s) + viVe (6))
1-7
(x] a) + (F)((())))))) ))
-log
Σ Tw, (als) exp
(1-
22

---

1-7 Band, (x;V (s)
7 (2]
+ 2j ()))
2 Σ Tim (als))
1 1- Eard, (sāV;" (s) + x ziVe xiVe"((s)
1
(1-7)
aEA
Σ acA
Tan, (a|s)
7
[o] (Q)(s, - O. T (s)))) + o (Qi(s,a) - (s.((s))))) a]]]
1 1-7
+
Twe (a|s)
1 y (, " (s,a) + =iQ: " (s, a))
11
77 1-7 Band, (z ue (s) + x =iVo"'((8)
(s))
REA
Tw, (als)
17 [xt (Q)(s, Q² (s. a)) + o5 (Qi(s,a) - Q: [ss.))]]
(62)
1-7
acAA
which complete the proof of the first line of (17). The second line of (17) can be proved analogously.
A.6.2 Proof of Lemma A.4
First, the first two statements (65) and (68) have already been established in [53, Lemma 7J. So we focus on (17) throughout this subsection.
Consider the first line of (17), applying Lemma (A.1) and following the pipeline for (59) yields
of VF (p) Vitui (p) x (V=' (p) - Ve-'!()) (p)
Bond (DKI, (TT [Tm..)) - DKL(TT [[Twee))) + 1 11
Bonnd log (s)
1
1-7 Band (I.V (s) + =VVe (s))
1
1
Eond. Σ T" (als) [x] (Q TT (s - Q((5.a)) + x8 (Q (s. - a)))
+
aed
Eg Eynd (DeL (TT ([TT) - DKL ([T
17
+ E
1
log logzztic (s)
- " -i- Bandp (x/VT"" (s) + =iVe""
1)
+ 1
Tum (als) 10 (s,a) - Qi(s,)) a) + 1e (s, aill )
aEd
+
1
Eand.
E (als) [x] (s.) - a)( + 2 (Q (s,a) o))
aeA
(
Evaad (DKL (I=we) - DKL(T") (|FT/1((())))))
17
1
Vitnet(()) (p) - v, (p) + o ( (ve (p) -v.
ve- -V) [v)m
+
7-7
+
1
Eand,
E
Σ
Tw, (als) [ 12=tt (s.a)
- a)) + 2 t. (s) - - |
(1 - 7)2
aEA
1
+
Eard
Twess (a|s) | 10== (s. - Q7(s a) a)) Q: (s) ()(s,a)
(1 7)²
aEA
+ 1 E Eynd
Σn n (als) [x] (s;aa) - Qi(s, a)) + 1 (Qe TH (s,a) (s)s
aEd
2 21max
Eowd- (DKL (TT (T'I| - [TTe=))) + (1 7)2 |1 - well2
11
3
+ (1 71-
b 12 tu (s. - + +8 +2
- (s, - (0((s,
23

---

+ 2nyomaxx |S||AA
5 Ba (DKI,(T ([Tm)) ) - DKI. (TT [ |
(1 (1-7)
+ 3(1 + 17mma)))
Qi(s, 0)112 + + 11 (i(s.
(63)
(1 7)²
where (i) holds by applying Lemma A.3, the penultimate inequality holds by the Lipschitz property of V." (p) and Ve= (p), and the last inequality can be verified following the last line in the proof of [53, Lemma 7].
Similarly, we have
vl V (p) Vi (0)) + vf Ve (p) Vo-': (0)
$ Band (DKL. (T "TTT) (KL(TT (T + 2xpo
max 2xpomax(yf (,/ + S
(1-7)3
+ 3(1 + 1mmax)
[ot -i(s, (saa)) + x =i1l0t (s,a) - Qi(s, (s,,a=
(64)
(1 7)2
which complete the proof.
A.6.3 Proof of Lemma A.S
Invoking Lemma (A.4) for the four modes when Br, B t E Beonf and summing up them together for t = 1,2, ... T yields
Be and
(p)
Vr W! (p)) 7
x (p) - V (p)) + +'(VVV (p) - Ve Ve (0))
7
EEB
BE B
+
Σ
VVV wu (p) - V (p)) + VVV"' (p) - Vo_ we (p)) +n Σ (V.5 (p) - V²" (p))
Boond
EE
27j² 2 max SA|A (T BBson' ))
37(1 (1 + 7)0max
Eandd DKI (T (Tom)
Σ
(vt vi)
Epi
(1 - 7)3
(1 - 7)
Bronf soft
(65)
where Epi is defined in (24).
Then we consider several different modes separately:
When & Be: we have V > b b+ At, which indicates that
VoTue (p) V. (p)
Vi(e) Va (p) + VeT (p) - V. h+ IV THe (p) -V, 2 ha
=1Qa(Tmm)) - -Qi l--
(66)
when t E Boott V. > b - 1,, one has
Ve_ (p) V-d (p)
= =V - v=" (p) + v.== (p) - V22 - --V- (p) Vil 2 -h- - -1(Qc(TT.)) Qi1l2
(67)
Summing up the above modes and plugging them back to (65) leads to
(V (p)
V.TTe (p)) +1
Σ
(V; (p) V. (p)) (=)( +n Σ
(V (p) -V7 (p)) (p))
17
AEB
DE DE
+ nh
|Bel=
17h" Σ
nh
v
LE
soft
24

---

S BBnrd (T.. ([Tmo) + 211200 max (81|A| (1 - 7)3 (T
+ 3r + 171mmxx)
1Bson
1) +
Σ
(uf + vi)
Epi
(1 - -7)
AE Bsoon!
(68)
To continue, invoking (53, Lemma 2] leads to when the iterations of policy evaluation obey Tpi
Ō T log (iI|A
(1-7)19111 With probability least 1 8, we have for all 1 S st
(1 - I
10t TH - 0il12 S
2
T
(1 A)|i|A
11 1)A
and -Qi00 - 2V
(69)
T
T
Combining this fact with the definition in (24) directly leads to
Epi Σ
O T
Qill2 +
Σ || - - -illl=
Σ
(0ill - Qi + =111Q= Qi1l2)
EEB,
BEB
BEB
Σ
(vEl1 THE
Q7
vi 10 T.! - -Q1112
B
v(1 (
(70)
Plugging (70) back into (68) complete the proof:
( (p) Vr (p)) +1
Σ
x
(V, Tox (p) - V (p)) +n Σ
vV, (0) V,"" (p))
EEB
EE B
BE EEB
+ n/ |Bel n/
x n/h
Σ Vr
1EBoaa
211² 22 max (S||A|
31p(1 + 17Ummxx
S Egnd DEL(TT ((T
(T
1Bconf 1) + Σ (of + vt))
ppi
(1 73
(1 - (1-
LE EScon!
2n max S||LA|T
31 + 1/mmxx))
) VSSAT
< Bawd DKL(FT (Tr (TTo))
(71)
(1 733
(1 ---))11
since (vf + vf) 2.
A.6.4 Proof of Lemma A.6
The first claim is easily verified since if B, U Boott = 0, then [B&l = T. Applying Lemma A.5 gives
2n S||AA| 3n(1 + 7f0 )/ / /SS A/T
nh [Bl n < Eardr DKL (TT" (wo) )
(1 73
(1 1.5
(72)
which contradict with the assumption (25). So we have B, U Booft # 0.
Then the rest of the proof focus the second claim. Towards this, if
Σ (V,"' (p) TTe (p)) + Σ (V, (p) - V," THe (p))
Σ
(V, (p) - V3, (p)) S 00
tES
(73)
then the condition (b) holds. Otherwise, applying Lemma A.5 yields
n/ w/t11B.)
n/2"
Σ I n/ Σ 4
BEBm Aoft
re graet
2 max |
37(1 + 7mmmx)) //S/ S/SI
< Eyned (x ((TT) +
(74)
(1 (1-7)3
(1 1.5
25

---

Then if [B, U Bsotel < T/2, we have |Bel 2 and thus
n/ T
2 nh T 17
[Bll 1
2(T
|Bel)nt S Inh"" |Bel 17h
Σ
nh
Σ
EE Be
"/h" a |Bel 1//2 Σ
2 n/h" Σ y
EE Bron
2172 |S||A|T
3n(1 1 + 1)0 ) //ITA
Eg (
(75)
7)3
(1 7)2.5 1.5
which yields
nh+T S Bond DKL(T"I|T +
2 U |S||A| 71
31y(1 + 17Umaxx ) |
|
(76)
- 7-
that is contradict with the assumption (25).
26

---

B Practical Algorithm
Algorithm ESPO: Improving Efficiency of Safe Policy Optimization.
Inputs: initial policy with parameters Tayy positive slack value h [0, too), negative slack
value hT E (-oc, O). the cost value Ver (p) step t. the cost limit b, positive sample penalty 5+ E [0, +oo), negative sample penalty <- E (-1, 0), gradient angles rr,c, sample size X.
2:
for 1 O,..., T do
3:
if 1+ iteratively decreases then
4:
h,t -- l1t h /T
5:
end if
6:
if h2 iteratively increases then h T
7:
h +- h
8:
end if
9:
if G iteratively decreases then
10:
G +. - -
end if
12:
if GE iteratively increases then
13:
G + 4 - -//TT
end if
15:
if Va Fw: (p) > (ht + then
Adjust sample size X, with Equation (7).
17:
Update policy Tar. to ensure safety with Equation (2),
18:
else if (1, + 6) < Ves Twr (p) S. (ht + b) then
For gradients gr- and ges 0r,c I 90 then
20:
Adjust sample size X₁ with Equation (7).
Update the policy Tar. with Equation (3). else
22:
Adjust sample size X, with Equation (6).
24:
Update the policy Ta,, with Equation (4).
25:
end
26:
else if V==." (p) < (h, b) then
27:
Adjust sample size X, with Equation (7). Update policy Tw, to maximize reward V (p) with Equation (5).
29:
end if
30:
Policy evaluation under Tw, involves estimating the values of rewards and constraints. Sample pairs (sj,a, from the buffer B, according to the distribution Ta,, and compute the estimation ViT (p) and V. (p), where 8; represents the state and a represents the action, is is the index for the sampled pairs.
31:
32: end for
33: Outputs: Tey
27

---

SafetyVla
SuletyWWalkor
Eaicada Roowar
#200
Iverage Ephoade Cos
Croims-0
71
capn lest
sx 10
3200
30
Con Limie4)
2000
10
300
Epochs
Epochs
(a)
(b)
(c)
SatetyWWlkerZevv
SafetyWelker2divt
SaretyWakervd
Average Ephorde Howarz
5932
Average Eptende Cork
EiPt thart peasty L01. persily
crie
Doal Lambeed
2002-
50
5002
100
203
303
402
502
100
230
330
620
300
Triining Epochs
Training Epochs
(d)
(e)
Figure 4: Ablation experiments: Experiments of different cost limits and sample sizes.
C Ablation Experiments
To further evaluate the effectiveness of our method, we conduct a series of ablation experiments regarding different cost limits, different sample sizes, learning rates, gradient weights, and update style analysis. These ablation experiments are instrumental in providing a deeper insight into our method, shedding light on its strengths and potential areas for improvement. Through this rigorous evaluation, we aim to substantiate the adaptability of our method, ensuring its applicability and effectiveness in a wide range of safe RL scenarios.
Different Cost Limits: As depicted in Figures 4(a)-(c), we evaluate our method on the Safery - tasks under different cost limits, maintaining identical sample manipulation settings. Our method exhibits similar reward performance at cost limits of 30 and 40. This similarity in performance attributed our method's capacity dynamically adjust the sample size, a critical factor in optimizing for reward maximization while ensuring safety. Moreover, the training time for the task with a cost limit of 30 is 63 minutes, slightly longer than the 58 minutes required for the limit of 40. This observation can be explained by the increased challenge and larger conflict between reward and safety presented at the lower constraint limit of 30, necessitating a more significant number of samples for effective optimization. Notably, our method can ensure safety across these various constraint-limiter tasks and outperforms CRPO in reward performance and training efficiency.
Different Sample Sizes: As illustrated in Figures 4(d)-(f), we conduct an assessment of our method on the Safery Wd tasks, exploring different sample sizes while keeping the cost limit settings constant. In these experiments, we compare the outcomes of using sample sizes set at 1.2X and 0.5.X against 1.0XX and 0.5.X. Notably, both settings successfully ensured safety. On the one hand, the reward performance achieved with sample size of 1.2X and 0.5.X surpasses that of 1.0XX and 0.5X, indicating the effectiveness of larger sample size enhancing performance; on the other hand, the training time for the sample size of 1.2X and 0.5.X is recorded at 67 minutes, which is longer than the 58 minutes required for the sample size of 1.0X and 0.5X. Despite this increased training time, it remains less than the 71 minutes recorded for CRPO. These results underscore the potential benefits of utilizing more samples to improve performance in safe RL tasks. Importantly, in both sample manipulation settings, method ensures safety and outperforms CRPO in terms of reward performance and training efficiency.
x. and of represent the weight of the reward gradient (resp. the safety cost gradient) in the final gradient w/+1- So x; + x = all the time and, for instance, x = 1 (resp. xf = 1) indicates we only use reward gradient (resp. safety cost gradient), and of = o = 0.5 denotes reward and safety cost objectives are considered equally important in the overall gradient. In general, ot and of are hyperparameters in the framework that we can either pre-set as fixed value or adaptively adjust during the running process as needed. For instance, we can set x² to be larger if we care more about the reward performance; otherwise, set xद to be
Different Gradient Weights:
28

---

Safetysla
SuletyWWallor 2d-44
Averag Ephoade Cost
8200
-0
Cmber
54
3200
Cosl mi -00
1200
xnw acee
Epochs
300
20
50
10-00.5
(a)
(b)
(c)
Vafety 14
SatetyWbillrrddivt
SatetyWakkr2dvvd
Acerage Epsede Cost
ESPO 10ustiom03
ESRO youraiin-A
70
4200
19
58
3200-
Avaraqe
1000
ESPO rours-rro two is
Esto court i-ncc
150
Training Epochs
zie
300
405
502
150
Training Epochs
200
300
400
300
Ir-0.1
1=0.01
(d)
(e)
Figure 5: Ablation experiments: Experiments of different gradient weights xf and x and different learning rates.
larger to enhance safety. Throughout our experiments, we just set 27 = x = 0.5 for simplicity, which also showed superior performance than prior arts (see Section 5). We provide ablation study for the hyperparameters to evaluate its effect on the performance, shown Figures 5(a)-(c). test the sensitivity of the performance w.r.t. the hyperparameters z and zf we carry out experiments using other two combinations of z and of (x) = 0.4, xf = 0.6 or z 0.6, o = 0.4). The results show that our proposed ESPO performs well using such different o. o settings and is even better than using क. = o 0.5 sometimes.
Different Learning Rates: We conduct ablation studies on hyperparameters- - learning rates (l,), shown Figure Figures 5(d)-(f), reveal that ESPO is robust to variations in learning rates. The results demonstrate that ESPO performs well under reasonably varying learning rates.
Update Style Analysis: the Sgfery/HmmmoidSStnddn..
The analysis of update style in our experiments, illustrated in Table 3 for
task, offers insightful contrasts between our algorithm, ESPO, and CRPO method. these experiments, we observe the following update patterns: 1) CRPO's 's Update Style: CRPO's 's approach to optimization involved 178 updates focused solely on reward optimization and 322 updates dedicated cost optimization. This distribution suggests a significant emphasis on cost optimization, indicating that CRPO struggles to manage safety constraints. 2) ESPO's 's Update Style: ESPO, on the other hand, showed a more dynamic update pattern. It conducted 298 updates focused on reward optimization, indicating more efficient approach toward maximizing rewards. However, unlike CRPO, ESPO engages 199 updates characterized by simultaneous optimization of both reward and cost. Additionally, 3 updates focused exclusively on optimizing cost. By optimizing both aspects simultaneously, ESPÓ demonstrates novel method of navigating the complex landscape of safe RL, which may contribute to its overall efficiency and effectiveness observed in the task performance.
Update
Al Algorithm
Reward
Reward & Cost
Cost
ESPO (ours)
298
199
CRPO
178
322
Table 3: Update style analysis. The Reward update represents the number of times the algorithm updates its policy primarily focusing on maximizing rewards, the Cost update refers to the number of cost updates where the safety violation happens and the primary focus is on minimizing costs, the Reward& & Cost update corresponds to the number times the optimization of reward and cost updates are executed simultaneously.
29

---

D Detailed Experiments
D.1 Additional Experiments
The results of our experimental evaluations on the SajeryHumanodSttadduu--v task, as depicted in Figures 6(a)-(c), show the superior performance of our algorithm, ESPO, in comparison with SOTA primal baselines, CRPO and PCRPO. Key observations from these results include: ESPO demonstrates a remarkable ability outperform CRPO and achieve comparable performance with PCRPO in reward while ensuring safery. Another notable aspect of ESPO's performance is that our method required less time to reach convergence than these baselines. This efficiency is crucial in practical applications where time and computational resources are often limited. ESPO requires only approximately 76.5% and 74.01% % of the training time that CRPO and PCRPO need, respectively, to achieve superior performance. Specifically, as depicted in Table I, while CRPO and PCRPO utilize million samples for the SaferyyHunanidSStnduup
task, our method requires only 5.1 million samples for the same task. This reduction in samples is a significant advantage, highlighting ESPO's effectiveness in learning efficiency.
These results from the SuffetylummnoidSSandaaa task further demonstrate the effectiveness of our method in safe RL environments, showcasing its potential as reliable and efficient solution for optimizing rewards while adhering to safety constraints.
SwfelythumanoidSteee
SafelyltumanniiSttndd
Avarage Rowarrd
140030
2030-
1930
1:4
154
220030
PCRPC
3030
Corl Linit 1 100
ACAP0
114
L20030
$0030
2030
Training
00
1502
60030
2000
40000
1338
Number
3
Steps
Number
ESPOjOunO)
EAPO
PERPO
(a)
(b)
(c)
Figure 6: Performance comparisons of safe RL methods on Sajery/lummmoodSttnnuuu--- tasks.
D.2 Experiment Settings
The Safety-MuJoCo benchmark is primarily used for primal-based methods. while the Omnisafe benchmark is mainly utilized for primal-dual based methods. Moreover, the Safery-MuJoCo bench- mark is different from the Ommisafe benchmark in safety settings. Safety-MuJoCo encompasses broad safety constraints including both velocity limits and overall robot health. Accounting for multiple factors requires algorithms consider both speed regulation and broader integrity. In contrast, the Ommisafe benchmark primarily focuses on robot velocity as the critical constraint. For instance, cost of is emitted whenever the robot's 's velocity exceeds predefined limit. This singular focus on velocity provides a more targeted, yet still challenging, evaluation context. Through these experimental setups, we aim comprehensively assess the effectiveness of our method in varying scenarios, ranging from the multifaceted safety constraints in Safety-MuJoCo to the velocity-centric constraints in Omnisafe. For more details, see [30] and [32]. To ensure fair evaluation of our method's effectiveness, we conducted all experiments using at least three different random seeds.
The key parameters used in the tasks of Safety-MuJoCo benchmarks are provided in Table 4, Table 5 and Table 6. Note, to encourage more learning exploration, we initiate the optimization of safety after 40 epochs. Experiments in the tasks of Safety-MuJod benchmarks are conducted 20.04.3 LTS system, with an AMD Ryzen-7-2700X CPU and an NVIDIA GeForce RTX 2060 GPU.
Ubuntu
The key parameters used on the tasks of Omnisafe benchmarks are provided in Table 5, Table 6, and Table Experiments on the tasks of Omnisafe benchmarks are conducted on a Ubuntu 20.04.6 LTS system, with 2 AMD EPYC-7763 CPUs and 6 NVIDIA RTX A6000 GPUs.
30

---

Parameters
value
Parameters
value
gamma
0.995
tau
0.97
12-reg
le-3
cost kl
0.05
damping
le-1
batch-size
[16000,
epoch
500
episode length
1000
grad-c
0.5
neural network
MLP
hidden layer dim
64
accept ratio
0.1
energy weight
1.0
forward reward weight
1.0
Table 4: Key parameters used in Safety-MuJoCo benchmarks. In ESPO, the sample size of each epoch is determined by Algorithm with Equations (7) and (6), in which the X is 16000.
Tasks
Ç+
Tasks
Ç+
5
SaferyyHopprrVlnncityyyi/
0.1
-0.4
SafetyAmtVelocity-v1
0.1
-0.4
SaferyHumanoidStaddnpww.
0.0
-0.5
SafetyWalker2d
0.0
0.5
Safety v-4
0.0
-0.5
SafetyWalker2d.v4
0.2
-0.5
SafetyReacher-v4
0.1
-0.3
Table 5: Sample parameters used in Omnisafe and Safety-MuJoCo experiments. The re- sults of www.Hoopppeeelooiyyyy.. and SafetyAAntVeoocityyv// are shown in Figure the results of SaferyyHumanoiSStaddipwww and Safery Walker2d-v4 are shown in Figure 2, the results of SafetyWalker2d-v44 are shown in Figures 4 (a), (b) and (c), the results of WafelyWaller2d-v44- and SogfeyyWllee2ddv44 are shown in Figures 4 (d), (e) and (f); the results of SafetyReacher-vt experiments shown in Figure
Tasks
b
h+
h
Tasks
h+
h-
BafetyHopperVelocityyy.
25
9
SaletyAntVelocityvvi
0.5
0.25
-0.25
Safery/unaanidStaandp--vv
1400
300
Wafe
40
+00
0
Safety Walker2d v4-a
30
+00
Safety 5afety
40
+o0
0
SafetyReacher-vd
40
0
Table 6: Cost limit and slack parameters used in Omnisafe and Safety-MuloCo experiments. and SefetyAnnVelocittyvv are shown in Figure the re- The results of SafetyHopperVeloree and Safety Walker2d-v4 are shown in Figure 2, the results of sults of SaferyHlmanoodSttnddye SafetyWalker2d-ve and SafetyWalker2d-ee of SofetyWalker2de are shown in Figures 4 (d), (e) and (f); the results of SafetyReacher-vt experiments are shown in Figure
are shown in Figures 4 (a), (b) and (c), the results
31

---

values
al gggrtttmss
CUP
PCPO
PPOLag
ESPO
parameters
device
cpu
cpu
cpu
cpu
torch threads
1
1
vector eny nums
1
parallel
1
epochs
500
500
500
500
steps per epoch
20000
20000
20000
update iters
40
10
40
10
batch size
64
128
64
128
target entropy coef reward normalize
0.01
0.01
0.02
0.01
0
False
False
False
False
cost normalize obs normalize use max grad norm max grad norm use critic norm
False
False
False
False
True
True
True
True
True
True
True
True
40
40
40
40
True
True
True
True
critic norm coef
0.001
0.001
0.001
0.001
gamma cost gamma lam lam
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.95
0.95
0.95
0.95
0.95
0.95
0.95
0.95
clip adv estimation method
0.2
0.2
gae
gae
gae
gae
standardized rew adv standardized cost adv cg damping
True
True
True
True
True
True
True
True
0.1
\
0.1
cg iters hidden sizes activation
15
[64, 64]
[64, 64]
[64, 64]
[64, 64]
tanh
tanh
tanh
tanh
0.0003
0.001
0.0003
0.001
lagrangian multiplier init
0.001
0.001
lambda
0.035
0.035
Table 7: Key hyparameters used in Omnisafe experiments. In ESPO. the steps of each epoch is determined by Algorithm 1, with Equations (7) and (6), in which the is 20000. The parameters for the baselines consistent with those of Omnisafe, and their performance meticulously fine-tuned in Omnisafe [32].
32

---

E Impact and Limitation Statements
Impact Statements: This paper presents work aiming to advance the field of safe RL, and we be- lieve the study can significantly benefit multi-objective optimization efficiency, such as optimizations with or even hundreds of objectives. Additionally, this paper shares the general societal impact of progress in this domain, including both potential positive applications and negative consequences such as misuse. As capabilities in safe RL advance toward real-world deployment, continued monitoring and assessment of broader impacts remain warranted, well as ensuring ethical deployment.
Limitation Statements: The study currently focuses on simulation experiments; however, in the future, we aim to deploy our method in real-world applications. Additionally, although it is necessary to set sample size parameters for policy optimization, our method demonstrates superior performance across multiple tasks compared to SOTA baselines. Notably, our method's s performance won't 't be heavily sensitive to the hyperparameters from our observations, as observed our analysis (refer to Section 5.3). It is important to acknowledge that there is no free launch, and algorithm could not address everything?.
htsp:::///ocallhoot////ss there-an-aaee theere--maagggrrttt-------rrrree
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Abstract and Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Appendix E.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
34

---

Justification:
See Section 4.4 and Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Experiment Section 5, and Appendix D.2.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
35

---

Answer: [Yes]
Justification:
See Experiment Section 5.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSubmic / //nips. wttpccccpppccccceeececceccccccct Sttp:///ggiiieeeeeeeuuuuiiiiiiii ssisnnoillllo onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix D.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See Experiment Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
36

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix D.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eerpp/.///ee cc
Answer: [Yes]
Justification:
The research conducted with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
37

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The research does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper already cited related packages that we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
38

---

Answer: [Yes]
Justification: See Experiment Section 5.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "vbqJsGtawP", "reviewer": "Reviewer_qn1v", "summary": "This paper presents a novel algorithm for safe reinforcement learning, ESPO,\nwhich independently collects gradient information for reward optimization and\nconstraint satisfaction. It then makes a dynamic choice about how to combine\nthese two gradients in order to find an optimal safe policy. In addition, the\nproposed algorithm dynamically adjusts the number of samples used for each\ngradient computation resulting in a more sample-efficient learning process than\nexisting techniques. Theoretical results show that ESPO achieves near-optimal\nreward and constraint satisfaction with high probability. Additional results\nshow that ESPO spends more time in safe regions during training than prior work.\nIn experiments, ESPO is able to achieve comparable or better reward and safety\nbehavior to prior approaches while requiring less training time.", "questions": "- How are the coefficients $x_t^r$ and $x_t^c$ computed? What impact do they\n  have on performance?\n- How are the other hyperparameters ($\\chi^+$, $\\chi^-$, $h^+$, $h^-$) chosen?\n  Does it require careful tuning for each environment?", "limitations": "The authors adequately discuss limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Safe RL is quite an important area, and advancements in efficient safe RL make\n  it more applicable to real-world scenarios.\n- The proposed algorithm is quite intuitive and handles a tricky issue\n  (oscillation) appearing in other safe RL algorithms\n- The algorithm is well-grounded in theory with Theorem 4.1. I also appreciate\n  4.2, showing the improved constraint satisfaction at training time compared to\n  existing work.\n- The experimental results are promising. ESPO achieves comparable or better\n  results in terms of both reward performance and constraint satisfaction to\n  existing work, but uses fewer samples.", "weaknesses": "- There is minimal discussion of the coefficients $x_t^r$ and $x_t^c$ even\n  though these seem like they should be critical to the algorithm's performance.\n- It seems that the hyperparameters of ESPO require careful tuning which likely\n  inhibits the deployment of this algorithm in practice. (I'm looking at Tables\n  5 and 6 for this claim which show that the hyperparameters are quite different\n  for different benchmarks.)\n- In some cases the either $h^-$ or $h^+$ is infinite, meaning the algorithm\n  never performs a pure constraint satisfaction step or a pure reward\n  optimization step.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reply to Reviewer qn1v\n> **Q1:** More discussion of the critical coefficients $x_t^r$ and $x_t^c$ of the algorithm's performance. How are the coefficients $x_t^r$ and $x_t^c$ computed?\n\n**A1:** Thanks for raising this point! We conduct new ablation experiments and have added a detailed discussion for $x_t^r$ and $x_t^c$ in the algorithm design section with an intuitive explanation. Please refer to **Q2** in the general response.\n\n\n> **Q2:** Do the hyperparameters of ESPO require careful tuning? (Tables 5 and 6 use different hyperparameters for different benchmarks.)\n\n**A2:** Thanks for raising this point; we appreciate the opportunity to clarify it. \n\n\n- Tuning hyperparameters for different tasks is not unique to ESPO, but widely considered in designing safe RL algorithms. For instance, PCPO [1] requires fine-tuning the projection distance $D$ within a primal-dual safe RL framework. CUP [2] involves fine-tuning the safety hyper-parameter $v$, and FOCOPS [3] necessitates fine-tuning the safety temperature $\\lambda$. For practical considerations, hyperparameter tuning is somewhat necessary for heavily distinct tasks, while it's often a one-time process for each new kind of environments.\n- Table 5 and Table 6 show the choices of the hyperparameters for different tasks: sample size adjustment parameters ($\\zeta^+, \\zeta^-$ defined in Equation (6-7)), safety constraint threshold (limit $b$), and soft region threshold parameters ($h^+, h^-$). **We indeed provide an ablation study in Figure 4 in Appendix C lines 688-709**, over the benchmark SafetyWalker2d. Specifically, Figures 4(a)-\\(c) show the results of ESPO when the safety constraint threshold (limit $b$) is different (30 or 40) with comparisons to baseline CRPO, which exhibits similar and stable reward performance when the limit $b$ varies. In addition, Figures 4(d)-(f) show the ablation w.r.t. sample size adjustment parameters ($\\zeta^+, \\zeta^-$) and demonstrate the stability of the performance under different ($\\zeta^+, \\zeta^-$) settings. We acknowledge the need of reasonable fine-tuning hyperparameters and are actively working on adaptive techniques in future work.\n\n\n\n- > [1] Yang, T., et al. (2020). Projection-Based Constrained Policy Optimization. ICLR 2020.   \n> [2] Yang, L., et al. (2022). Constrained update projection approach to safe policy optimization. NeurIPS 2022.  \n> [3] Zhang, Y., et al. (2020). First order constrained optimization in policy space. NeurIPS 2020.\n\n> **Q3:** In some cases either $h^-$ or $h^+$ is infinite, meaning the algorithm never performs a pure constraint satisfaction step or a pure reward optimization step. How are they chosen and do they require tuning for each environment?\n\n**A3:** Our ESPO algorithm framework intentionally designs and allows for such choices. Different tasks have different preferences for optimizing rewards or prioritizing safety constraints. So generally, we need to choose/tune diverse $h^+, h^-$ to both satisfy the goal of different tasks and also the efficiency of the learning process, with intuitions shown below:\n\n- We can choose $h^- = -\\infty$ (i.e., always consider the safety constraint objective even if constraints are already satisfied). Such a choice will potentially make less oscillation around the safety constraint requirement threshold and always update the reward objective and safety constraint objective together. So this choice fits the tasks that the agent also wants to keep safe during the learning process, but not only exploiting the process, or less oscillation of the optimization process, which can accelerate the learning process.\n- We can choose $h^+ =  \\infty$ (i.e., always consider the reward objective even if constraints are violated). This choice is suitable for those tasks that have less priority for satisfying the safety constraint, or less oscillation of the optimization process can accelerate the learning process.\n- When $h^-, h^+$ are finite (the common choice): creating a soft region $[h^- +b, h^+ +b]$. When the safety cost objective falls into this soft region around the required limit $b$, both the reward objective and the safety cost objective will be considered for the optimization update rule, which can reduce the oscillation around the safety limit to some extent. With such finite choices, the algorithm keeps the possibility of optimizing only the reward or safety cost objective when the safety objective is pretty good or is largely violated, respectively.\n\n> **Q4:** How are the other hyperparameters ($X^+$, $X^-$, $h^-$, $h^+$) chosen? Does it require careful tuning for each environment?\n\n**A4:** The reviewer raises an important point about hyperparameter selections. To the best of the author's understanding, the reviewer is asking about how to choose the sample size adjustment parameters ($\\zeta^+, \\zeta^-$ defined in Equation (6-7)), and soft region threshold parameters ($h^+, h^-$).\n- The intuition of choosing $h^-$, $h^+$ are provided in above answer **A3**. Note that different tasks have different preferences for optimizing rewards or prioritizing safety constraints. So generally, we need to choose/tune diverse $h^+, h^-$ to both satisfy the goal of different tasks and also the efficiency of the learning process. \n- The sample size adjustment parameters $\\zeta^+$ and $\\zeta^-$ determine the final sample size $X(1+\\zeta^+)$ or $X(1+\\zeta^-)$ in different scenarios (whether the gradient conflict occurs). As such, the final sample size typically is set close to the original sample size $X$. **We provide ablation study w.r.t. $\\zeta^+, \\zeta^-$ in Figure 4(d)-(f) in Appendix C lines 688-709**, over the benchmark SafetyWalker2d. Figures 4(d)-(f) demonstrate the stability of the performance under different ($\\zeta^+, \\zeta^-$) settings. It verifies that the ESPO is typically not sensitive to $\\zeta^+, \\zeta^-$ and just needs reasonable tuning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TjOLezDsRF", "reviewer": "Reviewer_oXwv", "summary": "The paper introduces Efficient Safe Policy Optimization (ESPO), an approach that enhances safe reinforcement learning by dynamically adjusting sample sizes based on gradient conflicts. ESPO optimizes reward and safety, improves convergence stability, and reduces sample complexity. The experiments shows the proposed approach outperforms existing methods for both higher reward and improved safety with fewer samples and less training time.", "questions": "None", "limitations": "The authors discussed limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduced Efficient Safe Policy Optimization (ESPO), which dynamically adjusts sample sizes based on observed conflicts between reward and safety gradients. The approach is novel\n2. The paper provides a comprehensive theoretical analysis of ESPO, including convergence rates and optimization stability. \n3. The paper conducted experiments on the Safety-MuJoCo and Omnisafe benchmarks, ESPO demonstrates significant improvements over existing primal-based and primal-dual-based methods.", "weaknesses": "The paper experimented on SafetyReacher-v4, SafetyWalker2d-v4, and SafetyHopper/AntVelocity. Those safety tasks does not test generalization, such as those with safety gym.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reply to Reviewer oXwv\n\n> **Q1:** The paper experimented on SafetyReacher-v4, SafetyWalker2d-v4, and SafetyHopper/AntVelocity. Those safety tasks do not test generalization, such as those with safety gym.\n\n**A1:**  We appreciate the reviewer's insightful comments. To the best of our understanding of the reviewer's question, we clarify the relationship between the benchmarks ([Safety-MuJoCo](https://github.com/SafeRL-Lab/Safety-MuJoCo) and [Omnisafe](https://github.com/PKU-Alignment/omnisafe)) used in this paper and prior popular benchmark (safe gym). We have added this important clarification in the new version.\n- [Safety-MuJoCo](https://github.com/SafeRL-Lab/Safety-MuJoCo) is a new environment benchmark adapted from [MuJoCo](https://github.com/google-deepmind/mujoco) and [Omnisafe](https://github.com/PKU-Alignment/omnisafe) is an algorithm benchmark (does not include new safe RL environments) that support the environments in [Safety-Gymnasium](https://github.com/PKU-Alignment/safety-gymnasium). Here, Safety-Gymnasium is a popular suite, an evolved version of the original Safety-Gym, developed to support the latest gym API and maintain compatibility with ongoing research needs, given that [Safety-Gym](https://github.com/openai/safety-gym) is no longer actively maintained.\n- We use SafetyReacher-v4, SafetyWalker-v4, SafetyHumanoidStandup-v4 in [Safety-MuJoCo](https://github.com/SafeRL-Lab/Safety-MuJoCo). [Safety-Gymnasium](https://github.com/PKU-Alignment/safety-gymnasium) employs cost constraints such as velocity limits and rewards based on the robot’s speed. In contrast, [Safety-MuJoCo](https://github.com/SafeRL-Lab/Safety-MuJoCo) benchmarks enable more kinds of safety cost constraints, such as the robot’s health—monitoring falls and joint forces— which haven’t been included/emphasized in [Safety-Gymnasium](https://github.com/PKU-Alignment/safety-gymnasium). This provides a more comprehensive evaluation of the algorithms to generalize across more types/numbers of safety-critical constraints.\n- Other environments (SafetyHopper/AntVelocity) that we used in this paper are exactly from [Safety-Gymnasium](https://github.com/PKU-Alignment/safety-gymnasium). Omnisafe mainly serves as an algorithm platform that is used to test safe RL baselines in environments from Safety-Gymnasium.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3uxL1xAnU8", "reviewer": "Reviewer_by8k", "summary": "The paper introduces an approach, Efficient Safe Policy Optimization (ESPO), aimed to improve the efficiency of safe reinforcement learning (RL). ESPO tries to enhance sample efficiency through sample manipulation, addressing the challenges of sample inefficiency in safe RL, which often requires extensive interactions with the environment to learn a safe policy. The proposed method dynamically adjusts the sampling process based on the observed conflict between reward and safety gradients.", "questions": "1. How does ESPO perform in environments with high-dimensional state and action spaces compared to low-dimensional ones?\n\n2. What is the impact of noisy gradient estimates on the performance and stability of ESPO?\n\n3. Can ESPO be effectively integrated with off-policy or model-based RL methods to further enhance sample efficiency?", "limitations": "1. ESPO's reliance on gradient conflict signals for sample manipulation might limit its applicability in environments where gradient estimation is noisy or unreliable.\n\n2. The method's scalability to very large or real-time environments is not thoroughly explored, raising questions about its practical deployment in such settings.\n\n3. The sensitivity of ESPO to various hyperparameters, such as learning rates and sample size thresholds, needs further investigation.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well motivated, articulates the challenges in existing safe RL methods and justifies the need for dynamic sample manipulation.\n\n2. ESPO's dynamic sample manipulation based on gradient conflicts seems relevant to the field of safe RL, potentially reducing computational costs and improving learning efficiency.\n\n3. The paper provides both theoretical analysis and empirical validation.", "weaknesses": "1. While the paper evaluates ESPO on two benchmarks, additional evaluations on more diverse and complex environments would strengthen the generalizability claims.\n\n2. More in-depth comparisons with a broader range of SOTA methods could provide a clearer picture of ESPO's relative performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reply to Reviewer by8k\n\nMany thanks to the reviewer for recognizing our contributions in terms of theory and practice.\n\n> **Q1:** Adding evaluations on more diverse, complex, or real-time environments would strengthen the generalizability claims.\n\n**A1:** Thank the reviewer for insightful comments. We agree that testing on additional benchmarks or safe RL tasks is essential to substantiate the generalizability of ESPO. Following the reviewer's suggestion, \n- **We add three new sets of experiments on either a more complex class of safe RL problem --- multi-objective safe RL, or integrating our proposed technical modules with other safe RL algorithms to show the generalizability**. Please refer to **Q1** in the general response for details. The results from these additional experiments affirm that our method not only generalizes well to diverse, safe RL problems but can potentially bring benefits for extensive safe RL algorithms. \n- More experiments on large and real-time environments leave to future work: (1) extending ESPO to safe MARL tasks, which involve multi-agent strategic interactions while maintaining safety constraints with large-scale applications in collaborative robotics or autonomous systems. (2) Deploying ESPO in real-robot control tasks requires addressing real-world challenges such as sensor noise, actuator delays, and environmental uncertainties.\n\n> **Q2:** More in-depth comparisons with a broader range of SOTA methods\n- **A2:** Thank you for raising this point. First, we recall that we include the SOTA and classical ones of both primal methods (CRPO, PCRPO) and primal-dual methods (PCPO, CUP, PPOLag), which are the main classes of safe RL algorithms. So, instead of including more similar baselines, we choose to extend our ESPO to a new safe RL problem --- multi-objective safe RL and compare it to this problem's SOTA baseline CRMOPO. Details can be referred to the previous answer **A1**. These additional experiments verify the efficiency of ESPO (or its variants) not only in the standard safe RL problems, but also in more complex tasks. \n\n> **Q3:** How does ESPO perform in environments with high-dimensional state and action spaces compared to low-dimensional ones?\n\n**A3:** Thank you for your insightful question. We observe that the proposed method ESPO demonstrates superior performance over baselines in both low-dimensional tasks (SafetyHopperVelocity-v1 (action: 3D, state: 10D) and SafetyReacher-v4 (action: 2D, state: 10D)) and relatively high-dimensional tasks (SafetyAntVelocity-v1 (action: 8D, state: 27D), SafetyWalker-v4 (action: 6D, state: 17D), and SafetyHumanoidStandup-v4 (action: 17D, state: 45D). ), but with particularly strong performance in low-dimensional tasks. For details, please refer to Figures 5 and 6 in the general response file.\n- It is an interesting direction to implement ESPO for tasks with higher dimensional state and action, such as using an image as the state. This will involve more challenges, such as representation learning and computer vision. ESPO has potential in such high-dimensional tasks due to its sample efficiency advantages compared to prior art matches the pressing need of reducing required samples in those tasks -- such as the video game benchmark Atari (with the image as input) is typically one of the hardest benchmarks to solve and cost a lot of times to collect samples [1].\n\n[1] Ye, Weirui, et al. \"Mastering atari games with limited data.\" NeurIPS 2021.\n\n> **Q4:** What is the impact of noisy gradient estimates on the performance and stability of ESPO due to its reliance on gradient conflict signals? \n\n**A4:** We appreciate the reviewer's insightful question. Typically, for all policy-based or actor-critic (safe) RL algorithms, the noisy gradient estimates can bring challenges since they are usually estimated with a batch of samples but not infinite samples. \n\n- **Inherent challenges for safe RL problems:** The reviewer is correct that the noisy gradient estimates bring challenges for ESPO, while it is not primarily brought by algorithm design, but the inherent daunting challenges for safe RL problems ---a good balanced update direction is hard to find using noisy reward gradient and safety cost gradient estimates (there may be conflict happens between reward and cost). \n- **The sample manipulation using gradient conflict signals (our key technical module in ESPO) can potentially reduce the effects of noisy gradient estimates.** Instead of limiting the applicability, the proposed sample manipulation is actually inspired by the noisy gradient estimate issue and designed to reduce its effect. In summary, the sample manipulation module will use more samples to estimate the final gradient when it needs more accuracy (when there is possibly a conflict between the reward and cost and a balance is required) and fewer samples when it tolerant more error --- improve sample efficiency. We acknowledge that the gradient conflict signal metric may also be noisy, but it won't directly hurt the final gradient direction. ESPO's superior performance implicitly shows that the performance is not sensitive to the gradient conflict signal errors.\n\n> **Q5:** Can ESPO be effectively integrated with off-policy or model-based RL methods?\n- **A5:** The answer is Yes. The core capability of ESPO lies in its use of sample manipulations based on reward and cost gradients to improve sample efficiency. This mechanism can be seamlessly incorporated into the frameworks of off-policy and model-based approaches, where gradients are readily available.  While it has not been tested, its fundamental principles suggest that such integration is feasible and potentially very beneficial. This is an exciting direction for future work.\n\n> **Q6** The sensitivity of ESPO to various hyperparameters, such as learning rates and sample size thresholds.\n- **A6:** Thank the reviewer for the comments on hyperparameters. Please refer to **Q3** in the general response for details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EYJaNn2Sg7", "reviewer": "Reviewer_qrLu", "summary": "This paper proposes a safe RL algorithm that adjusts the sample number during one on-policy update based on the conflict between the reward gradient and the cost gradient. When the policy's cost value is near the constraint threshold and there is a gradient conflict, a larger sample number is used; otherwise, a smaller sample number is applied. The paper also provides theoretical results about convergence rates, reducing oscillation, and sample efficiency. Finally, the paper evaluates the algorithm on two safe RL benchmarks.", "questions": "1. Eq. (3)(4) of PCRPO, serving as an essential component of your algorithmic pipeline, is kind of indigestible.  How to set $x^r_t$, $x^c_t$? And how is Eq. (3) functioning as 'projecting reward and cost gradients onto their normal planes'? It would be better to add more details or illustrations about PCRPO (at least in the appendix).\n2. I see the $h^+,h^-$ dynamically change in rows 4, 7 in Algorithm 1 ESPO. Is it an existing component of PCRPO or a new trick in ESPO?\n3. From the description of Theorem 4.1 and Proposition 4.2, I cannot find the association between sample manipulation and these two theoretical results. Can I view them as conclusions about PCRPO and not tailored to ESPO?\n4. It seems that Assumption A.7, which bridges sample size and performance, serves as a very important theoretical base to verify the sample size manipulation. There should be more intuitive explanations and discussions to justify this assumption.\n5. I guess you use a fixed number of training epochs for all algorithms. But training epochs are not a good metric for comparison, especially when each epoch corresponds to different sample steps, so maybe the x-axis of Figures 2 and 3 should be set to sample steps rather than training epochs to fairly demonstrate the sample efficiency of all algorithms. Besides, Table 1 should report the sample steps used to reach the same performance rather than the sample steps in a fixed number of epochs.", "limitations": "The limitations are discussed in the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- It is an interesting and innovative attempt to improve sample efficiency for safe RL by dynamically adjusting the sample number of on-policy updates based on gradient conflict.\n- Some efforts on theoretical analysis are provided to justify the proposed approach.\n- Empirical results demonstrate the capability of this work to improve performance and sample efficiency.", "weaknesses": "- The sample manipulation is an interesting mechanism, and the gradient conflict, as a manipulation signal, occurs widely in many safe RL algorithms, including both primal and primal-dual methods. However, it is solely applied to PCRPO, which may weaken its generalization and persuasiveness and makes it seem like a minor improvement tailored to PCRPO.\n- The description of the main backbone PCRPO is insufficient, making it difficult to understand the whole algorithm pipeline without prior knowledge of PCRPO. See question 1, 2.\n- Some points about the theoretical results remain to be clarified. See questions 3 and 4.\n- Some experimental settings are debatable. See question 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reply to Reviewer qrLu\n\nWe appreciate the reviewer for recognizing our contributions in both practice and theory, and for providing constructive suggestions.\n\n> **Q1:** The sample manipulation is an interesting mechanism with the gradient conflict as a signal.  Is it solely applied to PCRPO or can be extended to other safe RL algorithms?\n\n\n**A1:** Many thanks to the reviewer's recognition of the potential generalization of our sample manipulation approach. The reviewer is insightful that using the gradient conflict as a manipulation signal (metric) can improve the sample efficiency of extensive safe RL algorithms, not only PCRPO. Please refer to **Q1** in the general response. \n\n\n\n> **Q2:** How to set $x^r_t$ or $x^c_t$ ? Explain Eq. (3)(4) of PCRPO and how is Eq. (3) functioning as 'projecting reward and cost gradients onto their normal planes'? Add more details about PCRPO.\n\n**A2:** Thank the reviewer for the clarification suggestions. \n\n- $x_t^r$ and $x_t^c$ represent the weights of the reward gradient and the safety cost gradient in the final gradient $w_{t+1}$. Please refer to **Q2** of the general response for more details, where we explain how to choose/set $x_t^r,x_t^c$ and also **conduct new ablation studies using varying $x_t^r$ and $x_t^c$**.\n- As the reviewer suggested, we have added an intuitive description for Eq.(3) in the main text and a more detailed introduction and illustration for PCRPO in the appendix. Recall the first line of Eq.(3) serves as the update rule when there is some conflict between the reward gradient and the safety cost gradient. In that case, to explain \"project the reward and cost gradients onto their normal planes\", we take the first term of Eq.(3) $g_r  - \\frac{g_r \\cdot g_c}{ ||g_r||^2} g_c$ as an example. $\\frac{g_r \\cdot g_c}{ ||g_r||^2} g_c$ represent the vector projection of reward gradient $g_r$ on the cost gradient $g_c$. And the corresponding surrogate reward gradient $g_r - \\frac{g_r \\cdot g_c}{ ||g_r||^2} g_c$ becomes a vector that is perpendicular to $g_c$ --- the surrogate reward gradient not only improves reward but also won’t conflict with the cost gradient $g_c$ (since it is perpendicular to $g_c$, which we called the normal plane).\n\n\n> **Q3:** I see the dynamically change in rows 4, 7 in Algorithm 1 ESPO. Is it an existing component of PCRPO or a new trick in ESPO?\n\n**A3:** We apologize for this confusion. The dynamic change in rows 4 and 7 of Algorithm 1 (ESPO) is indeed an existing component from PCRPO, not a new trick introduced in ESPO. This mechanism in PCRPO is designed to dynamically adjust the range of soft region (by setting $h^+, h^-$) to adapt the optimization behaviors. This trick can improve the trade-offs between reward performance and safety constraints objectives, which we maintained in ESPO to leverage its efficiency.\n\nThe contributions of ESPO focus on developing the new sample manipulation paradigm using gradient conflict as the metric/signal, which shows powerful advantages in sample efficiency, as shown in the paper and our new experiments (please refer to the previous answers **A1** to Q1).\n\n> **Q4:** Are the theoretical results in Theorem 4.1 and Proposition 4.2 conclusions about PCRPO but not tailored to ESPO?\n- **A4:** This is indeed one of the essential theoretical contributions of this work. The reviewer is correct that the convergence guarantees (Theorem 4.1) and the advantages of reduced optimization oscillation (Proposition 4.2) for our ESPO also hold for the prior work PCRPO, which are direct implications of our results. Notably, PCRPO is currently one of the state-of-the art safe RL algorithms, while still needing theoretical guarantees. This work closes this gap for PCRPO. In addition, ESPO is a broader algorithm framework  --- PCRPO can be seen as a special case with identical sample size choices for all training iterations, which all the theoretical results hold for.\n\n> **Q5:** Assumption A.7 serves as a key to verify the sample size manipulation. Give more intuitive explanations and discussions to justify this assumption.\n- **A5:** We have added discussions to justify Assumption A.7 in the new version: In words, Assumption A.7 assumes a local Lipschitz property of the Q function error term $\\delta_t = \\|Q_r^{\\pi_{w_t}}  - \\overline{Q}_t^r\\|_2$ with respect to the sample size $s_t^B$ used for estimating $\\overline{Q}_t^r$ at any time step $t$. Typically, this can be satisfied easily since $\\delta_t$ generally will decrease monotonically as the sample size increases in a polynomial order, such as $\\delta_t \\approx O(\\sqrt{\\frac{1}{s_t^B}})$ that usually holds with high probability verified by high-dimensional statistics (some concentration inequalities).\n\n\n> **Q6:** Change the x-axis of Figures 2 and 3 to sample steps rather than training epochs for fair comparisons. Besides, Table 1 should report the sample steps used to reach the same performance rather than that used in a fixed number of epochs.\n\n**A6:** As the reviewer suggested, to provide a more fair and informative evaluation for the proposed method, we have made the following changes to show the results:\n\n- For Figures 2 and 3: We have revised them to use the number of sample steps as the x-axis rather than training epochs, shown in **Figures 5 and 6 in the pdf of the general response**. This more reasonable illustration further underscores the sample efficiency of our proposed ESPO compared to baselines.\nFor both Table 1 and Table 2: We have revised them (in the general response pdf file Table 1 and Table 2) to report the number of sample steps required to reach reasonable performance thresholds, rather than after a fixed number of epochs. \n- Using these more fair metrics that the reviewer suggested, the results demonstrate even more efficiency of our ESPO compared to the baselines, in terms of not only sample efficiency, but also reward and safety performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response:

We thank the reviewers for their careful reading of the paper and their insightful and valuable feedback. Here, we provide **new experimental results** and discussions to answer some common questions raised by reviewers. 

**We attached a pdf file to show the required new experimental results/updated figures** for all reviewers, listed below:
* Figure 1: Integrate our algorithmic techniques in primal-dual based safe RL.
* Figure 2: Integrate our algorithmic techniques in primal based safe RL.
* Figure 3: Integrate our algorithmic techniques in a more complex class of problems ---- safe multi-objective RL.
* Figure 4: Conduct new ablation experiments regarding hyperparameters of gradient update rules or learning rates.
* Figures 5 & 6: Update figures to use the number of sample steps as the x-axis rather than training epochs.

**Q1: Generalization of our key technical module in the algorithm (sample manipulation) to other safe RL algorithms or problems (Q1 of reviewer qrLu)/ Broader evaluation of the proposed method ESPO (Q1 of reviewer by8k).**


**A1:** We appreciate the reviewers' acknowledgment of the potential for our sample manipulation approach as a general method. Following the reviewers' suggestions, we conduct **three new sets of experiments** to demonstrate the generalization and advantages of our sample manipulation module (the key technical module in our algorithm) by integrating them with representative existing safe RL algorithms:
- **1) Integrating with primal-dual method --- [TRPO Lagrangian](https://cdn.openai.com/safexp-short.pdf) for standard safe RL problems:** see Figure 1 in the general response file.
- **2) Integrating with primal method --- [CRPO](https://proceedings.mlr.press/v139/xu21a/xu21a.pdf) for standard safe RL problems:** see Figure 2 in the general response file.
- **3) Adapting ESPO to a class of more complex safe RL problems --- multi-objective safe RL problems:** The results are shown in Figure 3 in the general response file. We adapt ESPO to the multi-objective safe RL problem (termed as ECRMOPO) and evaluate it on a [safe multi-objective RL benchmark](https://arxiv.org/pdf/2405.16390),  with comparison to the SOTA baseline [CRMOPO](https://arxiv.org/pdf/2405.16390).
   

Summing up the results from these experiments affirms that our sample manipulation method can improve the sample efficiency of diverse safe RL problems and also extensive safe RL algorithms (including both primal-based and primal-dual-based methods). Those results also serve as a broader and more comprehensive evaluation of our proposed sample manipulation approach. We will emphasize the broad power of the sample manipulation design in the revised manuscript.

**Q2: Questions for choosing and tuning hyperpaprameters: $x_t^r$ and $x_t^c$ of the optimization update rule.**

**A2:** We have added a detailed discussion for $x_t^r$ and $x_t^c$ in the algorithm design section and provide an intuitive explanation here:
- **How to choose $x_t^r, x_t^c$.** $x_t^r$ (resp. $x_t^c$) represents the weight of the reward gradient (resp. the safety cost gradient) in the final gradient $w_{t+1}$. So $x_t^r + x_t^c=1$ all the time and for instance, $x_t^r=1$ (resp.  $x_t^c=1$)  means we only use reward gradient (resp. safety cost gradient), and $x_t^r = x_t^c = 0.5$ means reward and safety cost objectives are considered equally important in the overall gradient. In general, $x_t^r$ and $x_t^c$ are hyperparameters in the framework that we can either pre-set as a fixed value or adaptively adjust during the running process as needed. For instance, we can set $x_t^r$ to be larger if we care more about the reward performance; otherwise, we set $x_t^c$ to be larger to enhance safety. Throughout our experiments, we just set $x_t^r = x_t^c = 0.5$ for simplicity, which also showed superior performance than prior arts (see Section 5).
- **As the reviewers suggested, we add an ablation study for the hyperparameter $x_t^r$ and $x_t^c$ to evaluate its effect on the performance, shown in Figures 4(a)-\(c\) in the general response pdf file.** To test the sensitivity of the performance w.r.t. the hyperparameters $x_t^r$ and $x_t^c$, we add experiments using other two combinations of $x_t^r$ and $x_t^c$ ($x_t^r=0.4, x_t^c=0.6$ or $x_t^r=0.6, x_t^c=0.4$). The results show that our proposed ESPO performs well using such different $x_t^r, x_t^c$ settings and is even better than using $x_t^r = x_t^c = 0.5$ (in our paper) sometimes.


> **Q3: Ablation studies for other parameters, such as learning rates and sample size thresholds.**
- **A3:** We appreciate the reviewers' valuable comments. We indeed have ablation studies in Appendix C (due to the limit of the main text), and we have **conducted more ablations on learning rates and gradient weights**, see Figure 4 in the general response file.
- **During rebuttal, we add ablation studies on key hyperparameters --- learning rates ($l_r$) and the weights of gradients ($x_t^r, x_t^c$), shown in Figure 4 in the general response file**, reveal that ESPO is robust to variations in learning rates. The results demonstrate that ESPO performs well under reasonably varying learning rates or weights of gradients.
- **Ablation study for sample size thresholds ($\zeta^+,\zeta^-$) are provided in Figure 4(d)-(f) in Appendix C lines 688-709 of the paper**, over the benchmark SafetyWalker2d. Figures 4(d)-(f) demonstrate the stability of the performance under different ($\zeta^+, \zeta^-$) settings. It verifies that the ESPO is typically not sensitive to $\zeta^+, \zeta^-$ and just needs reasonable tuning.

Meta-review: This paper proposes a safe RL method called ESPO that leverages the gradient conflict to enhance sample efficiency during policy learning. The base algorithm is mainly based on PCRPO (reference [19]), which decouples the safe policy learning process into solely reward maximization, cost minimization, and another stage that simultaneously optimizes both reward and cost. The proposed method ESPO improves over PCRPO by leveraging the conflicts between the reward and cost gradient to dynamically adjust the sample size accordingly. By assigning more samples for situations with gradient conflicts (reward and safety are more challenging to balance for policy learning), and reducing the samples for situations with no conflicts, the proposed method can achieve better sample efficiency as compared to PCRPO. The paper also provides a detailed theoretical analysis (mostly based on the theoretical analysis in CRPO (reference [38])). The strengths and weaknesses are summarized below:

**Strengths:**

-	The idea of leveraging reward and cost gradient conflict to dynamically adjust sample size is interesting. The empirical results also demonstrate its effectiveness.
-	Some efforts on theoretical analysis are provided to justify the proposed approach.

**Weaknesses:**

-	The practical algorithm and theoretical analysis are mostly built upon existing works (PCRPO and CRPO). The new component introduced in this paper is a simple sample size adjustment scheme, which is somewhat incremental. The theoretical analyses are mostly borrowed from CRPO with some adaptation, which in my opinion is not sufficiently acknowledged in the paper. Also, I feel that the theoretical analysis still has some gaps to fully cover the practical algorithm. 
-	As mentioned by one of the reviewers, the proposed method contains lots of hyperparameters, which can be tricky to tune in practice. The authors also used quite different hyperparameters for different benchmark tasks, which is not a very good practice. I suggest the authors present the results produced by a minimal set of hyperparameters in the final version.
-	Decoupling reward maximization and cost minimization in safe RL has also been explored in other existing online and offline safe RL works such as [1, 2], which are not adequately acknowledged in the paper. Also, more comparisons with SOTA methods should be conducted, as suggested by reviewers.
-	Lastly, more description of the backbone algorithm PCRPO also needs to be added to the main text to improve the readability of this paper.

The paper received generally positive feedback from the reviewers. Most of the concerns from reviewers have been addressed during the Author Rebuttal phase. I hope the authors can further improve the paper to address the remaining issues in their final version.
Decision: Accept (poster)