GUARD: A Safe Reinforcement Learning Benchmark
Authors: Weiye Zhao
Yifan Sun
Feihan Li
Rui Chen
Ruixuan Liu
Tianhao Wei
Changliu Liu
Venue: Accepted by TMLR ()

Abstract: Due to the trial-and-error nature, it is typically challenging to apply RL algorithms to safety-critical real-world applications, such as autonomous driving, human-robot interaction, robot manipulation, etc, where such errors are not tolerable. Recently, safe RL (i.e. constrained RL) has emerged rapidly in the literature, in which the agents explore the environment while satisfying constraints. Due to the diversity of algorithms and tasks, it remains difficult to compare existing safe RL algorithms. To fill that gap, we introduce GUARD, a Generalized Unified SAfe Reinforcement Learning Development Benchmark. GUARD has several advantages compared to existing benchmarks. First, GUARD is a generalized benchmark with a wide variety of RL agents, tasks, and safety constraint specifications. Second, GUARD comprehensively covers state-of-the-art safe RL algorithms with self-contained implementations. Third, GUARD is highly customizable in tasks and algorithms. We present a comparison of state-of-the-art on-policy safe RL algorithms in various task settings using GUARD and establish baselines that future work can build on.

Published in Transactions on Machine Learning Research (03/2024)
GUARD: A Safe Reinforcement Learning Benchmark
Weiye Zhao
wekyesha crn. eduu
Robotics Institute
Carnegie Mellon University
Yifan Sun Robotics Institate
w cww.. edu
Carnegie Mellon University
Feihan Li
feihan/@@ondrw..om..
Robotics Institute
Cornegie Mellon University
Rui Chen Robotics Institute
ruic33andrew. cww. edu
Cornegie Mellon University
Ruixuan Liu
ruizuan(@andrew.co www. crvu, edu
Robotics Institute
Cornegie Mellon University
Tianhao Wei
twei@Sondree.cmm.
Robotics Institute
Carnegie Mellon Universily
Changliu Liu
dluurBandrew.com cmau.
Robotics Institade
Cornegie Mellon University
Reviewed on OpenReview: https: // openreview. net/ forum? id= kZFKwApeQO
Abstract
Due to the trial-and-error nature, it is typically challenging to apply RL algorithms to safety- critical real-world applications, such as autonomous driving, human-robot interaction, robot manipulation, etc, where such errors are not tolerable. Recently, safe RL (i.e., constrained RL) has emerged rapidly in the literature, which the agents explore the environment while satisfying constraints. Due to the diversity of algorithms and tasks, it remains difficult to compare existing safe RL algorithms. To fill that gap, introduce GUARD, Generalized Unified SAfe Reinforcement Learning Development Benchmark. GUARD has several advantages compared to existing benchmarks. First, GUARD is a generalized benchmark with a wide variety of RL agents, tasks, and safety constraint specifications. Second, GUARD comprehensively covers state-of-the-art safe RL algorithms with self-contained implementations. Third, GUARD is highly customizable in tasks and algorithms. We present comparison of stato-of-the-art on-policy safe RL algorithms in various task settings using GUARD and establish baselines that future work can build on.
'Equal Contribution

---

Published in Transactions on Machine Learning Research (03/2024)
1 Introduction
Reinforcement learning (RL) has achieved tremendous success in many fields over the past decades. In RL tasks, the agent explores and interacts with the environment by trial and error, and improves its performance by maximiizing the long-lerm reward signal. RL algorithms enable the development of intelligent agents that.
can achieve human-competitive performance in wide variety of tasks, such as games (Mnih et al. 2013
Silver ct al. 2018 OpenAI ct al.) 2019 Vinyals ct al., 2019), manipulation (Popov et al. 2017 Chen ct al.
2023 Agostinelli et al. 2019 Shek et al.
2022).
autonomous driving (Isele et al. 2019 Kiran et al.) 2022
Gu et al.. (022a)), robotics (Kober et al.
2013
Brunke et al.) 2022). and more. Despite their outstanding
performance in maximizing rewards, recent works (Garcia & Fernández 2015 Gu et al. 2022b Zhao et al.,
2023) focus on the safety aspect of training and deploying RL algorithms due to the safety concern in woal-world safety-critical applications, e.g:; human-robot interaction, autonomous driving, etc. As safe RT. topics emerge in the literature, it is crucial to employ standardized benchmark for comparing and evaluating the performance of various safe RL algorithms across different. applications, ensuring a reliable transition from theory to practice. A benchmark includes 1) algorithms for comparison; 2) environments to evaluate algorithms; 3) set of evaluation metrics, etc. There are benchmarks for unconfined RL and some safe RL,
but not comprehensive enough (Duan et al. 2016 Brockman et al. 2016 Ellenberger 2018 2019 Yu et al.
2019 Osband et al. 2020 Tunyasuvunakool et al. 2020 Dulac-Arnold et al. 2020 Zhang el al. 2022a).
To create a robust safe RL benchmark, we identify three essential pillars. Firstly, the benchmark must be generalized, accommodating diverse agents, tasks, and safety constraints. Real-world applications involve various agent types (e.g-; drones, robot. arms) with distinct complexities, such different control ofgrees-of-freedoon (DOF) and interaction modes (e.g., 2D planar or 3D spatial motion). The performance of algorithms is influenced by several factors, including variations in robots (such as observation and action space dimensions), tasks (interactive or non-interactive, 2D or 3D), and safety constraints (number, trespassibility, movability, and motion space). Therefore, providing a comprehensive environment to test the generalizability of safe RL algorithms is crucial.
Secondly, the benchmark should be unified, overcoming discrepancies in experiment setups prevalent in the emerging safe RL litcrature. A unified platform ensures consistent evaluation of different algorithms in controlled environments, promoting reliable performance comparison. Lastly, the benchmark must be extensible, allowing researchers to integrate new algorithms and extend setups to address evolving challenges. Given the ongoing progress in safe RL, the benchmark should incorporate major existing works and adapt to advancements. By encompassing these pillars, the benchmark provides a solid foundation for addressing these open problems in safe RL research.
In light of the above-mentioned pillars, this paper introduces GUARD, a Generalized Unified SAfe Reinforcement Learning Development Benchmark. In particular, GUARD is developed based upon the Safety Gym (Ray et al. 2019b,, SaieRI-Kit (Zhang et al. 2022a) and SpinningUp [Achiam 2018). Unlike existing benchmarks, GUARD pushes the boundary beyond the limit by significantly extending the algorithms in comparison, , types of agents and tasks, and safety constraint specifications. The code is available on Githul The contributions of this paper are as follows:
1. Generalized benchmark with a wide range of agents. GUARD genuinely supports 11 different agents, covering the majority of real robot types.
2. Generalized benchmark with a wide range of locomotion tasks. GUARD comprehensively supports 7 distinct robot locomotion task specifications, which can be combined to represent a wide spectrum of real-world robot lasks that necessitate intricate locomotion for successful completion.
3. Generalized benchmark with a wide range of safety constraints. GUARD genuinely supports 8 distinct safety constraint specifications. These included constraint options comprehensively cover the safety requirements encountered in real-world applications.
4. Unified benchmarking platform with comprehensive coverage of safe RL algorithms. Guard implements 8 stale-of-the-art on-policy safe RL algorithms following unified code structure.
2

---

Published in Transactions on Machine Learning Research (03/2024)
5. Highly customizable benchmarking platform. GUARD features modularized design that enables effortless customization of new robot locomotion testing suites with sell-customizable agents, tasks, and constraints. The algorithus in GUARD are self-contained, with a consistent structure and independent implementations, ensuring clean code organization and climinating dependencies between different algorithms. This self-contained structure greatly facilitates the seamless integration of new algorithms for further extensions.
2 Related Work
Open-source Libraries for Reinforcement Learning Algorithms Open-source RL libraries are code bases that implement represeniative RI. algorithms for efficient. deployment and comparison. They often serve as backbones for developing new RL algorithmss, greatly facilitating RL research. We divide existing libraries into two categories: (a) safety-oriented RL libraries that support safe RL algorithms, and (b) general RL libraries that do not. Among safety- libraries, Safety Gym (Ray et al. 2019b) is the most famous one with highly configurable tasks and constraints but only supports three safe RI. methods. SafeRI.KKit (Zhang et al., 2022a) ) supports five safe RL methods while missing some key methods such as CPO (Achiam (Gronauer 2022) offers support for CPO but is limited in its overall safe RL support, encompassing total of four methods. Compared to the above libraries, our proposed GUARD doubles the support to cight methods in total, covering a wider spectrum of general safe RL rescarch. General RL libraries, on the other hand, can be summarized according to their backend into Py PoTorc (Achiam 2018
et al. 2017). Bullet-Safety-CC
Weng et al.) 2022 Rafin et al. 2021 Liang et al. 2018). Tensorflow (Dhariwal et al. 2017 Hill et al. 2018
Jax (Castro ct al. 2018: Hoffman et al. 20200, and Keras (Plappert] 2016). In particular, SpinningUp
Achiam
2018) serves as the major backbone of our GUARD benchmark on the safety-agnostic RL portion.
Benchmark Platform for Safe RL Algorithms To facilitate safe RL research, the benchmark platform should support wide range of task objectives, constraints, and agent types. Among existing work, the most reprosentative one is Safety Gym (Ray et al.) 2019b) which is highly configurable. However, Safety Gym is limited in agent types in that it does not support high-dimensional agents (e.g., drone and arm) and lacks tasks with complex interactions (e.g., chase and defense). Moreover, Safety Gym only supports naive contact dynamics (e.g., touch and snap) instead of more realistic cases (e.g., objects bouncing off upon contact) in contact-rich tasks. Safe Control Gym (Yuan et al. 2022) is another open-source platform that supports very simple dynamics (i.e., colt,ole, 1D/2D quadrotors) and only supports navigation tasks. Bullet Safety Gym Gronnuer 2022) provides high-fidelity agents, but the types of agents are limited, and they only consider navigation tasks. Gymnss-Gmmnasium (Ji et al.] 2023) provides a rich array of safety RL task categories. However, it faces limitations in implementing supporting modern safe RL algorithms and offering flexible testing suite for each category. These weaknesses are particularly notable in the context of locomotion tasks, where there are very limited options for robots, constraints, and objectives. Compared to the above platforms, our GUARD supports a much wider range of robot locomotion task objectives (e.g., 3D reaching, chase and defense) with a much larger variety of eight agents including high-dimensional ones such as drones, arms, ants, and walkers.
3 Preliminaries
Markov Decision Process An Markov Decision Process (MDP) is specified by tuple (S, R, P. where S is the state space, and A is the control space, R: Sx E is the reward function, 0 0 < 1 is the discount factor, p: S J0, 1] is the starting state distribution, and P : S x A x S + [0.1] is the transition probability function (where P(s'|s, a) is the probability of transitioning to state s' given that the previous state was s and the agent took action a at state s). A stationary policy x : S + P(A) is a map from states to probability distribution over actions, with (a)ss denoting the probability of selecting action a in state s. We denote the set of all stationary policies by II. Suppose the policy is parameterized by 0; policy search algorithms search for the optimal policy within a set Tl C IT of parameterized policies.
3

---

Published in Transactions on Machine Learning Research (03/2024)
The solution of the MDP is policy that maximizes the performance measure [(T) computed via the discounted sum of reward:
x Σ (r(()),).s(41)
=
(1)
where -= [s0, 40, 81,"..] is the state and control trajectory, and 7 ~ π is shorthand for that the distribution over trajectories depends on т : so ~ a, ~ T(-[st,, S141 ~ P(-ss,,a)) at). Let R( E RR(ss, at s(+1) be the discounted return of trajectory. We define the on-policy value function as (*) E ( 80 = s), the on-policy action-value function Q" (s( = Erww [R(TT) so 5,00 a], and the advantage function A (s, = QT (s. a) (s).
Constrained Markov Decision Process A constrained Markov Decision Process (CMDP) is an MDP augmented with raistsaints that restrict the set of allowable policies. Specifically, CMDP introduces a set of cost functions, C1, C2, ... Cm, where C2 : S x A S + maps the state action transition tuple into a cost value. Similar to equation we denote Jc, (T) = Er
(st, de, St+1] as the cost measure for policy x with respect to the cost function C,. Hence, the set of feasible stationary policies for CMDP is then defined π { πT Vi, Jc, S dj), where d, E R. In CMDP, the objective is to select feasible stationary policy that maximizes the performance: max [TT). Lastly, we define on-policy value, action-value, rEllaille
and advantage functions for the cost as Vc, Q6, and Ag, which as analogous to V*, Q", and A", with C2 replacing R.
4
GUARD Safe RL Library
4.1
Overall Implementation
As a highly self contained safe RT. benchmark, GUARD contains the latest methods that can achieve on-policy safe RL:: (i) end-to-end safe RL algorithms including CPO (Achiam et al. 2017). TRPO-Lagrangian (Bohez
eet al. 2019 TRPO-FAC (Ma et al. 2021), TRPO-IPO (Liu et al. 2020). and PCPO Yang et al.
2020b); (ii)
hierarchical safe RL algorithms including TRPO-SL SL (TRPO-Safety Layer) Dalal et al. 2018 and TRPO- USL
TTRPOUUnoolling Safety Layer) (Zhang et al. 2022a). We also include TRPO (Schulman et al., 2015) as
an unconstrained RL baseline. Note that GUARD only considers model-free approaches which rely less on assumptions than model-based ones. We highlight the benefits of our algorithm implementations in GUARD:
GUARD comprehensively covers a wide range of on-policy algorithms that enforce safety in both hierarchical and end-to-end structures. Hierarchical methods maintain a separate safety layer, while end-to-end methods solve the constrained learning problem as a whole.
GUARD provides fair comparison among safety components by equipping every algorithm with the same reward-oriented RL backbone (i.c., TRPO (Schulman et al. 2015)), implementation (i.e., MLP policies with [64, 64] hidden layers and tanh activation), and training procedures. Hence, all algorithms inherit the performance guarantee of TRPO.
GUARD is implemented in PyTorch with a clean structure where every algorithm is self-contained, enabling fast customization and development of new safe RI algorithms. GUARD also comes with unified logging and plotting utilities which makes analysis easy.
4.2 Unconstrained RL
TRPO We include TRPO (Schulman et al. 2015) since is state-of-the-art and several safe RI. algorithms are based on it. TRPO is an unconstrained RL algorithm and only maximizes performance J. The key idea behind TRPO to iteratively update the policy within a local range (trust region) of the most recent version Te. Mathematically, TRPO updates policy via.
Tk+1 = arg max ((x)
s.t. D (KI(T, Tx) S 26,
(2)
Hlll,
4

---

Published in Transactions on Machine Learning Research (03/2024)
where DK is Kullback-Leibler (KL) divergence, 5 > 0 and the set {a IIg DKu(A, S is called the trust region. To solve equation 2 TRPO applies Taylor expansion to the objective and constraint at to the first and second order, respectively. That results in an approximate optimization with linear objective and quadratic constraints (LOQC). TRPO guarantees a worst-case performance degradation.
4.3
End-to-End Safe RL
4.3.1
Constrained Policy Optimization-base based Algorithms
Constrained Policy Optimizaiton (CPO) (Achiam et al. 2017 handles CMDP by extending TRPO. Similar to TRPO, CPO also performs local policy updates in trust region. Different from TRPO, CPO additionally requires *x+1 to be constrained by П, n llc. For practical implementation, CPO replaces the objective and constraints with surrogate functions (advantage functions), which can easily be estimated from samples collected on Th, formally:
CPO
7k+1 arg max E [A (s,a)) Flllp
(3)
soud"
ama
s.t. (T) [c,(TT) +
E s aret
nC (8, a)
where T - (1- P(s s/T)) is the discounted state distribution. Following TRPO, CPO also performs Taylor expansion on the objective and constraints, resulting in a Linear Objective with Linear and Quadratic Constraints (TOTLC)). CPO inherits the worst-case performance degradation guarantee from TRPO and has a worst-case cost violation guarantce.
PCPO Projection-base Constrained Policy Optimization (PCPO) Yang al. 2020b is proposed based on CPO, where PCPO first maximizes reward using a trust region optimization method without any constraints, then PCPO reconciles the constraint violation (if any) by projecting the policy back onto the constraint set. Policy update then follows an analytical solution:
28
9o 11
11
+ g
28
max 0, V 9
M++1 = TK
H H-1 g
[ 1--10c 9e
(4)
H
9. L L-
where Ie is the gradient of the cost advantage function, g is the gradient of the reward advantage function, H. is the Hessian of the KL divergence constraint, b is the constraint violation of the policy Th L I for Le norm projection, and L H for KL divergence projection. PCPO provides lower bound reward improvement and an upper bound on constraint violation.
4.3.2 Lagrangian-based Algorithms
TRPO-Lagrangian Lagrangian methods solve constrained optimization by transforming hard constraints into soft constraints in the form of penalties for violations. Given the objective [(T) and constraints (Tc, (T) S de). TRPO-Tagrangian (Bohez et al. 2019) first. constructs the dual problem
max min - [TT) + Σ (T). -d)) 2
(5)
The update of is done via trust region update with the objective of equation 2 replaced by that of equation while fixing 7₂ The update of X; is done via standard gradient ascend. Note that TRPO-Lagrangian does not have theoretical guarantee for constraint satisfaction.
TRPO-FAC Inspired by Lagrangian methods and aiming at enforcing state-wise constraints (c.g., preventing state from stepping into infeasible parts in the state space), Feasible Actor Critic (FAC) (Ma et al. 2021
5

---

Published in Transactions on Machine Learning Research (03/2024)
introduces multiplier (dual variable) network. Via alternative update procedure similar to that for equation 5 TRPO-FAC solves the statewise Lagrangian objective:
max min (TT) + Σ Bandra (T) - d4)],
(6)
Vi,g. sElls
where Àg (s) is parameterized Lagrangian multiplier network and is parameterized by for the i-th constraint. Note that TRPO-FAC does not have theoretical guarantee for constraint satisfaction.
TRPO-IPO TRPO-IPO (Liu et al., 2020) incorporates constraints by augmenting the optimization 2 with logarithmie barrier functions, inspired by the interior-point method (Boyd & Vandenberghe 2004). Ideally, the augmented objective is ((Jc((T) d,) = 0 if [c, d; S 0 or -00 otherwise. Intuitively, that enforces the constraints since the violation penalty would be -co. To make the objective differentiable, I(-) is approximated by <(x) = log(- where t > is a hyperparameter. Then TRPO-IPO solves equation 2 with the objective replaced by (TPO(=) = [(T) + (.o,,, -d).
objective in equation
TRPO-IPO does not have theoretical guarantees for constraint satisfaction.
4.4 Hierarchical Safe RL
Safety Layer Safety Layer (Dalal al. 2018), added top of the original policy network, conducts quadratii-poogaamiing bassenasnddd--ddddd---- constrained optimization to project reference action into the nearest safe action. Mathematically:
at cafe arg min 1 511a -lla-a 1/ s.t. a C1 ( d;
(7)
where of of (-1ss), and De. (s.) (s.) Ta, (8 8t) x Ci(st, 0e 81+1) is y parameterized linear model. If there's only one constraint, equation 7 has closed-form solution. solutinn...
USL Unrolling Safety Layer (USL) (Zhang et al. 2022b) proposed to project the reference action into safe action via gradient-based correction. Specifically, USL iteratively updates the learned Qc(s, a) function with the samples collected during training. With step size v and normalization factor 2, USL performs safe
gradient descent as a, = a ref
-/2 8
roj [Qc(s,, aet) d..
00
5
GUARD Testing Suite
5.1
Robot Options
In GUARD testing suite, the agent (in the form of robot) perceives the world through sensors and interacts with the world through actuators. Robots specified through MuJoCo XML files. The suite is equipped with 8 types of pre-made robots that we use in our benchmark environments as shown in Figure 1 The action space of the robots are continuous, and linearly scaled to [-1, +1].
Swimmer consist of three links and two joints. Each joint connects two links to form a linear chain. Swimmer can move around by applying torques on the joints.
Ant is a quadrupedal robot composed of a torso and four legs. Each of the four legs has a hip joint and knee joint; and can move around by applying 8 torques to the joints.
Walker is bipedal robot that consists of four main parts a torso, two thighs, two logs, and two feet. Different from the knee joints and the ankle joints, each of the hip joints has three hinges in the x, and = coordinates to help turning. With the torso height fixed, Walker can move around by controlling 10 joint. torques.
Humanoid is also a bipedal robot that has torso with a pair of legs and arms. Each leg of Humanoid consists of two joints (no ankle joint). Since we mainly focus the navigation ability of the robots in

---

Published in Transactions on Machine Learning Research (03/2024)
Swimmer
Ant
Walker
Humanoid
Hopper
Arm3
Arm6
Drone
Figure 1: Robots of our environments.
designed tasks, the am joints of Humanoid are fixed, which enables Humanoid to move around by only controlling torques.
Hopper one-legged robot that consists of four main parts torso, thigh, leg, and a single foot. Similar to Walker, Hopper move around by controlling 5 joint torques.
Arm3 is designed to simulate fixed three-joint robot arm. Arm is equipped with multiple sensors on each link in order to fully observe the environment. By controlling joint torques, Arm can move its end effector around with high flexibility.
Arm6 is designed simulate robot manipulator with fixed base and six joints. Similar to Arm3, Arm6 can move its end effector around by controlling torques.
Drone is designed to simulate quadrotor. The interaction between the quadrotor and the air is simulated by applying four external forces on each of the propellers. The external forces are set to balance the gravity when the control action is zero. Drone can move in 3D space by applying 4 additional control forces on the propellers.
5.2 Task Options
We categorize robot tasks in two ways: (i) interactive versus non-interactive tasks, and (ii) 2D space versus 3D space tasks. 2D space tasks constrain agents to planar space, while 3D space tasks do not. Non-interactive tasks primarily involve achieving target state (e.g., trajectory tracking) while interactive tasks (e.g., human- robot collaboration and unstructured object pickup) necessitate contact non-contact interactions between the robot and humans or movable objects, rendering them more challenging. On variely of tasks that cover different situations, GUARD facilitates a thorough evaluation of safe RL algorithms via the following tasks. Soe Table 17 for more information.
Goal (Figure 2a] requires the robot to navigate towards series of 2D or 3D goal positions. Upon reaching a goal, the location randomly reset. The task provides a sparse reward upon goal achievement and dense reward for making progress toward the goal.
Push (Figure 2b requires the robot pushing ball toward different goal positions. The task includes a sparse reward for the ball reaching the goal circle and dense reward that encourages the agent to approach both the ball and the goal. Unlike pushing a box in Safety Gym, it is more challenging to push a ball since the ball can roll away and the contact dynamics are more complex.
Chase (Figure 2c) requires the robot tracking multiple dynamic targets. Thosc targets continuously move away from the robot at slow speed. The dense reward component provides bonus for minimizing the distance between the robot and the targets. The targets constrained to circular area. A 3D version of this task is also available, where the targets move within restricted 3D space. Detailed dynamics of the targets is described in Appendix B.5.1
Defense (Figure 2d requires the robot to prevent dynamic targets from entering protected circle area. The targets will head straight toward the protected area or avoid the robot if the robot gets too close. Dense reward component provides a bonus for increasing the cumulative distance between the targets and the protected area. Detailed dynamics of the targets described in Appendix B.5.2
7

---

Published in Transactions on Machine Learning Research (03/2024)
(a) Goal
(b) Push
(c) Chase
(d) Defense
Figure 2: Tasks of our environments.
(a) 3D Hazards
(b) Ghosts
(c) 3D Ghosts
Figure 3: Constraints of our environments.
5.3 Constraint Options
We classify constraints based on various factors: trespassibility: whether constraints are trespassable or non-trespassable Trespassable constraints allow violations without causing any changes to the robot's behaviors, and vice versa. (ii) movability: whether they immovable, passively movable, actively movable; and (ifi) motion space: whether they pertain to 2D or 3D environments. To cover comprchensive range of roistraint configurations, introduce additional constraint lypes via expanding Safety Gym. Please refer to Table 18 for all configurable constraints.
are dangerous 3D areas to avoid. These are floating spheres that are trespassable, and the robot penalized for entering them.
3D Hazards (Figure 3a
Ghosts (Figuree 3b) are dangerous areas avoid. Different from hazards, ghosts always move toward the robot slowly, represented by circles the ground. Ghosts can be either trespassable non-trespassable. The robot is penalized for touching the non-trespassable ghosts and entering the trespassable ghosts. Moreover, ghosts can be configured to start chasing the robot when the distance from the robot is larger than some threshold. This feature logether with the adjustable velocity allows users to design the ghosts with different. aggressiveness. Detailed dynamics of the targets is described in Appendix B.5.3
3D Ghosts (Figure 3el are dangerous 3D areas avoid. These are floating spheres as 3D versions of ghosts, sharing the similar behavior with ghosts.
6 GUARD Experiments
In our experiments, we aim to answer these questions:
Q1 What are the overall benchmark results?
Q2 How does the difficulty of constraints impact the algorithm performance?
Q3 What the detailed performance of different categories of safe RL algorithms?
Q4 How does task complexity impact algorithm performance?
Q5 How does adaptive multiplier impact Lagrangian- based methods?
Q6 How does feasibility projection impact CPO-based methods?
Q7 How does cost dynamics lincarization impact Hicrarchical-based methods?
Q8 What the individual algorithm performance across all tasks?

---

Published in Transactions on Machine Learning Research (03/2024)
6.1 Experiment Setup
In GUARD experiments, our objective is to assess the performance of safe RL algorithms across diverse range of benchmark testing suites. These suites are meticulously designed, incorporating all available robot options as detailed in Section 5.1 and all task options outlined in Section 5.2 Additionally, we offer scamless integration of various constraint options into these benchmark testing suites, allowing users to select desired constraint types, numbers, sizes, and other parameters. Considering the diversity in robots, tasks, ronst types, and difficulty levels, we have curated 72 test suites. These predefined benchmark testing suites follow the format [Task)_ [Roo_t[[R______ [Robot]](Constraing {Constraint Number) (Constraint Type}. For comprchensive list of our testing suites, please refer to Table 20
Nole that the ladder of construint difficully levels can be readily established by introducing varying numbers, sizes, and types of constraints, as detailed in [Section 4.2, (Ray et al. 2019a))/.
Remark 1.
Comparison Group The methods in the compari- group include all methods in GUARD Safe RL Li-
Rlacade
1.0
braray: (i) unconstrained RL algorithm TRPO (Schulman
L2
ct al. 2015 (ii) end-to-end constrained safe RL algo- rithms CPO Achiam et al. 2017), TRPO- PRPO--aaganggan (Bo-
6.8
10
0,
0os
hez et al. 2019). TRPO-FAC (Ma et al. 2021 TRPO- IPO (Liu et al., 2020), PCPO (Yang et 2020b and (iit) hierarchical safe RL algorithms TRPO-SL SL (TRPO-Safety Layer) (Dalal et al. | 2018). TRPO-USL (TRPO-Unrolling Safety Layer) (Zhang et al. 2022a). For hierarchical safe
0.4
0.2
02
0.0
TRPO Lagrampan
KCR
TRPo
1590-55
0.0
TRPO Lagraniaan
PCtO
18P0-00
TRPO
TRPD
RL algorithms, we incorporate a warm-up phase (consti- Figure 4: constraint difficulty ablation study with tuting 1/3 of the total epochs) dedicated to unconstrained TRPO training. The data gencrated during this phase is utilized to pre-train the safety critic for subsequent epochs. The larget cost for all safe RL methods is set to zero, aligning with our objective of achieving zero violations. To ensure consistency, shared configurations, including hidden layers, learning rate, and target KL, are uniformly applied across all methods. Simultancously, unique parameters such as Lagrangian learning rate, IPO parameter, and Warmup ratio are fine-tuned to optimize the performance of each respective method. Further details are provided in Table 19
Goal Point 8(Constraint)
6.2 Evaluating GUARD Safe RL Library and Comparison Analysis
Overall Benchmark Results The summarized results can be found in Tables 21 to 25 and the learning rate curves are presented in Figures [4 to 18 In Figure 5] we select 8 sets of results to demonstrate the
performance of different robots, tasks and constraints in GUARD. At a high level, the experiments show that all methods can consistently improve reward performance.
Different methods have different trade-offs between rewards and cost. When comparing constrained RL methods to unconstrained RL methods, the former exhibit superior performance in terms of cost reduction. By incorporating constraints into the RL framework, the robot can navigate its enviromment while minimizing costs. This feature is particularly crucial for real-world applications where the avoidance of hazards and obstacles is of utmost importance. However, current safe RL algorithms (i.e. CPO, PCPO and Lagrangian methods) are hard achieve zero-violation perfor- mance even when the cost. threshold is set zero. Compared with them, hierarchical RL methods (i.e., TRPO-SL and TRPO-USL) perform better at cost reduction. Nevertheless, although these methods excel at minimizing costs, they may sacrifice some degree of reward atlainment in the process.
0Hazerds
BGhosts
50
40
30
20
10
Swimmet
Am3
Orane
Hopper
Humannii
Ant Walker
Figure 6: Raw TRPO cost score in Goal_[Robot] (ilobbot__(Constraini with different constraint difficulty.
Impacts of Constraints Difficulty To gain an intuitive understanding of constraint difficulty, we present visual comparison in Figure The
9

---

Published in Transactions on Machine Learning Research (03/2024)
Seuard Perfrrunce
Sow3rr Port mancca
3.0
2.5
2.0
1.3
1.4
-500
TEPO TE5-3C
1110 19
TIQLUS teo.t
TTA0 1ER0.LL
TVO015
1300.L
ToaalEriintereets
1.
TeRO
19P0
IARO
XL
Cost Paroorarr
3.0
1.26
4.00
0.75
9.6
0.50
8.4
0.
D3
0.25
0.00
TeaaEnvintorress
100
1+-3
1802
1892
Cost Fare Ferformnn
1.0
0.0
0.
Coal Narta The
20
0.4
4.5
3.0
Tralfruinterctss
(a) Goal Loal_PPontt__e Poinntfflaazads Bllazards
(b) Goal_ Walker_ Biazzrds
(c) Goal Arm3 _ Bllazards
(d) Goal Goal_Drone, ___n__ron______________ -iiazadds
1e1
Sewend Performnce
5OW3d Performance
12
3.9
Demaar Derformance
0.0
mancce
3.8
TTC Logrongion
1753-00
3.0
TEX--0
P00
TE00 vagltt
1270-00
1000
TIN proggen
TES
701
-22
Testeeniittrreess
10
16
16]
12
341
341
1kA Lagusggar
10 Lagargas
tapo-m mo
KF2
Tero uagrangan
100
Cost Perfommnne
9
Cuss Pnrformanca
3.0
Performence
Cont Wentrnment
0.6
0.4
N
9.4
0.2
Realttninttects
Jac
Ie-2
3.1
Ie-3
Legungge
19P0
KF5
Cosk Performano
2.0
9.8
25
8.8
Cost Bare
64
(e) Www__ooooonttseeeeeeeeee
(f) Lwteauhoottteeeeeeeeaeaaeee 8Hazards
(g) Sbaas__Pont__HHaeee
(h) Beffense_Pont__Baazee
Figure 5: Comparison of results from four representative tasks. (a) to (d) cover four robots on the goal task. (e) shows the performance of a task with ghosts. (f) to (h) cover three different tasks with the point robot.
10

---

Published in Transactions on Machine Learning Research (03/2024)
figure illustrates the raw cost comparison for TRPO across different robot options on the testing suite where we vary 8(Constraint} from 8Hazards to 8Chosts. Tt is essential to note that Hazards and Ghosts share the same cost computation rule, with the distinction that Ghosts adversarially move towards the robot. The comparison reveals some significant increase in raw TRPO cost when transitioning from Hazards to 8Ghosts for every robot, option, indicating the escalating challenge posed to safe RL algorithms.
Goal_ [RRooot]_[Coonsssstss__
Roward
CP hi
Roward
c10 Herric be
To assess the impact of constraint difficulty on the per- formance of various safe RL algorithms, analyze the performance changes across different algorithms using lesting suites. Specifi-
Hé sould
as
05
the Goal_Point, Baaa_ooonttBCConsrraiinnt cally, we vary 8(Constraint) from 8Hazards to 8Chosts, with the former representing an casier constraint where all hazards are static, and the latter indicating more challenging constraint where all hazards are adversarially moving towards the robot. To ensure fair comparison across diverse tasks, we report the normalized reward and normalized cost for each algorithm using the following metric:
Cost
Copt
(a) Gual_Poite 8Hazards.
(c) Goal_Ant 2att__8aaardss.
Logen sagargua
Cost 1000
Doot
Costrato
(b) Goal Point, -Ghosss..
(d) Goal Ant 8Ghosts.
rewardnormatzad = min rewardalgoritae rewardTRPO 1)
(8)
Figure 7: Comparison of performance of different cat- egories of algorithms in four representative test suites across low-to-high dimensional robots and easy-to-hard constraints. The scores of reward, cost and cost rate are normalized with respect to TRPO. Red triangle serves as the baseline TRPO performance. A higher normalized reward score and lower normalized cost/cost rate scores indicate better performance along each axis.
cost coslalgorithm
(9)
COSUTRPO
where rewardaormaized are obtained from Tables to 25 Before applying Equation ( in cases where negative values exist for rewardnormaliedd in the results, we shift
to
all values to be above zero with respect to the minimum rewardhormatized observed within the same experiment.
The comparative results are succinetly presented in Figure 4 Notably, as the constraint difficulty transitions from an casier to a more challenging level, majority algorithms within the Safe RL Library showcase safer policy learning behaviors. Additionally, they manage to maintain roughly the same, and in some instances, achieve higher reward performance. This observation underscores two kcy findings: (i) safe RL algorithms exhibit resilience to the increased difficulty of constraints, and (ii) heightened constraint difficulty serves as a more effective metric for distinguishing safety performance. Notably, the cost performance gap between TRPO-Iagrangian and TRPO-IPO is magnified in the Ghost environment, where more sophisticated safe policy is in need to avoid adversarial hazards.
Characteristics of Different Categories of Safe RL Algorithms Within the Safe RL Library, three primary categories of safe RL algorithms exist: (i) Lagrangian- based methods (TRPO-Lagrangian, TRPO-FAC, TRPO- IPO), (ii) constrained policy optimization- methods (CPO, PCPO), and (iit) hierarchical brasee methods (Safe Layer, UST.). To investigate the performance character- istics inherent in each category, we carefully select four representative testing suites spanning high/low dimen- sional robots and pasy/hard constraints. Subsequently, we chart the algorithm-wise averaged TRPO normalized re- ward, cost, and cost rate. For example, the algorithm- wise averaged score for hierarchical-basse methods is computed by taking the mean across Safe Layer and USL. Additionally, the TRPO normalized cost rate is defined as:
Logrrrgen besed Heporchoo baies
Lograngan besed Hkmerchkcal caved
0.
Figure S: TRPO normalized performance metrics (Reward and Cost.) of three aigorithm categories across varied diffi culty levels in Goal (Robot) 8Hazards
cost raleatgoritho
cost ralenormalized
(10)
cost. roteT RPO
11

---

Published in Transactions on Machine Learning Research (03/2024)
TARO TRAO
1.0
1.0
The summarized comparison results are presented in fig. 7 Generally, in terms of safety performance, Lagrangian- based methods demonstrate the highest efficacy, followed by CPO-based methods, while Hierarchical-basee meth- ods exhibit. comparatively lower performance. Several key observations emerge: (i) Lagrangian-base based mothods priori- tize safety performance and are willing to make concessions in reward performance when necessary, evidenced by significantly lower reward performance in high-dimensional tasks. This behavior of Lagrangian- methods have also been witnessed in [Figure 9, Doggo experiments, (Ray et al. 2019a)] (ii) CPO-based methods tend to maintain a high standard of reward performance akin to unconstrained RL, but this strategy compromises safe behavior, particularly in high high-dimensional tasks. This obscrvation exactly matches the reported results [Figure 8, Figure 9 (Ray ct al. 2019a)] [Figure 6, (Yang) et al. 2023)]- ). (iii) Hierarchical-basee methods only demonstrate effectiveness in low-dimensional systems, struggling to learn reasonable safe behavior in high-dimensional systems. This limitation is attributed to the escalating complexity of cost function dynariic in high-dimensional systems, posing challenges for effective cost function approximation. This finding aligns with the results reported in [Figure 3, Safety Layer, Recovery
6.8
c.
2.4
0.4
0.2
0.2
0.0
Hopper
Naster
Figure TRPO normalized performance metrics (Reward and Cost) of Lagrangian and FAC across varied difficulty levels Goal {Robot] 8Hazards
RI, (Zhang et al. 2023 [Section 6.1, (Zhao et. al. 2021
Effects of Task Complexity on Various Categories of Safe RL Algorithms To examine the impact of increased task complexity, particularly in the context of higher degrees of freedom (DOF) in robots, we summarize the TRPO normalized performance motrics (reward and cost) for three algorithm categories across various difficulty levels in the Goal [Robot} 8Hazards testing suites, with an incremental increase in robot DOF, in Figure The performance characteristies of different algorithm categories exhibit interesting variations with heightened task complexity.
Surprisingly, there is no discernible trend in reward and cost performance across all algorithm categories as task complexity increases. However, Lagrangian and Hierarchical-basee methods each align better with specific robot types. (i) For Arm robots, Lagrangian-based methods struggle to learn reward, whereas Hierarchical-based methods perform well on both reward and safety. (fi) In the case of /lggd/ robots, Tagrangian- based methods maintain a consistent, albeit mediocre, reward performance, coupled with optimal safcty performance. Converscly, Hicrarchical-basee based mothods struggle to learn safe policy, particularly on the Ant robot.
Regarding CPO-based methods, their reward performance remains consistently high even with increased task complexity. However, although CPO-based methods exhibit effective learning of safe policies with low-dimensional robots, they encounter difficulties in learning optimal safe behavior as task complexity exceeds a certain threshold.
Impacts of Adaptive Multiplier in Lagrangian-based Methods An essential differentiator between TRPO-FAC and TRPO-Lagrangian lies in the incorporation of a multiplicr network, i.c. an adaptive multiplior. To gain a comprehensive understanding of this technique, we compare the normalized performance metrics on Goal Goal_[Robot] [Robot]]SHaaards 8Hazards testing suites, both with and without this feature, as illustrated in Figure g It is evident that the introduction of the adaptive multiplier results in consistently stable and favorable performance across various difficulty levels of tasks. This reward behavior of adaptive multiplier has been observed in [Fig 2, (Ma et al. ] However, safety performance, leading to higher costs, particularly evident in Swimmer and Ant. This cost behavior of adaptive multiplicr has been reported in [Fig 3, FAC w/ Φo, (Ma et al. 2022
a trade-off, the adaplive multiplier tends to compromise
Impacts of Feasibility Projection in CPO-based Methods The policy rule of PCPO enhances CPO by projecting the reward-oriented policy back into the constraint set,, ensuring a feasible policy update at every iteration (feasibility projection). To assess the impact of this technique, we conduct a comparison of normalized performance metrics on Goal [Ral_[Rooot], [Robott]SHazards 8Hazards testing suites, both with and without this feature, as depicted in Figure [0] The figure indicates that feasibility projection is potent in trading off performance for significantly safer behavior in specific tasks, such the Drone task. This behavior
12

---

Published in Transactions on Machine Learning Research (03/2024)
of feasibility projection has been observed in [Fig 4(e), (Yang et al. 2020a However, this may not be universally applicable, as feasibility projection often leads to slightly less safe behavior and, in some cases, adversely affects reward performance without corresponding improvements in safety, as observed in the Arm6 scenario.
Impacts of Linearized Cost Dynamics Hierarchical methods offer the explicit projection of an unsafe reference action into safe action set, a process that involves deter- mining the cost given state-action pair. While one ap- proach involves directly learning the cost function, seen in USL, SafeLayer streamlines the process by linearizing the cost function dynamics and focusing solely on learning the gradient. To assess the effectiveness of this method, we conduct a comparison of normalized performance met- rics on Goal_ Goal_(Robot] [Roobot]_SHazarddd SHazards testing suites, both with and without this feature, as depicted in Figure [T The results reveal that the linearization is highly effective in achieving robust safe behavior in low-dinensional robot tasks, particularly evident in the cases of Drone and Arm3. However, the linear approximation of the cost function in SafeLayer becomes less accurate in scenarios with highly nonlinear dynamics, such as the complex Ant robot, leading to a significant increase in overall costs. Therefore, Linearization emerges as a powerful technique for tasks with low complexity, especially those involving low-dimensional robots. The similar failure of cost dynamics lincarization on complex tasks has been observed in [Fig 6 (c), Fig (d), (Zhao et al. 2021
CR
1.0
6.8
0.8
c.6
0.4
0.2
Hepper
Nasder
Figure 10: TRPO normalized performance metrics (Re- ward and Cost) of CPO and PCPO across varied difficulty levels Goal {Robot]. 8Hazards
Comprehensive Evaluation of Algorithm Perfor- mance Across All Tasks Lastly, for a holistic under- standing of the performance exhibited by each algorithm within the Safe RL Library across all tasks within the GUARD Testing suites, we present spider plots in Fig- ure 12 These plots illustrate the TRPO normalized re- ward, cost, and cost rate for each algorithm across five task options, with all performance metrics averaged over all available robot options. TRPO is utilized as the baseline in this figure, emphasizing its focus on maximizing reward performance.
TRAS-S
TRPO
1.0
0.8
Gevero
0.0
Amb Humanood
Hader
Figure 11: TRPO normalized performance metrics (Re- ward and Cost) of USL and SafeLayer across varied diffi- culty levels in Goal [Ro__t[o___________ si___Haaardsss
Within the category of CPO-based methods: (i) CPO maintains reward levels comparable to TRPO across all tasks but struggles to learn safe policy in the Chase task. (ii) PCPO exhibits a lower performance in both reward and safety aspects, particularly in Chase and Defense tasks.
Within the category of Lagrangian- methods: (i) Lagrangian maintains a balance between good reward and cost across all tasks, although its reward performance is not exceptional. (fi) FAC excels in achieving higher reward across all tasks, with slight sacrifice in safety performance in all tasks except Defense. (iii) IPO performs the worst in both reward and safcty metries, although it still manages to learn viable safe policies for all tasks.
Within the category of Hierarchical- based methods: (i) USL performs admirably in achieving satisfactory rewards for all tasks but only Iearns modiocre safe policies and struggles with the Chase task. (ii) SafeLayer demonstrates poor performance in reward performance and consistently fails to learn safe policies in most. tasks. However, it excels in achieving satisfactory safety performance, particularly in the Defense task.
7 Conclusions
This paper introduces GUARD, the Generalizod Unified SAfe Reinforcement Learning Development Bench- mark. GUARD offers several advantages over existing benchmarks. Firstly, it provides a generalized framework
13

---

Published in Transactions on Machine Learning Research (03/2024)
fral devowes
Aourd
Oug Evorsk
Fower
Seal Bhowwis
Roverd
fonas
1.A.
Defevs 26 conky
Oher SwORTS
Delone Meears
Noc Shapars
Deless Meorts
Chani Marrnd
(a) TRPO
(c) PCPO
(e)
TRPO-FAC
(g) TRPO-SL
Frail deoooees
Aourd
Fowrd
Seal Shozwiis
Reverd
Frait Busraoi
Fones
100
Deless 26 conk.
Chenc Shorts
Seear
Detrrs Weork
Cheii Maarsh
(b) CPO
(d) TRPO-Lagrangian
(f) TRPO-IPO
(h) TRPO-USL
Figure 12: Algorithm performance different tasks. The scores of reward, cost and cost rate normalized with respect to TRPO. The score for each task is averaged over all robots. Red houndary serves as the baseline TRPO performance. A higher normalized reward score and lower normalized cost/foost rate scores indicate better performance along each axis.
with a wide range of RL agents, lasks, and constraint. specifications. Secondly, GUARD has self-contained implementations of comprehensive range of state-of-the-art safe RL algorithms. Lastly, GUARD is highly customizable, allowing rescarchers to tailor tasks and algorithms to specific needs. Using GUARD, we present a comparative analysis of state-of-the-arl safe RL algorithms across various lask settings, establishing essential baselines for future rescarch.
Future work In our future endeavors, aim to work the following expansions: (i) To enhance the case of deploying Reinforcement Learning (RI) methods on real robots, we are actively incorporating additional realistic robot models into GUARD. (ii) The current GUARD tasks primarily focus on robot locomotion; however, we are planning to broaden the spectrum of available task types to empower users with a more extensive testing suite, including options such as speed control and contact-rich safety. (iii) Lastly, we plan to extend the range of safe RI. librarics to include the latest algorithms such as off-policy methods.
References
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pp. 22 31. PMLR, 2017.
Forest Agostinelli, Stephen McAleer, Alexander Shmakov, and Pierre Baldi. Solving the rubik's cube with deep reinforcement learning and search. Nature Machine Intelligence, pp. 1 1-8, 2019.
Steven Bohez, Abbas Abdolmaleki, Michael Neunert, Jonas Buchli, Nicolas Hoess, and Raia Hadsell. Value constrained model-free continuous control. arXiv preprint orXiv:1902. 04623, 2019.
Stephen P Boyd and Lieven Vandenberghe. Conver optimization. Cambridge university press, 2004.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.0111 015400 2016.
Lukas Brunke, Melissa Greeff, Adam W. Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P. Schoellig. Safe learning in robotics: From learning-base based control safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5(1):411-444 2022.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. httphttpp//aaxw...tr////////////
Rui Chen, Alvin Shek, and Changliu Liu. Robust and context-aware real-time collaborative robot handling via dynamic gesture commands. JEFE Rohotics and Aulomation Letters, 2023.
14

---

Published in Transactions on Machine Learning Research (03/2024)
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. CoRR, abs/1801 118/1180.08777, 2018.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John
Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https://github..co [genail/baselines] 2017.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pp. 1329-1338. PMLR, 2016.
Gabriel Dulac-Arnold, Nir Levine, Daniel J. Mankowilz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learning. 2020.
Benjamin Ellenberger. Pybullet gymperium.
gym 2018 2019.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437 1480, 2015.
Sven Gronauer. Bullet-safety-gym A framework for constrained reinforcement learning. Technical report, TUM Department of Electrical and Computer Engineering, Jan 2022.
Shangding Gu, Guang Chen, Lijun Zhang, Jing Hou, Yingbai Hu, and Alois Knoll. Constrained reinforcement learning for vehicle motion planning with topological reachability analysis. Robotics, 11(4), 2022a.
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXie preprint ar Xrv:22205. 10330, 2022b.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines.
2018.
Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momchev, Danila Sinopalníkov, Piotr Stańezyk, Sabela Ramos, Anton Raichuk, Damien Vincent, Téonard Hussenot, Robert Dadashi, Gabriel Dulac-Arnold, Manu Orsini, Alexis Jacq, Johan Ferret, Nino Vieillard, Seyed Kamyar Seyed Ghasemipour, Sertan Girgin, Olivier Pielquin, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Abe Friesen, Ruba Haroun, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcchre, Tom Le Painc, Srivatsan Srinivasan, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXi::2006.009: 2020. URL httts::/aaxiis org/ hangg/abs//ne abs/200// 2006 00979
David Isele, Alireza Nakhaei, and Kikuo Fujimura. Safe reinforcement learning on autonomous vehicles. arXiv preprint arXiv: 1010. 1910.00399 2019.
Jiaming Ji, Borong Zhang, Xuchai Pan, Jiayi Zhou, Juntao Dai, and Yaodong Yang. Safety-gymnasium. Gilflub reposilory, 2023.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEFE Transactions on Intelligent Transportation Systems, 23(6):4909 4926, 2022.
Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1288-1274, 2013.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning (ICMT., 2018.
15

---

Published in Transactions on Machine Learning Research (03/2024)
Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-point policy optimization under constraints. In Proceedings of the AAAT conference on artificial intelligence, volume 34, pp. 4940-4947, 2020.
Haitong Ma, Yang Guan, Shegubo Eben Li, Xiangleng Zhang, Sifa Zheng, and Jianyu Chen. Feasible actor-
critic: Constrained reinforcement learning for ensuring statewise safety, arXiv preprint orXv::1005.1069. 2021.
Haitong Ma, Changliu Liu, Shengbo Eben Li, Sifa Zheng, and Jianyu Chen. Joint synthesis of safety certificate and safe control policy using constrained reinforcement learning. In Learning for Dynamics and Control Conference, pp. 97 109. PMLR, 2022.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglon, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXit:1312.000 2013.
OpenAI, :, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d. O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schncider, Szymon Sidor, Ilya Sutskever, Jic Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arxiv preprint 1rXiv:1912.066 2019.
lan Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Sxepesvári, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado van Hasselt. Behaviour suite for reinforcement learning. In International Conference on Learning Representations, 2020.
Matthias Plappert. keras-rl. https://githii http:/hhwi.....e/ooetsttt/aaaaaa 2016.
Tvaylo Popov, Nicolas Heess, Timothy Lillicrap, Roland Hafner, Gabriel Barth-Maron, Matej Vecerik, Thomas Lampe, Yuval Tassa, Tom Erez, and Martin Riedmiller. Data-efficient deep reinforcement learning for dexterous manipulation. arXiv preprint 17X:::1004.0000 2017.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8 8, 2021. URL. httt:://jmle http://rrrrr/pppppe///////00000 htm4.mmmmm.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint Xr::::1910.01 7(1):2, 2019a.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. CoRR, abs/ hass/1910.01008 2019b.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889 1897. PMLR, 2015.
Alvin Shek, Rui Chen, and Changliu Liu. Learning from physical human feedback: An object-centric one-shot adaptation method. arXiu preprint NNN::200304455 2022.
David Silver, Thomas Hubert, Julian Schrittwieser, Toannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Gracpel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):11400 1144, 2018.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa, control: Software and tasks for continuous control. Software Impacts, 6: 6:100022, 2020.
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojtek Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, Timo Ewalds, Dan Horgan, Manuel Kroiss, Ivo Danihelka, John Agapiou, Junhyuk Oh, Valentin Dalibard, David Choi, Laurent Sifre, Yury Sulsky, Sasha Vezhnevets, James Molloy, Trevor Cai, David Budden, Tom Paine, Caglar Gul- cehre, Ziyu Wang, Tobias Pfaff, Toby Pohlen, Dani Yogatama, Julia Cohen, Katrina McKinney, Oliver
16

---

Published in Transactions on Machine Learning Research (03/2024)
Smith, Tom Schaul, Timothy Lillicrap, Chris Apps, Koray Kavukcuoglu, Demis Hassabis, and David Silver. AlphaStar: Mastering the Real-Time Strategy Game StarCraft II. https://deepmind.. htt.ddeemmind.coom...oooooo suarttee--aame-ssttrrratt--- 2019.
Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267):1 6, 2022. URL
http://apper///////11122222 httml
Tsung- Yen Yang. Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Projection-based constrained policy optimization. CoRR, abs/2010.03152, 2020a. URL httpss///axxv.oor//bb///11....11
Tsung-Yer Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. arXiv preprint arXtt:2010.00111 2020b.
Yujie Yang, Yuxuan Jiang, Yichen Liu, Jianyu Chen, and Shengbo Eben Li. Model-free safe reinforcement learning through neural barrier certificate. IEEE Robotics and Automation Letters, 8(3):1295- 1302, 2023.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement lcarning. In Conference on Robot Learning (CoRL), 2019.
Zhaocong Yuan, Adam W. Hall, Siqi Zhou, Lukas Brunke, Melissa Greeff, Jacopo Panerati, and Angela P. Schoellig. controontrl A unified benchmark suite for safe learning- based control and reinforcement learning in robotics. IEEE Roboties and Automation Letters, 7(4):11142-11148) 2022. doi: 10.1109/T.RA.
2022.3196132.
Linrui Zhang, Qin Zhang, Li Shen, Bo Yuan, and Xueqian Wang. Saferl-kit: Evaluating efficient reinforcement learning methods for safe autonomous driving. arXiv preprint arXiv:2206.0888 085228 2022a.
Linrui Zhang, Qin Zhang, Li Shen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Evaluating model-free reinforcement learning toward safety-critical tasks. arXiv preprint wxXiv:2212.0572p 2022b.
Linrui Zhang, Qin Zhang, Li Shen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Evaluating model-free reinforcement learning toward safety-critical tasks. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 37, pp. 15313 15321, 2023.
Weiye Zhao, Tairan He, and Changliu Liu. Model-free safe control for zero-violation reinforcement learning. In 5th Annual Conference Robot Learning, 2021.
Weiye Zhao, Tairan He, Rui Chen, Tianhao Wei, and Changliu Liu. State-wise safe reinforcement learning: A survey. The 32nd International Joint Conference on Artificial Intelligence (IJCAI), 2023.
17

---

Published in Transactions on Machine Learning Research (03/2024)
A
Appendix
B Environment Details
B.1 Observation Space and Action space of different robots
The action space and observation space of different robots are summarized in Tables to 16
18

---

Published in Transactions on Machine Learning Research (03/2024)
angular velocity (rad/ / s)
free magnetic flux density (T)
IIf Inf jointpos motor2 rot hinge jointvel motorl rot. hinge jointvel motor2 rot hinge angular velocity (rad/s, s))
angular velocity (rad/s s)
acceleration (m/ /s2)
torque (Nm) torque (Nm)
velocity (m/s)
angle (rod)
angle (rod)
force( (N)
forco (N)
force(M) (N)
force( ())
Unit
Unit
Joint
hinge
hinge
Joint
lIf Inf jointpos motorl rot hinge
freee accelerometer
free
free
Name XML
motorl rot
motor2. rot
Name in XML
velocimeter
gyro magnetomelen Inf touch pointl
Inf touch point2
Inf touch point3
touch point4
Max
Max
1
1
Table 2: Observation space of Swimmer
Inf
Inf
Inf
Inf
Inf
Inn Tnf
-lf Inf
Table Action space of Swimmer
Min
Min
IInf
-Inf
IIn
-Tnl
-1
0
3-axis linear acceleration of the torso (including gravity)
3-axis angular velocity velocity of the torso
Torque applied on the first rotor Torque applied on the second rotor
3D magnetic flux vector at the Lorso Contact force at the first tip Contact force at the first rotor Contact force at the second rolor Contact force at the second tip Angle of the first rotor Angle of the second rotor Angular veloeity of the first rohor
Angular velocity of the second rotor
3-axis linear velocity of the torso
Obscrvation
Action
Num
Num
0/1/2
3/4/5
6/7/8
9 9/10/11
12
13
14
16
17
18
19
19

---

Published in Transactions on Machine I
Learning Research (03/2024)
free free free magnetic flux density (T)
angular velocity (rad/a /s)
angular velocity (rad/ ss)
angular velocity (rad/s //d/))
angular velocity (rad s)
angular velocity (rad/ s)
angular velocity (rad/s /s)
angular velocity (rad/ ss)
acceleration (m/ /s/)
angular velocity (rad/
angular velocity (rad/
velocity (m (m/s) /s)
torque (Nm
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
force [N)
force( [))
force( (N)
force( (N)
force(N) (N)
force(N)
forco(N)
force(N) (N)
Unit
Unit
Joint
hinge
hinge
hinge
hinge
hinge
hingo
hinge
hinge
Joint
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
free accelerometer velocimeter gyro magnetometer touch ankle la
Name in XML
Name XML
touch. ankle 2a
touch ankle 3a
touch. ankle 4a
touch ankle 1b
Inf touch, ankle 2b
Inf touch ankle 3b
touch ankle 4b
jointpos hip 1
Inf jointpos hip 3
Tnf joint hip 4
Inf jointpos ankle 1
Inf jointpos ankle 2
Inf jointpos ankle
Inf jointpos ankle 4
Inf jointvel hip 1
Inf jointvel hip 2
Inf jointvel hip 3
Inf jointvel hip 4
Tnf jointvel ankle 1
Inf jointvel ankle 2
Inf jointvel ankle 3
jointvel ankle
Inf jointpos hip
ankle 1
ankle 2
hip 3
ankle 3
hip 4
ankle 4
hip
hip
Max
1
1
1
1
1
1
Max
Inf
Inf
Inf
Inf
Inf
Inf
Inf
Inf
Tnf
Inf
Inf
IIf Inf
Table Observation space of Ant
Table 3: Action space of Ant
Min
-1
-1
-1
-1
-1
-1
Min
-Inf
IIn
-Inf
-Inf
0
0
0
-Tnf
Inf
Inf
-nn
-nf
Inn
Inf
-nf
IIn
IIf
Inf
Jnf
Inf
-Inf
Torque applied on the rotor between the torso and front left hip Torque applied on the rotor between the front left two links Torque applied on the rotor between the torso and front right hip Torque applied on the rotor between the front right two links Torque applied on the rotor ween the torso and back left hip Torque applied on the rotor between the back left two links Torque applied on the rolor belween the torso and back right. hip Torque applied on the rotor between the back right two links
3-axis linear acceleration of the torso (including gravity)
3-axis angular velocity velocity of the torso
3D magnetic flux vector at the torso Contact force at the front. left ankle Contact force at the front right ankle Contact force at the back left ankle Contact force at the back left ankde Contact force at the end of the front. left leg
Contact force at the end of the front right leg
Contact force at the end of the back left log Contact force at the end of the back left leg
Angular velocity of the front left. ankle Angular velocity of the front right ankle
Angular velocity of the back left ankle Angular velocity the back right ankle
Angle of the back right ankle Angular velocity of the front left. hip
Angular velocity of the front right hip Angular velocity of the back left hip Angular velocity of the back right hip
3-axis linear velocity of the torso
Angle of the front. left. hip
Angle of the front right hip
Angle of the back left hip
Angle of the back right. hip
Angle of the front left ankle
Angle of the front right ankle
Angle of the back left ankle
Observation
Action
9/10/11 11
Num
Num
0/1/2
3/4/5
6/7/8
0
2
3
4
5
6
7
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
20

---

Published in Transactions on Machine Learming Research (03/2024) (
angular velocity (rad/s s/ s)
free free magnetic flux density (T)
angular velocity (rad) s)
angular velocity (rad/s ss)
angular velocity (rad/s s)
angular velocity (rad/ /s)
angular velocity (rad/s s)
angular velocity (rad/s (s)))
angular velocity (rad/ /s)
angular velocity (rad/s s))
angular velocity (rad/ /s
acceleration (m/ /s2)
angular velocity (rud/s,
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
free velocity (m/s)
angle (rad)
angle (rad)
angle (rad)
angle (rod)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
force( ())
force(N)
Unit
Unit
Joint
hinge
hingo
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
Joint
Inf jointpos right hip x hinge
-Tnf Inf jointpos right hip w. hinge
-if Inf jointpos right hip y hinge
Inf jointpos right, leg hinge
Tnf jointpos right. foot. hinge
Inf jointpos left hip x hinge
jointpos left hip z hinge
Inf jointpos left hip y hinge
hinge
hinge
Inf jointvel right. hip x hinge
jointvel right. hip 2 hinge
jointvel right, hip y hinge
jointvel right. leg hinge
jointvel right foot hinge
Inf jointvel left hip x hinge
jointvel left hip 2 hinge
-In Inf jointvel left. hip y hinge
hinge
IIf Inf jointvel left foot hinge
free accelerometer
Name XML
right. hip x
right. hip Z
right hip y
1 right. leg joint right. foot. joint. left. hip_x x
left hip tip_z z
left hip y
left leg joint
left. foot. joint
Name in XML
velocimeter
gyro mnagneloomeerr touch right foot
touch left foot
jointpos left leg
Inf jointpos left foot.
-hn Inf jointvel left. leg
Max
Max
1
1
1
Table 6: Observation space of Walker
Inf
Inf
Inf
Tnf
Inf
Inf
Inf
Inf
Inf
Inf
Tnf
-Iff Inf
IIf Inf
Table 5: Action space of Walker
Min
-1
-1
-1
-1
-1
Min
IInf
-Inf
IIn
-Tnl
-Inf
IIf
Tnn
-nf
Jnn
Inf
Inf
IIn
Inf
Inf
-In
-Inf
Torque applied on the rolor between torso and the right. hip (x-coordinate)
Torque applied on the rotor between torso and the right hip
Torque applied on the rotor between torso and the right hip (y-coordinate)
Torque applied on the rotor between torso and the left hip (x-coordinate)
Torque applied on the rotor between torso and the left hip o-coordinate)
Torque applied on the rotor between torso and the left hip (y-coordinate)
3-axis linear acceleration of the torso (inchuding gravity)
3-axis angular velocity volocity of the torso
Angular velocity of the right hip (x-coordinate) Angular velocity of the right hip (z-coordinate
Angular velocity of the right hip (y-coordinato)
Angular velocity of the left hip (c-coordinate)
Angular velocity of the left hip (z-coordinate)
Angular velocity of the left. hip (y-coordinate)
Torque applied on the right log rotor
Torque applied on the right foot, rolor
Torque applied on the left leg rotor
Torque applied on the left foot rotor
3-axis linear velocity of the torso
3D magnetic flux vector at the Lorso Contact force at the right foot Contact force at the left foot Angle of the right hip (x-ooordinate) Angle of the right hip (4-coordinate)
Angle of the right hip (y-ooordinate)
Angle of the left hip (x-coordinate)
Augle of the left hip (z-coordinate)
Angle the left hip (y-coordinate)
Angular velocity of the right. leg Angular velocity of the right foot
Angular velocity of the left leg
Angular velocity of the left foot
Angle of the right leg
Angle of the right foot.
Angle of the left log
Angle of the left foot
(z-coordinate)
Obscrvation
Action
9/10 9/10/11
Num
Num
0/1/2
3/4/5
6/7/8
3
4
5
8
9
12
13
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
21

---

Published in Transactions on Machine Learning Research (03/2024)
free angular velocity (rad/ rrad/s) s)
free magnetic flux density (T)
angular velocity (rad/ ss)
angular velocity (rad/s /
angular velocity (rad s)
angular velocity (rad/ s)
angular velocity (rad/a /s)
angular velocity (rad/ s)
free acceleration (m/s) /2)
angular velocity (rad)
angular velocity (rad
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
velocity (m/s)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
forco( (N)
force(N) (N)
Unit
Unit
Joint
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
Joint
-Tnf Inf jointpos right. hip x hinge
-In Inf jointpos right hip z hinge
Inf Inf jointpos right. hip y hinge
-Tn Tnf jointpos right. knee hinge
Inf jointpos left hip x hinge
Inf jointpos left hip z hinge
Inf jointpos left hip y hinge
Inf joint loft knoe hinge
jointvel right. hip hinge
jointvel right, hip z hinge
jointvel right. hip y hinge
jointvel right knce hinge
Inf jointvel left. hip x hinge
Inf Inf jointvel left hip z hinge
-Inf Inf jointvel left hip y hinge jointvel left knee hinge
free
Tnf velocimeter gyro magnetometer touch right foot
Name in XML
right hip x
right. hip 2
right. hip y
right knee
left hip X
left hip 2
left. hip y
left knee
Name in XML.
accelerometer
touch left foot
Max
Max
1
1
Table 8: Observation space of Homanokd
Inf
Inf
Inf
Inf
Inf
Inf
Inf
Inf
Inf
IIf Inf
Table 7: Action space of Ihuaanood
Min
-1
-1
-1
-1
-1
Min
-Inf
IIn
-Tnl
-Inf
Inf
Jnn
Inf
Inf
Inn
Inf
Inf
Jní
Torque applied on the rotor between torso and the right hip (x-0oordinate)
Torque applied on the rotor between torso and the right hip z-coordinate)
Torque applied on the rotor between torso and the right, hip g--coordinate)
Torque applied on the rotor between torso and the left hip (X-coordinate)
Torque applied on the rotor between torso and the left hip (x-coordinate)
Torque applied on the rotor between torso and the left hip (y-coordinate)
3-axis linear acccleration of the torso (including gravity)
3-axis angular velocity velocity of the torso
Angular velocity of the right hip (x-coordinatee Angular velocity of the right hip (z-0 fe-coordinate
Angular velocity of the right hip (y- (y-coordinate)
Angular velocity of the left. hip (x-coordinaate)
Angular velocity of the left hip (z- (z-coordinate)
Angular velocity of the left hip (y-coordinate)
Torque applied on the right. knee rotor
Torque applied on the left. knee rolor
3D magnetic flux vector at the torso Contact force at the right foot Contact force at the lefl. foot. Angle of the right hip (x-ooordinate) Angle of the right hip (s-coordinate)
Angle of the right hip (y-coordinate)
3-axis linear velocity of the torso
Angle of the left hip (x-coordinate)
Augle of the left hip (z-coordinate)
Angle the left hip (y-coordinate)
Angular velocity of the right knoe
Angular velocity of the left knee
Angle of the right knee
Angle of the left log
Observation
Action
Num
Num
0/1/2
3/4/5
6/7/8
9/ 9/10/11
2
3
5
6
7
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
22

---

Published in Transactions on Machine Learning Research (03/2024)
angular velocity (rad/ s/d/s) s)
magnetic flux density (T)
angular velocity (rad) //d/s)
angular velocity (rad) (s
angular velocity (rad/ ())
angular velocity (rad/ /s
angular volocity (rad/ /s)
angular velocity (rad/ /s)
accelleraton eeration (m/ (n/s²)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm))
torque (Nm)
torque (Nm)
velocity (m/s)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
angle (rod)
angle (rad)
force(N)
Unit
Unit
Joint.
hinge
hinge
hingo
hinge
hinge
hinge
Joint
hinge
hinge
Tnf jointpos hip y hinge
hinge
hinge
hinge
Tnf jointvel hip x hinge
Inf jointvel hip z hinge
hinge
hinge
hinge
hinge
free accelerometer
free
free free
Name in XML
thigh joint.
leg joint
foot. joint.
Name in XML
Inf velocimeter
gyro magnetoometen touch foot
Inf jointpos hip x
Inf jointpos hip z
jointpos thigh
Inf joint leg
Inf jointpos. foot
Inf jointvel hip y
Inf jointvel thigh
Inf jointvel leg
jointvel fool.
hip __
hip 2
hip y
Max
Max
1
1
1
Table 10: Observation of Hopper
Inf
Inf
Inf
Inf
Inf
Inf
Table 9: Action space of Hopper
Min
-1
-1
-1
-1
Min
-Inf
IIf
-Inf
-In
-Inf
IInf
-Tnf
Inf
Inf
-Tn
-In
Inf
TTf
Inf
-nn
Torque applied on the rotor between torso and the hip (x- xe-coordinate) Torque applied on the rotor between torso and the hip (z-coordinate) Torque applied on the rotor betwoen torso and the hip (y-coordinate) Torque applied the thigh rolor Torque applied on the leg rotor Torque applied on the foot rotor
3-axis linear acoeleratinn Lion of the torso (including gravily)
3-axis angular velocity volocity of the torso
Angular velocity of the hip (x-coordinate) Angular velocity of the hip (z-coordinate)
Angular velocity of the hip (y-coordinate))
3D magnetic flux vector at the Lorso Contact force at the foot Angle of the hip (x-coordinate) Angle the hip (z-coordinate) Angle the hip (y-coordinate))
3-axis linear velocity of the torso
Angular velocity of the thigh
Angular volocity of the leg
Angular velocity of the fool.
Angle of the thigh
Angle of the leg
Angle of the foot
Obscrvation
Action
Num
Num
0/1/2
3/4/5
6/7/8
9/10 9/10/11
2
3
5
12
13
14
16
17
18
19
20
21
22
23
24
23

---

Published in Transactions on Machine Learning Research (03/2024)
o
magnetic flux density (T)
magnetic flux density (T)
magnetic flux density (T)
magnetic flux density (T)
magnetic flux density (I)
angular velocity (rad/ ss
angular velocity (rad/s s)
angular velocity (rad/s (nd/s)
angular velocity (rad/s /s)
angular velocity (rad/s)
angular velocity (rad/ ss
angular velocity (rad/s (/d/s)
angular velocity (rad/ ss)
acceleration (m /s))
acceleration (m/s" (s²)))
acceleration (m /s))))
acceleratin tiration (m/ (m/s²) /s)
acceleration mm/s²)
torque (Nm)
torque (Nm)
torque (Nm)
velocity (m (m/s) /s)
velocity (m/s) /s)
velocity (m/s) s))
velocity (m/s)
velocity (m/s)
angle (rad)
angle (red)
angle (rad)
force(N)
Unit
Unit
Joint
hinge
hinge
hinge
Joint
Inf accelerometer link 1 free
Inf velocimeter link ] free
Inf gyro link 1 free
-Inf Inf magnet link 1free
Inf velocimeter link 2 free Inf accelerometer link frre
free
Inf scceleromettr link 3 free Inf mnagnetometer link 2free
Inf velocineter link 3 free
Inf gyro link 3 free
Inf magnet link 3free
Inf velocineter link 4 free Tnf accelerometer link free
gyro link 4 free
Inf magnetometer link 4free Inf
Tnf accelerometer link 5 free
Inf velocimeter link 5 free
gyro link 5 free
Tnf magnetometer link 5free
hinge
hinge
hinge
hinge
hinge
hinge
touch end effector
jointpos joint, 3
jointvel joint 1
Inf jointvel joint 2
jointvel joint. 3
Name in XML
Max Name XML
jointpos joint,
jointpos joint.
joint 2
joint 3
gyro link
joint
Max
Torque applied on the second joint (connecting the first and the -1 1
Torque applied on the third joint (connecting the second and the -1 1 third link)
Table 12: Observation space of Arm3
Inf
Inf
Inf
Inf
Inf
Inf
Tnf
Inf
'able 11: Action space of Arm3
Min
Min
IInf
-In
-In
Inf
-nn
-In
IIn
-In
-Inf
-Inf
IInf
-Tnf
-If
IIn
IIf
-Inf
-Inf
-Inf
-Tnf
IInf
-In
-If
-Inf
-Inf
-Inf
Torque applied on the first joint (connecting the base point and the first link)
3-axis linear acceleration of the first. link (including gravity)
3-axis linear acceleration of the second link (including gravity)
3-axis linear acceleration of the third link (ineluding gravity)
3-axis linear acceleration of the fourth link (including gravily)
3-axis linear acceleratio lleaatatoon of the fifth link (including gravity)
3-axis angular velocity velocity of the first link
3-axis linear velocity of the second link 3-axis angular velocity velocity of the second link
3-axis linear velocity of the third link 3-axis angular velocity velocity of the third link
3-axis angular velocity velocity of the fourth link
3-axis linear velocity of the fifth link 3-axis angular velocity velocity of the fifth link
3D magnetic flux vector at the first link
3D magnetic flux voctor at the second link
3D magnetic flux vector at the third link
3D magnetic flux vector at the fourth link
3D magnetic flux vector at the fifth link
3-axis linear velocity of the fourth link
3-axis linear velocity of the first link
Angular velocity of the second joint Angular velocity of the third joint Contact force at the end effector
Angle of the third joint Angular velocity of the first joint
Angle of the second joint.
Angle of the first joint
second link)
Observation
Action
12 /1//14
15/ /1//17
18/ 22/19/22 /20
21 /22 /23
24 /25 26
27 /28 /28/29 22
30 //11//33 /32
33 /3//33/ /35
30/ /3/33//33 /38
39 /40/41
42 //3/44 /44
45 //6/// 11/477 /47
48 /49 44/50 /50
51 /52 /53
54 /5/ /56
57 /58/59
9/10/11
Num
Num
0/1/2
3/4/5
6/7/8
1
2
60
61
62
63
64
65
66
24

---

Published in Transactions on Machine Learning Research (03/2024)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
torque (Nm)
Unit.
Join!
hinge
hinge
hingo
hinge
hinge
hinge
Name in XML
joint 2
joint. 5
joint 6
joint.
joint
joint
Max
-1 1
1 Torque applied the fifth joint (connecting the fourth and the fifth link)
Torque applied on the sixth joint (connecting the fifth and the sixth -1 1 link)
Table 13: Action space of Arm6
Min
Torque applied on the first. joint (connect ing the base point and the first link)
Torque applied on the second joint joint coonmectig the first and the second link)
Torque applied on the third joint (connccting the second and the third link)
Torque applied on the fourth joint (connecting the third and the fourth link)
Action
Num
2
5
25

---

Published in Transactions on Machine Learning Research (03/2024)
angular velocity (rad/s s)
magnetic flux density (I)
angular velocity (rad/s ss)
magnelic flux density (T)
angular velocity (rad/s s)
magnetic flux density (T)
angular velocity (rad/a s/)
magnetic flux density (T)
angular velocity (rad/s) s)
magnetic flux density (T)
angular velocity (rad/s) s)
magnetic flux density (T)
angular velocity (rad/s s)
magnetic flux density (T)
angular velocity (rad/s ss
angular volocity (rad/ s
angular velocity (rad/a /s)
angular velocity (rad/s s)
angular velocity (rad/s) /ss
angular velocity (rad/s s))
accoleration (m/ (m/s²)
/s2)
acceleration (m/ m//s²)
acceleration (m /s²)
acceleration (m/s" (s))s))
acceleration (m/ /²)
/s)
acceleratinn tio (m/ (m/s²)
acceleration (m/ mm/s²))
velocity (m (m/s) s)
velocity (m/s) /s)
velocity (m/s) mm/s)
velocity (m/s)
velocity (m (m/s) s)
velocity (m/s)
velocity (m/s)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
force(N) (N)
Unit
Joint
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
hinge
accelerometer link 1 free Inf
IInf Inf velocineter link 1 free
Inf gyro link 1 free
Inf magnetometer link 1free
-Inf Inf accelerometer link free
Tnf velocimeter link 2 free
gyro link 2 free
-Inf Inf velocimetor link. 3 free IInf Inf accelerometer link 3 free Inf magnel oimeker link 2free
gyro link 3 free
Tnf maggetomeetrr link 3free
Inf accelerometer link 4 free
Inf velocineter link 4 free
Inf magnetometer link 4free Inf
gyro link 4 free
Inf accelerometen link 5 free
Inf velocineter link 5 free
gyro link 5 free
-Inf Inf magnetometen link 5free
rocelerometer link 6 free
Inf velocineter link 6 free
gyro link 6 free
-Inf Inf accelerometer link 7 free Inf magnetometee link Gfree
-Inf Inf velocimeter link 7 free
free
7free
Tnf magnet magnetometer ommeter link
Tnf touch end effector
jointpos joint, 3
jointpos joint, 4
Inf jointpos joint, 5
Inf jointpos joint 6
jointvel joint 2
Inf jointvel joint. 3
Inf jointvel joint 4
Inf jointvel joint 5
Inf jointvel joint 6
Max Name XML
gyro link 7
jointpos joint
jointpos joint.
Inf jointvel joint
Inf acceleromen
Inf
Inf
Tnf
Inf
Inf
Inf
Inf
Inf
Tnf
Inf
Table 14: Observation space of Arm6
Min
IInf
-Inf
-Inf
-Tnf
Innf
IIn
IIff
-Inf
-Inf
IInf
-Inl
-Inf
Inn
-Iff
-Inf
-In
-Inf
-Inf
IIn
IInf
-Tnf
lnf
-Inf
IIf
-Tnf
-Inf
Inn
Inf
Inf
Inl
-nf
-Inf
0
3-axis linear acceleration of the first link (including gravity)
3-axis lincar acceleration of the second link (including gravity)
3-axis linear acceleration of the third link fincluding gravity)
3-axis linear acceleration of the fourth link (including gravity)
3-axis linear acceleration of the fifth link (including gravity)
3-axis linear cooelerationn of the sixth link (including gravily)
3-axis linear acceleration of the seventh link fineluding gravity)
3-axis angular velocity velocity of the first link
3-axis angular velocity velocity of the second link
3-axis angular velocity velocity of the third link
3-axis linear velocity of the fourth link 3-axis angular velocity velocity of the fourth link
3-axis linear velocity of the fifth link 3-axis angular velocity velocity of the fifth link
3-axis angular volocity volocity of the sixth link
3-axis angular velocity velocity of the seventh link
3-axis linear velocity of the first link
3D magnetic flux vector at the first link
3-axis linear velocity of the second link
3D magnelic flux veclor at the second link
3-axis linear veloeity of the third link
3D magnetic flux vector at the third link
3D magnetic flux vector at the fourth link
3D magnetic flux vector at the fifth link
3-axis linear velocity of the sixth link
3D magnelic fuxx vector at the sixth link
3-axis linear velocity of the seventh link
3D magnetic flux vector at the seventh link
Angular velocity of the first joint Angular volocity the second joint Angular velocity of the third joint Angular velocity of the fourth joint
Angular velocity of the fifth joint
Angular velocity of the sixth joint Contact force at the end effector
Angle of the first joint
Angle of the second joint.
Angle of the third joint
Angle of the fourth joint
Angle of the fifth joint
Angle of the sixth joint
Observation
12/13/14
15 /17
18/ 2//9//2/ /20
21 //2///22 22
24 /25 /2 /66
27 /28 /28/29 22
30/ /31 /32
33 33/34, /34 /35
36/ ///333333 /38
39/ /40/41
42 //3 /44
45, //6///// /46//7 /47
48 /49/ /49/50 (50
51 /52 /53
54 /5///5555 /56
57 /58/5 /59
60/ /6/ 10/61/66 /62
63/ /6//65 /65
66/67 /6 /6/68
69/ 77//71
72 72/73/ /73 /8/
75 75/76/ /76 /77
78 78/79) /79 179/80/ /80
81 11/82/88 /83
9/10/11
Num
0/1/2
3/4/5
6/7/8
84
85
86
87
88
89
90
91
92
93
94
95
96
26

---

Published in Transactions on Machine Learning Research (03/2024)
angular velocity (rad/ w/ad/s) s)
free magnetic flux density (T)
acceleration (m/s²)
velocity (m/s)
force (N)
force (N)
force (N)
force (N)
forco( (N)
force(M) (N)
force( ())
force( (V)
force(M) [))
force(N)
force(N)
force(N)
Unit
Unit
Joint
Joint
free accelerometer
Tnf velocimeter gyro free magnetomeetr Inf touch pla
free
Namo XML
Name: in XML.
touch plb
Inf touch p2a
Inf touch p2b
Inf touch. p3a
0 Tnf touch p3b
Inf touch p4a
Inf touch p4b
Max
Max
Inf
Inf
-In Inf
Inf
1
Table 16: Observation space of Drone
Table Action space of Drone
Min
-1
-1
-1
Min
-Inf
IInf
-Tnl
0
3-xxis linear acccleration of the torso (including gravity)
3D magnetic flux vector at the torso Contact force at the upper point the first propeller Contact foroe at the lower point of the first propeller Contact force at the upper point of the second propeller Contact force at the lower point the second propeller Contact force at the upper point of the third propeller Contact force at the lower point of the third propeller Contact force at the upper point of the fourth propeller Contact force at the lower point of the fourth propeller
Extra thrust force applied on the second propeller
Extra thrust foree applied on the third propeller Extra thrust, force applied on the fourth propeller
Ext.a thrust force applied on the first. propeller
3-axis angular velocity velocity of the torso
3-axis linear velocity of the torso
Observation
Action
9/ 9/10/11 /11
Num
Num
0/1/2
3/4/5
6/7/8
2
3
12
13
14
15
16
17
18
19
27

---

Published in Transactions on Machine Learning Research (03/2024)
B.2 Observation Space Options and Desiderata
The observation spaces are also updated to match the new 3D lasks. The 3D compasses and 3D pseudo-lidars are introduced for 3D robots to sensor the position of targets in 3D space. Different from the single lidar system of the original environment, the Advanced Safety Gym allows multiple lidars on different parts of the robot. For example, in Figure 13a the Arm robot is equipped with a 3D lidar and a 3D compass on each joint to obtain more environment information. Figure 13b shows a drone equipped with two 3D lidars observe the 3D hazards and the 3D goal. The "lidar halos" of two lidars are distributed on two spheres with different radii. The number of "lidar halos" is configurable for more dense observations.
(a)
Figure 13: Visualizations observation spaces
B.3 Layout Randomization Options and Desiderata
The layout randomization is inherited from the original Safety Gym. In order to generate 3D objects, the z coordinate can be configured or randomly picked after the æ and y coordinates are generated.
B.4 Task and Constraint Details
Table 17: Comparison between different tasks
GUARD Tasks
SafetyGym Tasks
Goal
Push
Chase
Defense
Goal
Button
Push
Interactive task Non-intoractive
task
Contact task
d
Non-contact task
2D task
3D task
Movable target
Immovable target
Single target
d
Multiple targets
General contact tar-
get.
28

---

Published in Transactions on Machine Learning Research (03/2024)
Table 18: Comparison between different constraints
New Constraints
Inherited Constraints
Ghosts Ghosts
Hazards Hazards Vases
Pillars
Buttons Gremlins
3D
3D
Trespassable Untrespassable
Immovable
Passively movable
Actively movable
3D motion
2D motion
B.5 Dynamics of movable objects
We begin by defining the distance vector darigin Zorigin Lobject; which represents the distance from the position of the dynamic object Tobject to the origin point of the world framework Zorigin- By default, the origin point is set to (0, 0, 0). Next, we define the distance vector drobot = Trobot - Pobject, which represents the distance from the dynamic object Tobject to the position of the robot Trobot- We introduco two paramcters: To: which defines a circular area centered the origin point within which the objects are limited to move. rı, which represents the threshold distance that the dynamic objects strive maintain from the robot. Finally, have three configurable non-negative velocity constants for the dynamic objects: vo, U1, and v2.
B.5.1 Dynamics of targets of Chase task
Up dordorggnn
if |[oriiin ro
Eobject
U1 #dyohot
if [dorigin vo and ||drobot ri S To and |drobot
(11)
if |dorigin
B.5.2 Dynamics of targets of Defense task
vo
if || dorigin
t objject
U1 #drobot:
if || dooosgin
S ro and [|droborl] $ 11 S rn and ||drotoot| > ri
(12)
U2 #doiigng
if [doriigin]
B.5.3 Dynamics of ghost and 3D ghost
Up doriiiinn
doriiin
> ro
Dobject
U/ u1oorobot:
dorsgin
S ro and ||drobor || > 11 S ro and
(13)
0,
if ||dorigin
||drobot S Tl
29

---

Published in Transactions on Machine Learning Research (03/2024)
C
Experiment Details
The GUARD implementation is partially inspired by Safety Gym RRa et al. 2019b) and Spinningup (Achiam
2018
which are both under MIT license.
C.1
Policy Settings
The hyper-parameters used in our experiments are listed in Table 19 as default.
Our experiments use separate multilayer perceptrons with tanh activations for the policy network, value network and cost network. Each network consists two hidden layers of size (64,64). All of the networks are trained using Adam oplimizer with learning rate of 0.01.
We apply an on-policy framework in our experiments. During each epoch the agent interacts B times with the environment and then performs policy update based the experience collected from the current epoch. The maximum length of the trajectory is set to 1000 and the total epoch number N is set to 200 default. In our experiments the Walker and the Ant were trained for 1000 epochs due to the high dimension.
The policy update step is based on the scheme of TRPO, which performs up to 100 steps of backtracking with a coefficient of 0.8 for line searching.
For all experiments, we use a discount factor of y 0.99, an advantage discount factor À 0.95, and a KI-divergence step size of бg 0.02.
For experiments which consider cost constraints we adopt a target cost d 0.0 to pursue zero-violation policy.
Other unique hyper-parameters for each algorithm are hand-tuned to attain reasonable performance.
Each model is trained on server with 48-core Intel(R) XRon(R) Silver 4214 CPU @ 2.2.GHz, Nvidia RTX A4000 GPU with 16GB memory, and Ubuntu 20.04.
For low-dimensional tasks, train each model for 6e6 steps which takes around seven hours. For high- dimensional tasks, train each model for 3e7 steps which takes around 60 hours.
C.2 Experiment tasks
The experiment settings, detailed in Table 20 encompass total of 72 combinations derived from 4 task types, 9 robot types, and 2 constraint types. For the purpose of comprehensive comparison in this paper, we have selected 36 experiments involving 8 hazards and 9 experiments featuring 8 ghosts.
C.3 Metrics Comparison
report all the 45 results of our test suites by three metrics:
The average episode returns J,.
The average episodic sum of costs M
The average cost over the entirety of training Pe-
All of the three metrics were obtained from the final epoch after convergence. Each metric was averaged over two random seeds.
30

---

Published in Transactions on Machine Learning Research (03/2024)
PCPO
200
30000
1000
(64. 64)
0.99
0.97
0.02
(64, 64)
80
Adam
0.001
(64, 64)
80
Adam
0.001
0.0
CPO
200
30000
1000
(64, 64)
0.99
0.97
100
0.8
0.02
(64, 64)
80
Adam
0.001
(64, 64)
80
Adam
0.001
0.0
0.0
TRPO-FAC
200
30000
1000
(64. 64)
0.99
0.97
100
0.8
0.02
(64, 64)
80
Adam
0.001
(64, 64)
Adam
0.001
0.0
Adam
0.0001
TRPO-IPO
Table 19: Important hyper-parannn pappr--prametesss.. of different algorithms in our experiments
200
30000
1000
(64. 64)
0.99
0.97
100
0.8
0.02
(64, 64)
80
Adan
0.001
0.0
0.01
TRPO- USL
200
30000
1000
(64, 64)
0.99
0.97
100
0.8
0.02
(64, 64)
80
Adam
0.001
(64, 64)
80
Adam
0.001
0.0
20
0.05
1/3
TRPO- Lagraninnn TRPO-SI SL [18' Dalal
200
30000
1000
(644 64)
0.99
0.97
100
0.8
0.02
(64, 64)
80
Adam
0.001
(64, 64)
80
Adam
0.001
0.0
1/3
200
30000
1000
(64. 64)
0.99
0.97
100
0.8
0.02
(64, 64)
80
Adamn
0.001
(64, 64)
80
Adam
0.001
0.0
0.005
TRPO
200
30000
1000
(641, 64)
0.99
0.97
100
0.8
0.02
(64, 64)
80
Adamn
0.001
N
B
L
?
5XL
60
Maxinun length of trajectory
TRPO hocktracking coefficient
Policy network hidden layers
Value network hidden layers
Cost network hidden layers
Advantage discount factor
TRPPO backtracking steps
Value network iteration
Value work optimizer
Cost net work iteration
Cost network optimizer
Lagrangian learning rate
USL correction iteration
Lagrangian optimizer
Policy Parameter
USL correction rate
Cost learning rate
Steps per epoch
Discount factor
Value learning
Warmup ratio
IPO parameter
Cost roduction
Target Cost
Target KL
Epochs
31

---

Published in Transactions on Machine Learning Research (03/2024)
hoal_Point: 8aazrrd
Push_ _ish_Poitt__ P
8Haaards
Leal_Pointt__0 8Chosts Poin_ttottBtsssss
Push
Push_Point, Point Postpotttntttttsss
8Chosts
@eal_Swimmer_BHaaee 8Hazards
2astmmer__HHaaads
Goal hw__SSimmer__hhheee
Push Swimmer 8Ghosts
Goal_Ant_BHaazaa
Push Bash_Ant___ Bech_Annt_HHazzaae
8Hazards
Ant__hon_hhhsss
Sehh_Walker_BHazaaee
Goal_HHumanoidd_ffaaaardd
Push_Humanoid__ee 8Hazards
Goal_Humanoid_8@hn
Push_Humannidd_BHhee
Goal Goal_Hooperr_BBazzardd
Maah_Hopper_BHazaaee
Goal_Hoopeer_Ghhoste
Push_Hopper_ 8Ghosts
@oal_Arm3_8Hazarde
Goal_Arm3_80n 8Chosts
Push_Arm3 8Chosts
Araa
8Hazards Ara__aarad
Goal _ar___rm6____1__ Arm6__Ghostss
8Ghosts
Push_Arm6 ____Arm6____ __Coosthhtss
Goal Goal_Drone Dda
8Hazards
Push Push_Drone Drone Dwwneeraaaaarreeeeeeeee
8Hazards
Liaal_Drone__ee @ronee8Ghosts
Push_Drome
(a) Goal
(b) Push
Chase Teaas_Pointt.. Lww.PontttHaazarddddoea
8Hazards
Defense_Poiie PotteePonnttttaaaadsss
8Hazards
Chase Chase_Point 8Ghosts
Defense LaciaeeePoitt_ttee _Point_8hhosts
8Ghosts
Thase_Swinmen 8Hazards
Defense Defense_Swinmer 2aaiaadrrd
Chase_Swimme 8Chosts
Defense Defense_Svimmer Swimmerw_rrr.. Baaeee_Swvimeee_ehhhee
Chase_Are 8Hazarda
Littense_Ant_BBaaaeee
Chase_Ant Stase_Antt8Ghosts
8Ghosts
Defense_Ane 1aatt__hhstts
Chase Chase_Walkee 1aaker_8Hazards
Defense _aaese__Waleer_ee Bat___akee___aaarraae
8Hazards
Chase Walker 8Ghosts
Defense Defense_Walker__ Beakker_@hhosts
thase__umanaan H
8Hazards
Lefense_Humanoid_ 8aazaadd
Chase Hassse_Hmmniiiddae 8Chosts
Defeense_Humannodd__00 8Ghosts
Chase_ @lase__Hopper_Bfeezee
Defennse_ioopeer_Bffeeaee
Shase_HHopper_BChostt
Tiiensse_Hopper__Bhhoo
Chase_Arm338HHazzrl
Befense_Arm33BHHazzree
Chase_Arm3_8Ghoss
Defensse_Arm3_Ghhoes
Chase_Arm6_8Hazee
Chase Laase_Arm6_00n Armo_Bhhostss
Defense Baaisse_rrm___hhhh0
Chase BareeeDrrnee_eee HHazards
Defense Drone 8Hazards
Lefeese_Dronee_e
Chase Laase_Drone___e 8Ghosts
Defense Defense_Drone____ 8Ghosts Drone_8Ghosts
Chase
(d) Defense
Table 20: Tasks our environments
32

---

Published in Transactions on Machine Learning Research (03/2024)
Table 21: Metrics of nine Goal_ (ool_((Rooo)_BHaaartte 8Hazards environments obtained from the final epoch.
Point 8Hazards
Goal Swimmer 8Hazards
Goal Ant 8Hazards
Algorithm
M.
Algorit hm
Algorit
J.
TRPO
20,2206
7.4530
0.0007
TRPO
21.5282
0.0117
TBPO
30.3094
7.9737
0.0097
TRPO-Lapangien
25,1503
2.5/31
0.00034
TRPOLLegnnnion
10.5685
4.5231
0.0074
35.0130
2.7954
0.0056
TRPO SL
19.0965
3,5200
200056
TRPO SL
0,2362
4.4453
0.0073
TRPO
24,0722
45.0755
0,0055
TRPO-USL
246624
7,0004
3.0050
TRPO-USI
30,2736
10.2552
0.0100
TRPOUUSL
50.2213
9.2237
0.0006
TRPO-IP
20,3057
1.4037
2.0049
TRPO-PO
9,5714
7.9958
0,0079
TRPO-PO
2,0000
6.3006
0,0000
TRPO-FAC
20.9207
2.1581
0.0088
TRPO-FAC
21,8480
7:801
0.0085
TRPO-FAC
48.3085
56730
0.0071
3,3388
0,006
CPO
26/6106
CPO
50,200
81199
0.0091
PCPO
23,0057
377118
9.3322
PCPO
89137
0.000
Goal Walker 8Hezards
Goal Humanoid 8Hazards
Goal Hopper 8Hazards
Algorithm
J.
M.
Algorithm
J.
M-
Algorithm
J.
M.
TRPO
56,7150
0.8112
0.0104
TRPO
11,6758
5:2832
0.0078
TRPO
$2,5406
7,8477
0.0062
TRPO-Lagrangan
53.7839
3.3714
0.0055
TRRO-Tagranienn
6,1281
7.0517
0.0066
TRPO-Lagranian
25:2150
6.4342
0.0069
TRPO-SL
39.5848
1275700
0.0125
TRPO-SI
9.1517
10.1755
0.0091
TRPO-SL
20.1250
8,4566
0.0005
TRPO-USE USL
57.1097
0.5209
0.1099
TRPO USL
10,5010
0,0079
TRPO USL
32,3092
8.1325
0.0000
TRPO-IPU
7,2728
6.715
0.0065
TRPO-IPU
2.5361
5,0712
0.0071
TRPO-IPO
4,0118
7.0667
0.0062
TRPD-FAC
TRPO-FAC
TRPO-FAC
CPO
51,0046
80:08
0.0083
CPO
11,9575
6.0818
0.0074
CPO
27.2541
8,0783
0.0176
PCPO
35,01000
100077
0,0089
PCPO
11,6731
6.8266
0,0074
PCPO
30,7607
6.4345
0.0076
Goal Arm3 8Hazards
Goal Arm6 8Hazards
Goal Drone 8Hazards
Algortthm
X2.
Algorithm
Algorithm
J.
M7.
At
TRPO
19,5716
23.8574
0.0233
TRPO
4.5703
15.0087
0.0206
TRPO
19,6492
1.0880
0.0012
TRPO-Lerangia
2.1431
0.0057
TRPO-Lunuuiaa
1,2186
6,8767
0.0107
TRPO-Lurnaaaa
17.5182
1.0470
0.0010
TRPG-ST,
TRPO-ST.
TRPO-ST.
0.001
TRPO-USL
15.6522
8.6734
0.01G3
2,5704
9.4153
0,0196
TRPO-USL USL
17,3535
1.1217
0.0008
TAPO-IPO
2,4211
12:5367
0.011S
TREO
0,8222
5.5569
0.0129
TRPO IPO
15.7180
0.8852
0.0007
TRPO-FAC
10,0048
5.3072
0.0085
TRPO-FAC
2.4243
5.9828
0.0124
ERPO-FAC
17.0156
1
0.6005
CPO
10.9057
22:1331
0.0210
CPO
4.3885
15.0155
0.0171
CPO
18.3072
1,0701
0.0010
PCPO
21.5110
16.2963
0,1211
PCPO
1,1525
13,8961
0,014L
PCPO
5,0076
0.2334
0.0003
Table 22: Metries of nine Goal_ Goal_[Robaa [Robot)_AGhosts 8Ghosts environments obtained from the final epoch.
Goal_Pointt_ 8Ghosts
Goal_ntt Ant_SGhosts
Algorithum
M
A
Algurit hum
Algurit
J.
A.
TRPO
6.8329
3.0073
30.3401
TRPO
30.5700
10:3785
0.0099
TRPO-Luranjja
20.3550
2.1498
3.0034
TRPO-Lunamjaa
15,5052
2.1046
0.0061
TRPO-Lunangia
28. 28.5846
2.9054
0.0060
TRPO-ST.
16:5488
100515
00008
TRPO-ST
7,8733
7.0575
0.0079
TRPO-SI,
30.7283
41.2.22
0.0322
TRPO-USI.
22.1995
5.8505
0.059
TRPO-USI.
30.1223
599:88
0.0105
TRPO-USE
01.2725
89165
0.0097
TRPO IPO
20,188
4.1369
2.0350
TRPO
9.9696
10.0275
0,0091
TRPO JPO
2,500
S.972
0.0061
TRPO-FAC
20,5450
2.8654
3.0036
TRPO-FAC
18.5050
4.4988
0,0000
TRPO-FAC
46.2423
5,5088
0,0074
CPO
26.5055
242288
0.0054
CPO
26/0033
0.0002
CPO
56.3422
98000
0.000
PCPD
00004
20.2757
PCPO
58.0051
98873
0.0996
Goal Walker 8Chosts
Goal Humanoid 8Chosis
Goal Hopper_8Chosts 8Chosts
Algoritiu
M..
6.
Al Algorithm
J.
37.
Algorit
J.
67.
6.
TRPO
63.2017
9.5771
0.0012
TRPO
11.1801
5.9922
0,0038
TRPO
31.6643
5.1509
0,0100
TRPO-Lagranjan
S3.2654
2.5072
0.0054
TRPO-Lagranien
5.0070
6.8812
0.0076
TRPO-Lggrangan
16.0000
4.4744
0.0070
TRPO-SL
37.5088
20.3778
0.01
TRPO-SL
8.8830
17.0632
0,0107
TRPO-SL
21.7761
12:4510
0.3122
TRPO-USI.
61.4547
0.0013
0.010
TRPO-USE.
10.086
9.6218
0,0006
TRPO-USE
31,2661
845500
0.000C
TRPO-PP
7,4640
9.1178
0,0090
TRPO JPO
1,0844
TRPO-IP
5,4526
12/0015
0,0082
TRPO-FAC
0.0071
TRPO-EAC
9.2134
100016
TRPO-EAC
29.1177
7.463
0.0057
CPO
60.1257
0:2117
0.0097
CPO
10,0778
10.0074
0.0092
CPO
20,0408
7.5681
0.0085
PCPO
13,5000
9 2002
0.0085
POPO
11.5003
0.000
0.000
PCPO 20.038
80181
0.0000
Goal Arm3 8Chosts
Goal Arm 8Chosts
Goal Drone 8Chosts
Algoritho
J,
17
Algorithm
J.
37.
Algorithm
J.
17.
Px
TRPO
$4,6600
35.7460
G.0334
TRPO
L0157
42.0135
0.0466
TRPO
17.9454
1.7287
0.8011
TRPO Ligrangion
16.4808
7.3123
0.0058
TRPO Ligrangian
0,5470
8.4307
0.0190
TRPO Ligrangion
18.9773
0.9215
0.0005
TRPO-SL
18.1207
10.7880
0.0174
TRPO-SL
0.0078
20.5000
0.0056
TRPO-ST.
12.1413
0.2500
0.0004
TRPO-USE
62.1621
110982
0.0223
41.7004
TRPO-USL
10.7517
0.974
0.001
TRPO-IPO
4,0235
105251
0.0000
TRPO-IPO
0/7356
12.5153
0.0233
TRPO-IPO
11,5210
0.0817
0.000
TRPO-FAC
37,9730
6.0701
0,0073
TRPO PAC
0.7861
3.4498
0.0170
TRPO-EA
20.1014
0,7630
0.0006
CPO
114.6706
15.190%
0.0159
CPO
9.9995
22.5001
0.0251
CPO
18.1728
1.2165
0.0005
PCPO
126.4001
10.1913
0.0143
PCPO
0.5845
15,0715
0.0162
PCPO
6.5276
0.3500
0.0003
33

---

Published in Transactions on Machine Learning Research (03/2024)
Table 23: Metrics of nine Push (Robot)ooo......o (Robott)_HHaaaree 8aaarrds environments obtained from the final epoch.
Push Point 8Hazards
Push Swimmer 8Hazards
Push Ant 8Hlzardd
Algorithm
M.
Algorithun hm
Algorit
J.
TRPO
11,3050
7.3535
0.0084
TRPO
86.1357
11.9215
0.0111
TBPO
18.4378
0.0091
TRPO-Lapangien
4.1189
1.8288
0.0057
TRPO-Lagnnnion
52.0899
4.5945
0.0070
1.1583
1.5048
0.0045
TRPO SL
3,0553
6.6139
20005
TRPO SL
13.1891
7.7554
0.0037
TRPO
3.5622
47.7902
0.3217
TRPO-USL
9.1904
6.6179
2.0054
TRPO-USI
64.000
9.4953
0.0085
TRPOLUSL
11.376
9.3000
0.0096
TRPO-IP
1,3370
4.0476
2.0031
TRPO-IPO
6,3843
5.4329
0,0077
TRPO-PO
1.3986
0.9120
0,0061
TRPO-FAC
6.0451
2.1250
0.0030
TRPO-FAC
18,2350
5,8075
0.0061
TRPO-FAC
2,5005
2,927
0.0050
3.6405
0.0066
CPO
57.4370
CPO
12,7051
7.5422
0.0082
PCPO
6.0005
PCPO
11,0101
0.0087
Push Walker 8Hazards
Push Humanoid 8Hazards
Push Hopper 8Hazards
Algorithm
J.
M,
Algorithm
J.
M.
Algorithm
M.
TRPO
6.0574
10.85:00
0.0080
TRPO
0.3565
10.6542
0.0096
TRPO
3.6134
10.9505
0.0096
TRPO-Lagrangan
1.5033
2.4231
0.0040
0.7407
3.1738
0.0062
TRPO-Lagranian
0.5381
2.0782
0.0052
TRPO-SL
L.753
17,5680
0.0082
TRPO-SI
0.2980
0.0239
0.0032
TRPO-SL
1.5115
52615
0.00/0
TRPO-USL USL
2.776
0,3000
0,0075
TRPO USL
0.5102
7:3110
0,0000
TRPO USL
2.302
11.2535
0,005
TRPO-IPU
0.7091
3.6377
0.0070
TRPO-IPU
0.8191
6.0952
0.0074
TRPO-IPO
7.4184
0.0683
TRPD-FAC
1.530
TRPO-FAC
TRPO-FAC
CPO
4.5113
7.8450
0.0075
CPO
0.8147
8.884
0.0090
CPO
2,3108
11,2012
0.0089
PCPO
9.2470
0,0075
PCPO
1.0445
$.1200
0,0084
PCPO
0.9085
8.833
0,008)
Push Arm3 8Hazards
Push Arm6 8Hazards
Push
Drone 8Hazards
Algortthm
Algorithm
17.
Algorithm
M7.
TRPO
0.0158
37.7114
0.0114
TRPO
1.1128
15,9050
0.0190
TRPO
0,9352
0.3321
0.0002
TRPO-Lerangia
-10.8565
2.7071
0.0002
TRPO-Lunuuiaa
7.1961
0.0110
TRPO-Leraaaae
1.0067
0.8197
0.0005
TRPG-ST,
TRPO-ST.
TRPO-ST.
TRPO-USL
42:2457
10,0055
0.0189
.06530
16.7100
0.0182
TRPO-USL USL
0:3410
0.0295
0.0001
TRPO-IP
4200880
25.0220
00214
TRPO JPO
1.1291
8,3642
00113
TRPO IPO
1,004
0.4229
0.0002
TRPO-FAC
114.8012
7,8064
0.0366
TRPO-FAC
1.0548
9,4750
0.0152
TRPO-FAC
1,0820
0,2380
0.0002
CPO
6.0219
11:3773
00178
CPO
1.3699
6.6375
0.0108
CPO
1.1261
0.3409
0.0045
PCPO
33,0294
10.4457
0.0207
PCPO
1,1439
10.0134
0.0112
PCPO
0.3844
0.0049
0.0001
Table 24: Metrics of nine Chase (Robot)
[Robot)_SHazards 8Haardds environments obtained from the final epoch.
Chase Point__ _Hazards
Chase_ Swimmer_ 88aazards
Chase Aut__Att_aaaaa 8Hazards
Algorithu
J,
Algorit
J.
A.
Algurithum
A.
TRPO
1.322
L2491
TRPO
6.1101
0,0100
TRPO-Luranjja
L,879
2.8816
0.0040
TRPO-Lenanian
4.2346
4.8800
0.0058
TRPO-Lunangia
-0.553
2.5016
0.0040
TRPO-ST.
0,818
3,9000
0.005
TRPO-SL
0.018
92661
0.0071
TRPO-SI,
0.7921
15.0546
0,022)
TRPO-USI.
1.153
5.7521
0.080
TRPO-USI.
1,2027
9-2911
0.0100
TRPO-USE
1.5841
50640
0.00%
TRPO IPO
0,7928
S,5512
0.0061
TRPO IPO
10085
10,5546
0,0050
TRPO JPO
0,0314
25520
0,002
TRPO-FAC
3.0587
0.0063
TRPO-FAC
0.6411
0.1616
0,0078
TRPO-FAC
-0.2G5
3.5600
0,008
CPO
1,8807
0.0055
CPO
1,2540
11671
0.0082
CPO
1.4304
5.7905
0.0087
PCPD
1.0355
7.0188
1,2152
PCPO
Chasc_ Walker 8aaardd
Chase Humanoid 8Hazards
Chasc Hopper 8Hazards
Algoriitu
J.
37.
A.
Algorithm
J.
M2.
Algorit
J.
64.
n.
TRPO
0,0590
T.6845
0.0088
TRPO
0.2330
12.1455
0.0152
TRPO
0.6099
12.1675
0.0134
TRPO-Lagranjan
0.0522
2.5167
0.0045
TRPO-Lagraniinn
-0,555
3.4254
0.0047
TRPO-Aggoanian
-0.9641
3.2170
0.0039
TRPO-SL
402116
10.7107
0.0091
TRPO-SL
-0,2211
11.8001
0.0121
TRPO-SL
6.4957
5.4556
0,0080
TRPO-USE.
0.0030
7.7185
0/0082
TRPO-USE
401385
180011
0.01.0
TRPO-USE
0.0819
11,0919
0.0131
TRPO-IP
0,8223
2.3054
0.0035
TRPO JPO
008074
0,0054
TRPO-IPO
0.7706
6.1216
0.0061
TRPO-FAC
27105
0,0047
TRPO FAC
0.5436
TRPO-EAC
37201
0.005
CPO
0.7408
10.4005
0.0086
CPO
4.8322
12.1665
0.0109
CPO
0.4829
67117
0.0083
PCPO
0.63447
58852
0.0080
POPO
400071
103111
0.01
PCPO
-01157
7.1800
0.001
Chase Arm3 8Hazards
Chase Arm SHazards
Chase Drone SHazards
Algooiith
J.
37g
Algorithm
J,
17.
f%
Algorithm
17
TRPO
0,7772
20,0230
0.0011
TRPO
-0.3060
60.5704
0,0008
TRPO
L.051
0.0933
0.6005
TRPO Ligrangion
0.2739
5.1092
0.0079
TRPO Ligrangian
.04856
2.4002
0.0075
TRPO Ligrangian
1.3455
0.0005
TRPO-SL
0.0007
(:2869
0.01
TRPO-SL
-0:420
12.1256
0/0237
TRPO-ST
-1.9055
0.3503
0.0002
TRPO-USE
0.7823
14.1736
0.0084
.0333
33,4553
B1675
TRPO-USL
1.2159
0.005
TRPO-IPO
4.4137
10.0885
0.00231
TRPO-IPO
-02855
11,6709
B1055
TRPO-IPO
0.2518
0.5785
0.0005
TRPO-FAC
0,3548
3.3440
0,0127
TRPO-PA PAC
.03183
13.2429
8.0253
TRPO-EA
1.1193
0.2374
0.0005
CPO
0.8051
17. 17.7
0.0252
CPO
-0.3278
16.0600
0.5217
CPO
0.7682
0.0075
0.0006
PCPO
0,7365
23.8312
0.0291
PCPO
.02883
45.6161
0.0633
PCPO
0.6172
0.6374
0.6012
34

---

Published in Transactions on Machine Learning Research (03/2024)
Table 25: Metrics of nine Defense {Robot} 8Hazards environments obtained from the final epoch.
Defense
Point 8Hazards
Defense Swimmer 8Hazadss
Defense Ant 8Hazards
Algorithm
Algorithm
Algorithas
TRrO-Lagraniau
12:2199
-5.0177
TRPO-Ligraniaa
1.5790
TRPO SL
89.822
3.3601
0.0700
TRPO
41,8925
1.3255
0.0118
TREO-SL
11.0395
14.7014
0.014
TRPO-USI
-100.7526
8.0235
0.086
TRPO-USI.
159.8015
13.5622
0.0150
TRPO-USE
20.1186
25,8225
0.0266
TRPO-IPO IPO
330,2252
0.7309
01005
TRPO-IPO
233.100 1902
7.6313
0/0770
TRPO-IPO
220.5713
6.019
00011
TRPO-FAC
300,000
0.7351
0.0015
TRPO-FAC
7457
0.8.0
0.0322
TRPO-FAC
-8111500
1.4305
00011
CPO
CPO
CPO
PCPO
Defense Walker 8 Hazards
Defense Hurnanoid 8Hazards
Defense Hopper 8Hazards
Algorichm
3,
AT.
Algorithm
J.
M2.
Algorithm
5.
17.
TRPO
63.0581
52.1051
0.0228
TRPO
-270.0028
4.3218
0.0012
TRPO
-79-1866
36.0422
0.0212
Laro-Lagraggan
221.8444
0.8080
0.0032
TRPO- TRPO-Lagr Legrangtan
-28-35466
30248
(100355
TAPO-Lagrangh
-Jst 2143
1.19K3
0.0020
TRPO-SL
-25.2992
21.2179
0.0142
TRPO-SL
.225.6866
2.0600
0.0019
TRPO-SL
207.0606
8.5136
0.0138
TRPO-USI,
10.397
25.4841
001822
TRPO-ESI
-S18.0001
355d5
0.0013
TRPO-USI,
57.7516
28. 28.5037
00231
TRPOJIPO
TRPO-IPO
TRPO-FAC
CPO
8,5413
PCPO
41,8441
17.0002
00161
PCPO
317.0540
2.053
010030
PCPO
279,4809
7.2800
0.007
Defense Arm3 8Hazards
Defense
Arm6 8Hazards
Defense
Drone 8Hazards
Algorithm
M2
Algorithm
Algorithm
16.
TRPO
169 5332
22.0750
0.1000
TRPO
183.9208
56.5331
0.0548
TRPO
-251.5723
0.0771
0.0002
TRPO
0.7071
0.0056
TRPO Ligrangian
TREO Ligrangion
245.7311
0.2276
TRPO-SL
112.3637
1.1055
TRPO-SL
TRPO-S
0.0000
0.0001
TRPO-USL
TRPO-USL
TRPO-IPO
$4.1030
8:005
TRPO-IPO
197.3447
0.0051
TRPO-IPO
0.3600
TRPO
180.9871
1,7731
0.0061
TRPO PAC
175.8237
233101
0,0199
TRPO PAC
215.4561
0.009:
0.0001
CPO
167 [06]
16 4305
0.0162
CPO PCPO
174.7701
22.1088
0/0355
CPO
-212.1858
0.0236
0.0022
PCPO
160.2841
22 22.2222
0,0189
174.4207
30.1276
0,0264
PCPO
-215.4305
0.3358
0,0003
35

---

Published in Transactions on Machine Learning Research (03/2024)
3.0 1m1
Rewardd Performncc
2.5
Performance
3.0
Rewardd Per Formencce
2.0
2.5
2.0
3.5
1.0
0.5
P3P
18P0-u5L
go
TRFO
BROOU2
CPC
300
CRO
TRPO Lagiinjonn
TRPO
PCP
0.0
MFO Legranginn
TAPO PP
PCF0
T300 Legrnnga
Eovinervtnsssatss
lnb
Ittalnnvinteracts
1eb
Totalennineeracts
145
TRPO
TRaO
TAPO
TRP
TRPO-US
CPO
IPP
1oMd
Cost Performance
0.8
TRPO S.
TRPO she.
Performance
TRPO
Cost performance
2.0
Saae FMC
0.6
8.8
0.6
0.4
0.6
0.4
TotalFnuinteracts
TottlEnvinteracts
10b
Envinteracts
Je-2
Cost Rate Performance
18P0 Lagrongian TRPO-5L
TRPO-AC
1.2
TAPO
TRPO-5L
TEPOLESC
Cost Rate Performance
3.0
TRFO-2
TAPo
TRPO-FAC
-
0.8
0.9
0.8
0.6
0.7
6.5
0.4
TotolEnvinteracts
le5
Totrl Envinteracts
Goal wtiploooiiiatttttaaaaaaaee BHazards
Goal_ Swimmer 8Hazards
Goal_Ant, Ant__ilaaardd
3e1
1e1
1.25
Reward Performance
3
1.00
Roward Performance
3.0
2.5
0.50
2.0
0.00
T8PO fac
Trro
T8P0
-0.50
T8PO PAC
TRAO Ric
Iointtnnitteracts
niiterasss
TotalEnvinteracts
1e1
Je1
1e1
TIPO
HAFO
00
IIFO
1.2
Lagganagn
Paro-Lagaargan
TRROUrO
anglan
Cost Pefforance
1.0
0.9
Cost Performance
1.0
0.8
0.7
0.0
0.6
0.5
0.2
0.4
0.5
0.3
3
2
ToitrErvinteracts
TetalEnvinteracts
Iwh
Tota Envinteracts
1e--
1e-3
1e-2
TSPO
TRPO
TARO-U
TRPO
TRPO-Uq
CP0
PCPO
jRFD-Lapaaneen
1aro-ro
P300
Cost Ratt Performance
Tea0
Cost Performance
TRINO
Cost Rate Porformance
TAPo
0.2
TrtaEnviinerrcts
Jub
TotalEnvinteracts
1ub
Total Envinteracts
105
Goal Walker__lazaaae 8llzzrrs
Goal, Hoaa__Hmanoidd d__ Biazarrs
Goal Maal_Hopper_ HacHHHoopeBaaararreeee BHazards
36

---

Published in Transactions on Machine Learning Research (03/2024)
2v1
1e1
2.0
Reward Perfrraanc
Reward Per Performence
2.5
1.5
1.0
1.5
Roward
0.5
.WPP
TEPO
1RPG USL
CPO
Lagrangies
TRAO
-0.5
TRPO-Laganggnn
TR20-P0
TotalFnvinteractt
1cc
StalFnvinterrcts
106
TotalFnvinteracts
1e5
le1
1775
IIFO
dro
TIPO-USL
oTO KP0
TIP
TEPO-ULL
TREO Lagrangian
TRPO
IRA0 Lagrangaa
TAPO PP
TRPO Lagranglan
TEPO PO
KM0
1.50
1070-51
3.5
TRPO-SL
IRPO-AC
1.25
2.5
3.0
1.00
2.5
0.75
Cost
1.0
1.0
0.5
0.00
TotalEnvinteracts
:e5
TotalEnvinseracts
166
Envinteracts
1e5
Cost Bate Perforaan
11/0
TRPD
IMPC
Pc0
Cost performanc
TRPD:
IIPO
TeAO
18P0 AMC
P00
Cost Rate Performance
TRPC-IN
1000
TRPO-FO
800
3.0
5
0.8
2.0
0.6
1.5
0.5
0.0
IttalEnvinterecs
1e6
TotalEnvinteracts
3e6
Loal_Arm3___
Ara3_@Hazaddds HHazards
Goal_Arm6 HHazarda
Boal_Drone__ 8Hazards
Figure 14: Goal_ Goal_(Robor (Robot)__taale 8tazzarde
37

---

Published in Transactions on Machine Learning Research (03/2024)
1s1
3.5
2.5
3.0
Reward Per formancce
Reward Performanc
2.0
1.5
3850
TRRO-16)
PP
TRPD
TRPO-U9
CR0
CRB
TIUO Lograngnn
TRP
POM
6.0
MFC Legrangian
PCF0
Loggaggng
PCrc
TRa
TRPO
TRP0
TSP
Eovalenvinteracts
le6
IntelEnvinteracts
1ed
Total MtaiEnvinteratts
1e5
Te00
InPO
1.2
TEPO-Lagranginn
TRPO
PC2
POPO
Taro-ro
Cost Perfornancc
0.8
TRPO
Teao 63
Performance
2.0
TEPO
TePO FM.
Cost Perforiance
1.0
0.8
0.6
0.6
0.4
0.4
0.4
0.2
TotalEnvinteracts
lab
Intaeenvinteracss
Iwb
Envinteracts
1.3
Cost Potte Performincs
1.2
Tea0
Cost Performence
TRPO
Cost Robe Performmnc
1.0
0.9
1.0
3.0
0,9
0.6
0.6
0.5
TotalFnuinteracts
Totalnniinteracts
Total Envinteracts
Goal_Pon Point_
Goal_Swiaie Swimmerimrrr 8Chosts
Ant A 8Ghosts
38

---

Published in Transactions on Machine Learning Research (03/2024)
1.25
3.3
Reward Perforancce
Performance
1.00
3.0
Performacc
0.75
0,30
Reward
1.0
0.00
IRFO
IPO
PCPO
-0.25
TRFO Lagrargeas glan
1320
TR20-aZ1
TRAO PO
KP0
RP0
TePO PO
KNN
PAc
Tota Envinteracts
leb
Tiinteniiteeracss
165
TotalEnvinteracts
Jeh
TRA0
TRFO-USE USL
1.2
TAPO
TROOLLET
CRO
TRPO
PCP
IArE Legrangionn
PCPO
Iaro-Lagrnnde
mimb-rg
TB50, St
TREd
Teab
ThPo RMC
Cost Peformance
1.0
Performance
3.0
Cost Cost Performance
1.0
0.8
0.0
0.6
0.4
0.5
TotalFnvinteracts
lub
TotalEnvinteracts
10b
TotalFnuinteracts
Je-2
1.2
TRPO
Tero
Cost Rate Peformance
1.0
TRPO-5L
Cost Rate Performance
TRPQ-44
0.8
0.7
0.8
0.9
0.5
0.4
0.7
BetalEnviinerreets
TotolEnvinteracts
le5
TotalEnvinteracts
Walker_6Ghoate @Ghosts Goal Walker_
Goal, Humanoid, Sahosss
Goal_ _e__HHoopp_____e 8Ghosts
162
1c1
Reward Performance
1.0
Reward Performanc
Reward Performancie
3.5
0.8
1.0
0.5
0.4
0.0
0.2
0.0
TRFO
TRPO
TeAO
PC
titttensinteracts
otalEnvinteracts
IotalEniinteracts
1e1
1e1
IIFO
INFO-USE
00
TRPO
T48
80
and
MPC-PP
TRPD Lagrancians
TRPO-S.
TAPO-AA
2.5
Cost Performance
Cost Porformance
2.4
1.5
Cost
4
2
Total Envinteracts
Total Envinteracts
1e-2
Im-2
1n-3
TRPO
TRPC-19
TRPO
TRPO-151
690
1.75
TRFO
TRPS-0
CP0
Cost Rate Performence
IRFDALggaargan
TA5O
PMC
Pcr0
Cost Performanc nce
HB Leggaaanna
TRAo AMC
TRPO-200
P30
1.50
TRPO
esc
130
1.00
0.75
0.50
Cort
0.25
Jub
TttaEnvinteracts
zub
Tota Envlntnnittss
Goal_ Arm3 8Chosts Ar______m_Cohss
Goal_Arme Arm___________11_11___11_____16_ _Chostts
Coal_Drone Drone__Chostss 8Chosts
Figure 15: Coal (Btoott)__Chhoetoc
30

---

Published in Transactions on Machine Learning Research (03/2024)
1e1
1e1
Reward Performence
1.0
Performence
Roward Rerformancs
0.4
T3P0
DP
10P0
100 0i
TRPO Lag angann
TRPO-FF
KCN
TRPO PO
20PO
TRP
PCP
TotalEnvinteracts
leb
TotalEnvinteracts
1e5
Tota Envinteracts
TRPO
TRaO
TAPO
TRP
CRO
TRPO-US
CPO
Cost Berformance
TRPO
TePo Rtc
1ord
Performance
1.2
TRPO St
TRPO Roc
Cost performance
1.0
TRPO
2.0
0.0
0.6
0.6
0.4
0.2
0.2
0.2
TotalFnuinteracts
TotalFnvinteracts
10b
Envinteracts
1.1
1.0
TRPO-SL
TRPO TRPO-1AC
Cost Performance
TRFO-5
TRA0-NC
KP0
Cost Rate Performnnce
TRPO-
TArO-FAC
TaPo
8.0
0.8
0.7
0.9
0.7
Cozt Ratc
0.6
0.8
0.5
0.4
0.6
0.4
0.3
6.5
TotolEnuinteracts
TotalEnvinteracts
1c6
Totrl Envinteracts
Push P 8Hazards
Push_ Swinmer__Haaeee 8Hazards
Push_Ant, Bathb_nnt_Btaaet sat_Biiaerdde
Roward Performance
Reward Performence
Reward Per Performencc
PRPOPPO TRPO-IC
-1.0
TEPO-SL
TRFO
TRRO
TRPO
Total Envinteracts
iocalEnvinteracts
italEnvinteracts
1e1
Je1
1e1
TIFF
ISF mur
HIFO
IRPOUS
00
HIFO
1.0
JerooLagaarrgnn glann St.
moro-Lgpargian
TRFD-Lag angiann
TRPO-FD
Cost Performance
Performance
2.0
Cost Perforrance
0.8
8.8
0.6
0.6
0.6
0.4
0.2
BttllEnvinteracts
TetaeEnvinteracts
Iwh
Total Envinteracts
1e-2
Ia-2
1e-2
3.1
TRPO
TAFO USL
0P0
1.1
TRPO
TERYUS
C95
TRPO
TRPO-UQ
CP0
1.0
sargs
mrro-aaaarglen
miD-2n
INFO RAC
Pc0
Performence
3.0
TRPO
murb-Lagaargann
1erouro ecc
P300
Cost Rate Performencc
jRFO-Lapaangnn
TAPO
1aro-ro
130
Cort Rate Pefformen
0.9
0.8
8.8
0.
0.7
0.8
0.
0.8
0.6
0.5
0.4
ittaEnvinteracts
Jo6
Trtalnnvinteracts
10b
Total Envinteracts
105
Push Walker__BPi 8llzzrrs
Push_Humanoiee _BHazarda
Push_ Bosh_Hopper___ Baa__Hoper_BHaazazree 0Hazards
40

---

Published in Transactions on Machine Learning Research (03/2024)
Ju2
10
Reward Performance
-2
Perfomanc
-0.2
Roward Performance
-1.0
-1.2
TIFO
TNPOUS.
OPO
-144
TVPO
TUPOUS
Cro
18.0 Lagrongins
TRPO-ULL
CPO
-8
TRPO-L agrangian
-1.6
Lagreng
TRAD-P0
TotaaFnninteraatts
1c6
ToralFnvinteracts
106
1e5
1c-1
mree
mrc-usL
TEPO
TEPO-US USL
DP PCP
TIPO
TUO USL
TRPO Ligrangion
TIPO io
K00
TRPO Lagrangian
TePO PO
TRPO Lagningan
T9P PPC
XPO
IICC-AC
IRFO-SL
IRPO-AC
TRPO-SL
TUOOAC
Cost Performence
Cost formancc
Cost Performance M
Enviitnvinteracts
le6
iotalEnvinteracts
166
TotalEnvinteracts
1e5
Cost Rate Performance
MPF-Luurarddnn HIreS.
TRFC-10
IRFC-PAC
PcP
Cost Performance
TRCO:
IP0
TRAO
I8PO AMC
Cost Rate Performance
TSPO T9PO SL
T5PO 1U/ Acc
Kt0
TatalEnvinteracts
1c0
TotalEnvinteracts
3e6
TotalEnvinteracts
Push Arm3_ _aaazadd
Push_Arm6 HHazarda
Push Drone 8Hazards
Figure 16: Push _Roba((Root)_Btte 8ilzzarde
41

---

Published in Transactions on Machine Learning Research (03/2024)
1.5
0.0
1.0
Reward Perfor Peformance
-0.2
Reward Performanc
0.5
Reward Performance
-0.4
-0.5
-1.0
-0.8
-1.0
TBPO
TRP-4q
PP
18F0
TRPO Mi
CP0
CRO
TNPO ongion
IRF--Po
101
mmro-ugraaggan
TiPo ro
PC10
1.0
TEnn PO
TCre
-1.2
PMC
TRFO
TRAo
RM
TetatEnvinteracts
le6
Total Envinteracts
1ed
Iotatnnvinteracss
1e5
Te0
InP0
TR00
09
TRPO
PC2
1.6
purD-Lapaagge
TRP0 aP
PCO
TRPO4PO
K30
TRIPO
TeAo ACC
1.4
TR-0 6MC
1.50
IRFO
TRPO FSC
Coxt Retorraare
Cost Rerfornanes
3.25
0.8
0.75
0.6
0.4
0.50
TotalEnvinteracts
lab
TocalEnvinteracts
Juh
Envitennins
1.4
Cost Robte Performnncc
1.2
TSPO
Teao
Performente
AMC
Cost Robo Performancc
TAPo
1.0
3.2
3.0
0.8
2.8
0.8
TotaFFnunneracts
TotalFniinteracts
Total Envinterrrss
Chase_Point Point_ Bllazards
Shase_Sximmer__ Bllazards
Chase Ant 8Hazards
42

---

Published in Transactions on Machine Learning Research (03/2024)
1.0
0.5
Reward Reffrrance
0.5
Reward Performance
Reward Performance
-0.5
-0.5
-1.5
TRP3
TRFO-UL
dPO
T3PO
1820 450
CPQ
TRPO
TRPO-2L
aP
2.5
TRPO-SL
IRPO lagrongarr
INFOUPO
KPO
18:0-P0
KCP0
-2.0
TRPO Lagningnn
INFOUPO
FSN
TotalEnvinteracts
365
TotatEnvinteracts
1e0
TotalEnvinteracts
Jeh
TR30
TRPO-LI
TRPO
TRPO
CR0
TRPO
TAPO-US
C20
1.4
K30
IRMO-Lagrargger
PC10
IRAD
TRPO PAC
TRPO Se
TRPO-FCC RMC
TRPO
ThPo RMC
Peroraans
Performance
2.0
Cost Performance
0.5
0.6
2.0
0.4
0.2
0.5
0.0
0.0
TotalFnvinteracts
leb
TotalEnuinteracts
Je-2
Cost Rate Performance
TePO--L
TRPO
PCo
TRPC-SL
TSPO-rIC
TIPO
Cose Rate Refoomance
TRPO-4
TRPO-TIC
2.50
1.4
0.8
3.25
0.6
0.4
0.50
0.8
BttalEnvintersets
TotalEnvinteracts
le5
IstalEnvinteracts
Chase Walker Welker_fllazare SHlzards
Chase Hww.noidd SHazards
Chase Chase_Hoppez hopper_8laazaee
1e-1
Reward Performannc
0.50
Roward Pertormance
Reward Performence
0.00
dh 1
-0.50
A.
-1.00
FAC
TRFO St
TRFO
IotalEnvinteracte
Total nvinteracts
EtalEnvinteracts
1e1
1e1
INFO
INPO-U
INFO-USE
00
800
TRPD-Lagaardinn
TRC--P0
Lagrancian
TRPO
TRFO-S.
TRPO-AAC
Cost Performance
Cost Performance
Cost Performance
2.0
0.1
4
Tota Envinteracts
Total Envinteracts
1e-2
TRSO I=-2
1n-3
TRPO
TRPC-19
200
TRRO-IRI
690
TRFO
TRPS-40
C90
Cost Rate Performance
jRFD-aggarrglnn
TRRO
TRPO PM
marcarg
rco
Cost Pertormanc 1ce
Leggunannn
TEPOLEO
Trao
1.50
1aro-ro
esc
130
1.00
0.75
0.58
0.25
Titannniiteeaatss
lub
TtalEnvinteracts
zub
Tota Envintiittsss
Chase_ Batee__ra________0 Biazards
Chase_Arme _iaazaddaaadss
Chase Laine__Drone_e Brone__llaaard 8Hazards
Figure 17: Chase {
SHazards

---

Published in Transactions on Machine Learning Research (03/2024)
Reward Performancs
Reward Performance
Reward Per Peffommnnce
0.0
-0.5
-3.5
-2
.2.0
1RP0
TRNO-UO
DP
T3PO
mao
TRPO Lagrangion
TRPO-IPO
PCPO
TIPO Lageing on
TAFO Lag anginn
vinterc
lnb
Tttaenniitteracss
1sb
TotalEniinteracts
TRPO
TEPO
TEPO
TEPO-USL
CRO
T3P
TSPO
10
#00-10
K310
100ur
TRPO St.
TB42 AMC
TRPO
T3P SL
T:00 FMC
Cost performance
Performance
Cost Performance
TotalFnuinteracts
10b
TotalFnninteraats
3.5
TAPo
Cozt Rato Peforaanc
TRPO-LL
TRPO-IIC
TRPO-L
Cosl Rate Performnnce
TRPO-.
TRPO-AAC
2.0
2.0
3.0
1.0
0.5
Cost
0.0
Toro/Envinteracts
TotolEnulntoracts
165
Totro Envinteracts
Defense Bocieee_Pottt__eeeeeeeeeeeeeeeee Biazards
Defense Suinnerrrreee 8Hazards
Defense_AAtt, tat_aHazards
1e2
1c2
1c2
-2.4
Reward Performancc
Reward Performance
-2.6
Roward Performance
-2.0
-3.0
-3.2
-3.4
-3.8
Lagrangian
TAPO
TRPO
TRPO
TRFO
intteeeninteraats
ottalnniiteeratss
IttaEnninterrccs
1e1
le1
1890
IRPO-SL
Fim 190
190
TRPO Lagrangian
TRPO-L.
Ture-rcc
TSPO Lagrang
Cost Porformncc
Perfoor formancc
Cost Performance
2
3
Tota Envinteracts
1ec
Enviternin
1m-22
1e-3
1e-2
Coss Rote Porformancc
TRPO
Tago-lagenneee TRAD SL
TRP0 fac
T8P0-19
10P0-a0
69
roo
Cost Performance
TRPO
"spo
7PPO-1
100-rr
695
2000
Cost Rate Performence
2.5
3.0
jRrD-Laganngen
TRPS
TAPO
TRPO-UQ
1aro-ro
P300
CP0
TotalEnvinteracts
Jub
TotalEnvinteracts
1ub
Total Envinteracts
105
Defense Defense_Walken Walker (aalker_BBazaane BHazards
Defense Humanoiimmmnd d_ BHazards
Defense_ Hei_HHoppe__aaarrdddee BHazzrds
44

---

Published in Transactions on Machine Learning Research (03/2024)
1u2
lu2
2.0
Reward Performance
Reward Perforannc
Reward Per Performecc
2.6
Mw
RIFO
TRPO-S.
1900
TRPO USL
CPo
18-0 TR30 Laggnaagnnni
TRFO-USL
CPO
aggraggnnn
10
TRPO agranglen
Totaalnninterratt
1c6
1e5
ToralFnvinteracts
Le5
TRPO
mure
mPc-UsL
TPPS
TEPO-US U00-ULLL
IIND-Lay argan
TRPO Lagrangion
TIPO Po
K30
TRPO Lagrangian
TePO PO
2000
IRFO-L
IIFC-AC
I8FO-SL
Cost Peformence
Cost Perfommance
1.0
0.8
Cost
0.2
Enviinnvinteracts
le6
iotalEnvinteracts
166
Total Envinteracts
165
Cost Rate Performance
INFO-SL IRFC-AC
TaFo-Lggaaddin
PC0
Cost Performance
I8PO
TeAO FAN
Cost Rate Performance
TSPO
1000
TAPO
TUO HSc
K30
2
TatalEnvinteracts
1e6
TotalEnvinteracts
3e6
TotalEnvinterack
Defense Arm3_ Arm3_8aazards 8Hazards
Defense Arm6 _aazadd
Defense Drone 8Hazarda
Figure 18: Defense Defens__[Robot. [Haaar))aeaaarddd
45


Reviews:
[{"review_id": "rAtiDwdENH", "reviewer": "Reviewer_3xHz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- GUARD includes a range of different robotic platform and safe RL tasks\n- GUARD implements a set of SoTA on-policy algorithms and ran evals on them to compare performances\n- GUARD serves as a good survey for existing literature of safe RL\n\nWeakness:\n- GUARD does not propose conceptually different types of constraints that 1) do not existing in SafetyGym and 2) challenge existing safe RL algorithms (it seems most problems can be solved by at least some algorithm); -> for 2), the paper motivated by how safe RL is important for realworld applications, and maybe you could include one task that is more realistic (visual realistic and diverse)\n- As a benchmark, it is important for GUARD to have comprehensive metrics evaluating different algorithms, probably along different axis of constraint complexity; while GUARD evaluated all the implementations of SoTA RL algorithms, it is hard to parse from the plots how different algorithms compare under GUARD", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new benchmark for GUARD safe reinforcement learning. GUARD includes a diverse set of robots with different level of control complexity. GUARD surveys and implements a range of SoTA algorithms. Details of the task design and evaluation of the algorithms are presented in the paper.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe this paper can be greatly improved if the authors think about how to summarize and present their evaluation results better. For example, you can compute the average area under curve for each of the algorithm along different constraint types and construct a spider web plot to help the audience better understand the pros and cons of different camps of algorithms.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lxGIGRdkU7", "reviewer": "Reviewer_haVh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper addresses an important area, i.e., reliable and comprehensive baselines for safe reinforcement learning, and presents a software suite for being able to compare algorithms in a unified way. \n\nWeaknesses\n- The main contribution appears to be the code repository implementing the benchmark, more so than the paper itself. I believe that this is fine, provided that the code has: (1) very clear documentation about how to build on top of the existing implementations (as opposed to just re-running the benchmark experiments); (2) guidelines for contributing back to the repository; (3) an active plan for maintaining and expanding the code repository. Without these elements in place, the usefulness of the code for future research is relatively limited. The main question is whether this is a code release equivalent to what an original paper includes for reproducibility or is more robust and can be used for future experimentation and benchmarking in novel environments and settings, with different architectures, etc. \n- The \"Limitations and future work\" section acknowledges a few important technical tasks including removing dependency on a library marked as deprecated \"recently\". However, this library was deprecated as of November 16, 2022 (https://github.com/openai/mujoco-py/commit/a13903b82f1ab316815e63b3575526005b3b2ae1). Therefore, it is important that there is a reasonable timeline for these changes to be made in order for GUARD to actually be useful as a modern benchmark suite.\n- It is not clear how this codebase would work with other libraries; it appears to be highly customized to a particular set of environments, and is not sufficiently modular. \n- The code has many inline comments but insufficient overall documentation. the code style overall is quite problematic and would be difficult for others to build upon. This includes pieces of code that have been commented out, TODO remarks in the code, extremely long classes with many `if` statements, etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces GUARD, a software suite for evaluating safe RL algorithms. There are a number of benchmarks (from prior work) that have been included in the codebase.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Since the main contribution of the paper is the code repository, more so than the paper itself, it is very important that the authors address the points in Weaknesses above pertaining to the code base [critical for acceptance]\n- There are a number of proofreading, formatting, and grammar issues that should be addressed, including a reference link overlaid on the number for page 3 and the top of page 4, \"more latest\" on page 10, etc.\n- The link to the GitHub repository on page 26 is not anonymized, and even includes author names when visiting the page.\n- Section C.2 on page 26 is missing.", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UFOj4Nii4X", "reviewer": "Reviewer_84ZG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The new evaluation suite seems to be a useful addition.\nWeakness: Analysis and discussion of the results is limited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors are primarily introducing a new benchmark for safe RL.\nThe article describes the more general set of tasks that are included a good range of the algorithms considered in this literature review.\nBesides introducing this new benchmark, they perform an evaluation of the implemented algorithms on the provided tasks and shows the results as plots, and they have a limited discussion of take-aways from it.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The main take-away in the discussion is formulated as  \"Comparing\nwith them, hierarchical RL methods (i.e., TRPO-SL and TRPO-USL) can perform better at cost reduction,\nbut these methods excel at minimizing costs, they may sacrifice some degree of reward attainment in the\nprocess.\" where them refers to constraint based methods. I think one should at least insert a \"while\" after \"but\".\nPerhaps one could also expand upon analysis and discussion of results and see if one can offer more insights to the reader.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: AI Safety is a topic of surging interest, and RL safety in particular is of significant importance in areas like Robotics. As such, widely adopted benchmarks will help steer the field forward. This paper was not recognized to be particularly novel, but the reviewers agree that the released code on GitHub and documentation of evals would be valuable for the field.
Decision: Accept (poster)