Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
Authors: Jiaming Ji
Borong Zhang
Jiayi Zhou
Xuehai Pan
Weidong Huang
Ruiyang Sun
Yiran Geng
Yifan Zhong
Josef Dai
Yaodong Yang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Safe Reinforcement Learning
SafeRL
RL Simulator
TL;DR: A Unified Safe Reinforcement Learning Benchmark.

Abstract: Artificial intelligence (AI) systems possess significant potential to drive societal progress. However, their deployment often faces obstacles due to substantial safety concerns. Safe reinforcement learning (SafeRL) emerges as a solution to optimize policies while simultaneously adhering to multiple constraints, thereby addressing the challenge of integrating reinforcement learning in safety-critical scenarios. In this paper, we present an environment suite called Safety-Gymnasium, which encompasses safety-critical tasks in both single and multi-agent scenarios, accepting vector and vision-only input. Additionally, we offer a library of algorithms named Safe Policy Optimization (SafePO), comprising 16 state-of-the-art SafeRL algorithms. This comprehensive library can serve as a validation tool for the research community. By introducing this benchmark, we aim to facilitate the evaluation and comparison of safety performance, thus fostering the development of reinforcement learning for safer, more reliable, and responsible real-world applications. The website of this project can be accessed at https://sites.google.com/view/safety-gymnasium.

Safety-Gymnasium:: A Unified Safe Reinforcement Learning Benchmark
Jiaming Ji Borong Zhang' Jiayi Zhou 1i, Xuehai Pan', Weidong Huang'
Ruiyang Sun', Yiran Geng'. Yifan Zhong Juntao Dai', Yaodong Yang 1,t
Institute for AI, Peking University
2 Beijing Institute for General Artificial Intelligence (BIGAI)
{j [jiamg-Ji ji, borongzh)@gmaill com, gaiejj@outlooke com
yaodong yaodong..ang yang@pku. edu. cn
Abstract
Artificial intelligence (AI) systems possess significant potential to drive societal progress. However, their deployment often faces obstacles due substantial safety concerns. Safe reinforcement learning (SafeRL) emerges as solution to optimize policies while simultaneously adhering to multiple constraints, thereby addressing the challenge of integrating reinforcement learning in safety-critic critical sce- narios. In this paper, we present an environment suite called Safeet--Gymasiive which encompasses safety-critical tasks in both single and multi-agent scenar- ios, accepting vector and vision-only input. Additionally, we offer library of algorithms named Safe Policy Optimization (SafePO), comprising 16 state-of- the-art SafeRL algorithms. This comprehensive library can serve as a validation tool for the research community. By introducing this benchmark, we aim facilitate the evaluation and comparison of safety performance, thus fostering the development of reinforcement learning for safer, more reliable, and respon- sible worldworld applications. The website this project can be accessed at https::/siits. cww/view/ view//iewww////ee///eeeee safety gymnasium
Introduction
AI systems possess enormous potential to spur socictal progress. However, their deployment is frequently hindered by substantial safety considerations 1244 Distinct from pure reinforcement learning (RL), Safe reinforcement learning (SafeRL) seeks to optimize policies while concurrently adhering multiple constraints, addressing the challenge of employing RL in scenarios with critical
safety implications 5 7778 9.. This strategy proves particularly pertinent in real-world applications
such autonomous vehicles [10] and healthcare [11]. where system failures unsafe actions
large language models (LLMss. some studies have also shown that the toxicity of the models can be reduced through SafeRL Incorporating safety constraints ensures adherence to predefined boundaries and regulatory standards, fostering trust and enabling exploration in environments with high-risk potential. Overall, SafeRL is instrumental in guarantecing the dependable operation of intelligent systems in intricate and high-stake domains.
result in grave consequences, such as accidents or harm to individuals.
12 K
"Equal Contribution. 'Corresoonding author.
Work done when Jiayi Zhou visited Peking University.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Simulation environments have become instrumental in fostering the advancement of RL. Eminent examples such as Gym [4]. Atari [5], and dm-control [T6] underline their importance. These versatile platforms permit researchers to swiftly design and execute varied tasks, thus enabling efficient evaluation of algorithmie effectiveness and intrinsic limitations. However, within the sphere of SafeRI. there is notable dearth of dedicated simulation environments, which impedes comprehensive exploration of SafeRL. In recent years, there have been strides to address this gap. DeepMind presented AI- --SSaftt-Grddwoote properties of intelligent agents T7. A Afterward, OpenAl introduced the Safety Gym benchmark suite, collection of high-dimensionaa continuous control environments incorporating safety-robor tasks [8]. Over the past two years, several additional environments have been developed by researchers, including safe-control-gye gym [19]. MetaDrive 20), etc.
suite of RL environments showcasing various safety
Compared to Safety Gyn Safeeyy-Gymasium inherits and expands the settings of some tasks of Safety Gym, aiming to bolster the community's ' growth further. Compared with Safety Gym, we have made the following major improvements:
Refactoring of the physics engine. Safety Gym utilizes mujoco-py to enable Python- based customization of MuJoCo components. However, mujoco-py stopped updates and support after 2021. In contrast, Safety-Gymassinn um supports MuJoCo directly, eliminating the reliance mujoco-py. This facilitates access to the latest MuJoCo features (e.g.. rendering speed and accuracy improved, etc.) and lowers the entry barrier, particularly due to mujoco-py's 's dependency specific GCC versions and more.
Extension of Agent and Task Components. Safety Gym initially supports only three agents and tasks. On this basis, Safety-Gymnasiun has been further expanded, introducing more diverse agents and task components and expanding safety tasks to cover multi-agent domains. Finally, Safety-Gymnasee um launched a high-dimensional test component based on Issac-Gym (21). further enriching the benchmark.
Enhanced Visual Task Support. The visual components of Safety Gym are simplistic (consist- ing of basic geometric shapes), and mujoco-py relies on OpenGL for visual rendering, which results in significant virtualization performance loss on headless servers. In contrast, Safety- Gymnasium, built on MuJoCo, achieves rendering speeds on CPU that are twice as fast as the former. Additionally, it offers more comprehensive visual component support.
Easy Installation and High Customization. Safety Gym is cumbersome to install and relies heavily on the underlying software. One of the design motivations of Safety-Gymnasti um the ease of use so that everyone can focus on algorithm design. Safety-Gymnasi can be easily installed with one simple command pip install highly integrated framework, Eyfeety-yyymastti only needs 100 lines of code to customize the required environment.
While benefiting from the
In this work, we introduce Safety- -Gymnayymmnnmmm a collection of environments specifically for SafeRL, built upon the Gymnasium [4 22 and MuJoCo [23] Enhancing the extant Safety Gym framework [8]. we address various concerns and expand the task scope to include vision-only and multi-agent scenarios. Additionally, we released SafePO, a single-file style algorithm library containing over 16 tate--of-the-are algorithms. Collectively, our contributions are enumerated as follows:
Environmental Components. We provide various safety-oriented tasks under the umbrella of Safett-Gymmasium These tasks encompass single-agent, multi-agent, and uision-based challenges, each with varying constraints. Our environments are categorized into two primary types: Gymnasium-basee featuring agents of escalating complexity for algorithm verification and comparison, and Issac-Gym Gyym-aaed, incorporating sophisticated agents that harness the parallel processing power of Issac-gym's s GPU. This empowers researchers to explore SafeRL algorithms in complex scenarios. Further details can be found in Section 4
Algorithm Components. We offer the SafePO algorithm library, which comprises a single-file style housing 16 diverse algorithms. These algorithms encompass both single-agent and multi- agent approaches, along with first-order and second-order variants, well as Lagrangian
'Again, we have no intention of attacking Safety Gym: the contribution of Safety Gym to the SafeRL community cannot be ignored, and Safety Gym also inspired this work. We hope that through our efforts. Safety-Cymmast um further promote the development of SafeRL and give back to the entire RL community.

---

and Projection-based methods. Through meticulous decoupling, each algorithm's 's code resides in an individual file. more in-depth exploration of SafePO presented in Section
and SafePO, we conduct detailed analysis of existing algorithms. Our analysis encompasses 16 algorithms across 54 distinct envi- ronments, covering various scenarios such as single-agent and multi-agent setups with varying constraint complexities. This analysis delves into each algorithm's strengths, constraints, and avenues for enhancement. We provide access to all metadata, fostering community verification and encouraging further research. Further details can be found in Section
Insights and Analysis. Combining
2 Related Work
Safety Environments In RL, agents need to explore environments learn optimal policies by trial and error. currently typical to train RL agents mostly or entirely in simulation, where safety concerns are minimal. However, anticipate that challenges in simulating the complexities will cause shift towards training
of the real world (e.g, human-AI collaborative control I 2
RL agents directly in the real world, where safety concerns are paramount [20 24 25], OpenAl
includes safety requirements in the Safety Gym [88, which a suite of high- nigh-dimensional continuous control environments for measuring research progress on SafeRL. Safe-control-gyne [9] allows for constraint specification and disturbance injection onto a robot's inputs, states, and inertial properties through portable configuration system. DeepMind also presents suite of RL environments, Al-Safety-Gridwond
17, illustrating various safety properties of intelligent agents.
SafeRL Algorithms CMDPs have been extensively studied for different constraint criteria [26 27 28: 29]. With the rise of deep learning, CMDPs are also moving to more high-dimensional continuous control problems, CPO 300 proposes the first general-purpose policy search algorithm for SafeRL with guarantees for near-constraint satisfaction each iteration. However, CPO's policy updates hinge on Taylor approximations and the inversion of high- -iggn-inesiional Fisher information matrices. These approximations can occasionally lead to inappropriate policy updates. FOCOPS (3] applies primal-dual approach to solve the constrained trust region problem directly and subsequently projects the solution back into the parametric policy space. Similarly, CUP [32 offers non-convex implementations through a first-order optimizer, thereby not requiring strong approximation of the convexity of the objective.
3 Preliminaries
3.1 Constrained Markov decision process
SafeRL 6 33 is often formulated as a Constrained Markov decision process (CMDP) ( which is tuple M A. R. C, y, Here and A are the state space and action space correspondingly. P(s'| |s, a) is the probability of state transition from to s' after taking action a. R(s'ls, a) denotes the reward obtained by the agent performing action a state 8 and transitioning to state s'. The set C {c,, b.) mm 1=1 where C are cost functions: q : S A + R and the cost thresholds are b;, i 1, m. p()) S + [0, 1] is the initial state distribution and the discount factor ? [0, 1).
stationary parameterized policy Ty is a probability distribution defined on S x A, (a)s)) denotes the probability of taking action a in state s. We II, {mo 0 RP) to denote the set of all stationary policies and is the network parameter needed to be learned. Let P denotes a state transition probability matrix and the components are: P.. [8, 8'1 = PT (s E.EA To (P(a)))P(s))))))) a). which denotes one-step state transition probability from Ty. Finally, let dgg (8) = - -7) Ei 7'P no (s, = s|sa) to be the stationary state distribution of the Markov chain starting so induced by policy no and d (s) E (s)] to be the discounted state visitation distribution on initial distribution
Pro E RSS
by executing
The objective function is defined via the infinite horizon discounted reward function where for a given ne. we have (ma))) = E M( 4z)[s9 ~ μl a, ~ Ta]. The cost func- tion is similarly specified via the following infinite horizon discount cost function: J ((())) =
E n² (,(4141 St, (s) u, ~ ne].
3

---

Then, we define the feasible policy set He as: He = - H. and ( S b,). The goal of CMDP is to search the optimal policy n, n. = arg max (TT))
3.2 Constrained Markov Game
Safe multi-agent reinforcement learning often formulated as a Constrained Markov Game (W, W,S,,A P, 74 R, C, C, Here, (I....,,1) the set of agents, S and A -T== A' are the state space and the joint action space (i.e, the product of the agents' action spaces), P Axx R the probabilistic transition function, μl the initial state distribution, 7 E [0, 1) is the discount fac- tor, R. : S x A + IR is the joint reward function, C = iss is the set of sets of cost functions (every agent has m² cost functions) of the form Cj S x A' - R. and finally the set of correspond- ing cost threshold is given by = {bt iEN
'EN
isssmmi At time step t, the agents are in a state St, and every agent i takes an action at according to its policy n' (a' 8t). Together with other agents' actions, n¹ (a |s) The agents
gives joint action a, = (at..... a.) and the joint policy Ta s) = I receive the reward R ( St, az). meanwhile each agent pays the costs Cj B1,, at). Vj 1, m'. The environment then transits to new state 8,41 ~ IP 82,, ar).
The objective of reward function are (TT) Beo ao 81 function are (T() EEa-- Ao:s ~ Fi.. [E
E 1= 7 R(s:, a.)), and costs
af)] c Vj I,..., m²
We are examining a fully cooperative setting where all agents share a common reward function. Consequently, the goal of safe multi-agent RL is to identify the optimal policy that maximizes the expected total reward while simultaneously ensuring that the safety constraints of each agent are satisfied. Then define the feasible joint policy set πe n a E II and JJ(TT) cj Vj 1..... m²) The goal of CMG is search the optimal policy π,, arg maxxxelll (TT).
Safety Environments: Safety-Gymnasi
um provides seamless installation process and minimalistic code snippets to basic examples, shown in Fiiurre 1 Due to the limited space of the paper, we provide more detailed description (e.g., detailed instructions, the composition of the robot's observation space and action space, dynamic structure, physical parameters, etc.) in Appendix and Online Documentation
Install PyPI:
pip install safety-gymnasee
import safetyygymnassine gymnasiun
# Create the safety-tas task environment
jymnnasummm nake ("Saffetyyoiittoall-ve""" render render_mode="e "human""
# Reset the environment
obs, info rnv.reset() while True:
# Sample randon action
act actiontion sample)
# Step the environment: costs returned
obs, reward, cost, terminated, truncated, info env env.step sacc()ct) if terminated truncated: break
Figure 1: Using Safety-Gymnasiun to create, step, render specific safety-task environment.
4.1 Gynasium-based Learning Environments
In this section, introduce Gymnasium bassdd environment components from three aspects: (1) the robots (both single-agent and multi- aulti-agent); (2) the tasks that are supported within the environment; (3) the safety constraints that upheld.
Online Documentation: www www.safety-oo gymnasium.

---

Supported Robots As shown in Figure 2 Safety-Cymnasee inherits three pre-existing agents from Safety Gym T8 namely Point, Car, and Doggo By meticulously adjusting the model parameters, we have successfully mitigated the issue of excessive oscillations during the runtime of Point and Car agents. Building upon this foundation, we have introduced two additional robots: racecar 35], and ant [23], to enrich the single-agent scenarios. As for multi-agent robots, we have leveraged certain configurations from multi-agent MuJoCo [36], deconstructing the original single-agent structure and enabling multiple agents to control distinct body segments. This design choice has been widely adopted various research works
(a). Point
(b). Car
(c). Racecar
(d). Doggo
(c). Ant
(a).
(b). Half' Cheetah
(c). Humanoid
(d). Hopper
Walker21)
Figure 2: Upper:
The Single-Agent Robots of Gymnasium-bers based Environments. Lower: The
Multi-Agent Robots of Gymnasium-bar baaed Environments.
Supported Tasks As shown in Figure 3, the Gymnasium-basee based learning environments support the following tasks. For more detailed task specification, please refer online documentationf
Velocity. The robot aims to facilitate coordinated leg movement of the robot in the forward (right) direction by exerting torques the hinges.
Run. The robot starts with random initial direction and specific initial speed as it embarks a journey to reach the opposite side of the map.
Circle. The reward is maximized by moving along the green circle and not allowed to enter the outside of the red region, its optimal path follows the line segments AD and BCº.
Goal. The robot navigates to multiple goal positions. After successfully reaching a goal, its location is randomly reset while maintaining the overall layout.
Push. The objective to move a box series of goal positions. Like the goal task, new random goal location is generated after each achievement.
Button. The objective is to activate series of goal buttons distributed throughout the environment. The agent's goal is navigate towards and contact the currently highlighted button, known as the goal button.
Supported Constraints As shown in Figure 3] the Gymnasium- based environments support the following constraints. For more detailed task specification, please refer to our online documentation.
Velocity-Cond Constraint involves safety tasks using MuloCo agents 23 In these tasks, agents aim for higher reward by moving faster, but they must also adhere velocity constraints for safety. Specifically, in two-dimensional plane, the cost is computed as the Euclidean norm of the agent's velocities (v, and v.).
'Task Specification Documentation:
of ww..arrrommeett/ttaassse....s html
5

---

(a), Velocity
(b). Rum
(c). Circle
(d). Goal
(s). Velucity
Hazards
Sigwalls
(0. Grenlims
Figure 3: Upper: Tasks of Gymnasium-based Environments; Lower: Constraints of Gymnasium- based Environments.
Pillars are employed to represent large cylindrical obstacles within the environment. In the general setting, contact with pillar incurs costs.
Hazards are utilized to model areas within the environment that pose a risk, resulting in costs when agent enters such areas.
Sigwalls are designed specifically for Circle tasks. Crossing the wall from inside the safe area to the outside incurs costs.
Vases represent static and fragile objects within the environment. Touching displacing these objects incurs costs for the agent.
Gremlins represent moving objects within the environment that can interact with the agent.
4.1.1 Vision-only tasks
Vision-only SafeRL has gained significant attention as a focal point of research, primarily due to its applicability in real-world contexts 40 41] While the initial iteration of Safety Gym offered rudimentary visual input support, there is for enhancing the realism of its environment. To effectively evaluate based SafeRL algorithms,, we have devised more realistic visual envi- romment utilizing MuJoCo. This enhanced environment facilitates the incorporation of both RGB and RGB-D inputs (as shown in Figure 5). An exemplar of this environment is depicted in Figure 4 while comprehensive deseriptions available in Appendix B.5
(a) Race
(b) The Vision Input Race
(c) FormulaOme
(d) The Vision Input of FormulaOne
Figure 4: Vision-only Tasks of Gymnasium-basee Environments.
4.2
Issac-Gym-based Learning Environments
In this section, we introduce Safety-Daxteroustiandd a collection of environments built upon DexterousHands [42 and the Isaac Gym engine (2]] Leveraging GPU capabilities, Safety- HexterousHands enables large-scale parallel sample collection, significantly accelerating the training process. The environments support both single-agent and multi-agent settings. These environments involve two robotic hands (refer to Figure (a) and (b)). In each episode, ball randomly descends near the right hand. The right hand needs to grasp and launch the ball toward the left hand, which subsequently catches and deposits it at the target location.

---

RGB
RGB-D
stpp-
s5op-150
slpp-300
(6) stcp-100
stcp-600
(6 step-750
Figure 5: The RGB and RGB-D input of Gymnasium- based Environments.
(a) Hand Catch Over
(b): Hand Over
(o): Dynamies
(d): Safety Joint
(e): Safety Finger
Figure 6: Tasks of Safety-DexterousHandds
For timestep 1, let Tb.t. Fig,t to be the position of the ball and the goal, dy.t to denote the positional distance between the ball and the goal dp,t || Zg,el2.. Let de, denote the angular distance between the object and the goal, and the rotational difference is dy. = 2 arcsin min(Idaa [Idg,1 1.0). The reward is defined as follows, Tt exp( 00.0002 (.2 + dr,)), where a is constant balance of positional and rotational reward.
Safety Joint constrains the freedom of joint @ of the forefinger (refer to Figure 6 (c) and (d)). Without the constraint, joint @ has freedom of [-20,, 20% The safety tasks restrict joint 4 within [-10%, 10%. Let ang_4 be the angle of joint @ and the cost defined o [[(an__4 g [-10", 10%).
Safety Finger constrains the freedom of joints 2, and of forefinger (refer to Figure 6 (c) and (c)). Without the constraint, joints @ and @ have freedom of [0 90% and joint @ of -0 20 The safety tasks restrict joints @, @, and within [22.5", 67.5%], [22.5%, 67.5%.. and [-10", 10 respectively. Let ang ang 3, ang 4 be the angles of joints @, @, C and the cost is defined as:
Ct I[[ang_2 2 [22.5", 67.5%, ang & [22.5", 67.5%), ang e [-10%, 10%)).
(1)
Safe Policy Optimization Algorithms: SafePO
This section provides detailed discussion of the design of SafePO, Features such as strong perfor- mance, extensibility, customization, visualization, and documentation are all presented to demonstrate the advantages and contributions of SafePO.
Correctness For benchmark, it is critical to ensure its correctness and reliability. Firstly, each algorithm is implemented strictly according to the original paper (e.g., ensuring consistency with the gradient flow of the original paper, etc.). Secondly, we compare our implementation with those line by line for algorithms with commonly acknowledged open-source code base to double-check the correctness. Finally, we compare SafePO with existing benchmarks (e.g., Setett-Sttrte--Agee and and SafePO outperforms achieves comparable performance with other existing implementatiinss, as shown in Table

---

Safe Policy
Constrained
SafePO
MAPPO-Lag
Optimization
Safe Navigation: Button,
PPO-Lag
Pure Policy
RCPO
HAPPO
TRPO-Lag
CPRO-PID
MAPPO
Safe Manipulation: StrddowHaritVver, FruighFFanisClooDDDra...c ele.
CUP
PPO
FOCOPS
Peojection
PG
PCPO
Safe Velocity: Mulli-Single-geait Ant, Swimmer, Humanuid, etc.
CPO
Natural PG
MАCPO
TRPO
Safety Vision:
Building, Fading,
Single Multi-Agent Pipeline
Environment Wrapper
Gymnasium Isaac-Gym API
User Interface
Keyboard Controller
Efficient Command
Policy
Customized
Benchmarking
Evaluator
Configuration
Tools
Figure 7: The Architecture of SafePD
Extensibility SafePO enjoys high extensibility thanks to its architecture (as shown in Figure 7). New algorithms can be integrated into SafePO by inheriting from base algorithms and only imple- menting their unique features. For example, we integrate PPO by inheriting from policy gradient and only adding the clip ratio variable and rewriting the function that computes the loss of policy π. Similarly, algorithms can be easily added to SafePO.
Logging and Visualization Another necessary functionality of SafePO is logging and visualization. Supporting both TensorBoard and WandB, we offer code for visualizing more than 40 parameters and intermediate computation results inspect the training process. Standard parameters and metrics such as KL-divergence, SPS (step per second), and cost variance are visualized universally. Special features of algorithms are also reported, such the Lagrangian multiplier of Lagrangian-bar -assed methods, g" H H-'-g. g.g" H-16, v*, and X CPO, proportional, integral, and derivative PID-Lagrangian algorithms, etc. During training, and obtain saved checkpoint models. The complete and comprehensive visualization allows easier observation, model selection, and comparison.
inspect the changes of every parameter, collect the log file,
Documentation addition its code implementation, SafePO comes with an extensive docu- m We include detailed guidance on installation and propose solutions to common issues. Moreover, provide instructions on simple usage and advanced customization of SafePO. Official information concerning maintenance, ethical, and responsible use are stated clearly for reference.
Table 1: A comparison between SafePO and other implementations. Results are based on 10 evaluation iterations using over 3 seeds under cost litp_lmmit222 25. 00. J stands for normalized reward from PPO's s performance, jJ signifies normalized cost relative to cost_limic and AvgR/AvgC represents the ratio of the means of both across 10 environments. The + indicates higher rewards better, while the + indicates lower costs (when beyond the threshold of 1.00) are better. Gray and Black depicts violation and compliance with the cost_limin limit.
PILL
POCOES ONUR
NAPPO SARO
SasNerAgee Agrst
SAAS Ra Aguen:
Safee Rart Agins
SANR Naviuatas
2"
JFF
271
$135
$7.6
JTT
$7.4
$1.5
$11
211
JR
053
1.55
013
3.55
4.30
3.53
-8.34
1.55
6.52
0.75
-8.35
0.83
8.31
042
8.00
0.6
8.34
1.31
8.53
1.13
1124
2.08
0.20
8.15
488
8.50
1889
0.35
15.05
8.50
1.3
8.6
0470
8.56
1.80
Salve
281
2Fng
19/
111
-11
171
391
F1
17
391
B111
(1:8)
1100
8.11
1710
0.43
1350
8.50
1370
111
1141
8.3
(18)
110
0311
0.36
8.55
2.75
8.5
831
8.38
1399
0.00
8.5
01.0
1400
0190
035
8.11
010
130
8.34
1151
1131
196
1115
0117
0480
018
168
0141
SafePO's Documentation: httpa///////aa...ooo.ooooaaaaaaa

---

6 Experiments and Analysis
Desbk p5.30
12N 1g
1800 ke
CW94D
1970 mp
8ON-
XS0
MADe
MAGN
macie (MADPOO
(6) Averago Egisodic Reward of Algoritimms
(b) Hor Chart Categorizing Algorithms into Four Classes Rosed on Average Fgisodic Cost
Figure 8: A bar chart analyzing the performance of different algorithms. The left graph compares episodic reward with PPO-Lag [8] (or MAPPO-Lag (3)] for multi-agent), The right graph shows episodic costs proportionally under varying constraints. Single-agent data is from 40 navigation and velocity tasks, and multi-agent data is from all 8 velocity tasks in Safety- Setty--ymmsiuns
Table 2: The performance of single-agent algorithms. JR stands for normalized reward from PPO's performance, and JC signifies normalized cost relative to cost_limit, The t indicates higher rewards are better, while the i. indicates lower costs (when beyond the threshold of 1.00) are better. Gray and Black depicts breach and compliance with the cost limit, while Green represents the optimal policy, maximizing reward within safety constraints.
PPO
PrO-Lag
TRPO-Lag
CPPO-PID
RCPO
CPO
PCPO
CUP
POCOPS
Swfety Navigation
J
J"
1
_
1
JG
1
J
J 1
_ 1
JG
J
J 1
|
.J
J
ANTBUTTONI
1.00
4.42
0.09
0.86
0.23
1.95
0.10
0.70
0.16
2.07
0.12
4.01
0.03
1:01
0.08
0.17
0.01
0.46
ANZCIRCLEI
1.00
16.81
0.79
2.55
0.65
1.05
0.69
1.90
0.63
1.04
0.47
1.07
0.28
1.57
0.60
0.82
0.02
1.22
ANTGOALI
1.00
1.81
0.26
0.94
0.25
0.74
0.67
1.94
0.29
0.78
0.19
0.55
0.89
0.42
0.34
1.33
0.09
0.67
ANzPusal
1.00
1.30
0.13
0.00
11300
0.80
0.13
11000
0.33
0.00
0..17
0.50
0.07
01:00
0.20
0.00
40.30
Q.IS
CARBUTTON]
1.00
16.09
0.01
0.47
40.34
1.08
-.10
0.40
-0.29
1.73
01.08
1.25
0.02
1.90
0.04
5.50
0,54
1.21
CARCIRCi
1.00
8.42
0.81
0.82
1.59
2.77
1.61
1.70
1.30
3.11
1.67
3.13
1.41
1.99
0.76
1.04
0.64
1.12
CARGIOALE
1.00
2.38
0.43
0.30
0.82
1.09
0.03
2.47
0.55
0.85
0.78
1.63
0.61
1.42
0.19
0.63
0.52
0.93
CARPOSHI
1.00
7.16
0.46
0.78
1.38
0.70
0.03
0.47
1.42
0.83
1.14
0.64
2.36
0.32
0.95
0.29
0.36
DoGOOBUTTON
1.00
7.57
0.01
0.03
0.00
1.27
0.01
0.00
0.01
0.09
0.00
0.15
0.00
0.25
0.02
0.45
0.06
3.68
DoodoCiacLEl
1,00
33.14
1.37
0.82
2.36
0.56
1.22
0.31
0.55
4.49
DocGoGoAL!
1,00
2.28
0.00
0.18
0.69
0.00
0.30
0.50
0.00
0.00
0.04
1.27
DoccoPossi
1,00
1.31
0.78
0.46
0.00
0.64
3.40
POINTBUTTON]
1,00
6,06
1.29
0.00
1.61
0.08
0.25
1.53
POINTCIeCLEl
1,00
8.10
1.35
1.24
0.90
P
1.00
1.93
0.71
0.56
1.32
POINTPUSHI
2,51
1.72
1.13
RACECARBUTTONI
13.73
0.00
0,10
3,37
RADECARCTUCLEE
1.30
15.87
0.80
2.18
0.58
0.79
0.81
0.22
0.77
2.11
ACCCCARGOAT.
1.00
4.26
0.51
1.19
0.27
-0.04
0.88
1.18
2.58
0.33
0.13
1:22
0.31
0.62
RACECARPUSHI
1.00
2.34
-0.40
0.00
0.74
1.29
.0884
2.57
0.58
1.92
0.94
0.13
-0116
0.18
-0.06
3.79
0.30
2.04
Sufety Velocity
J
1
JP
4
J
J
J f
3 4
J
.F0
J -
J
J00 J
ANTVEL
1,00
38.33
0.54
0.22
0.53
0.15
0.51
0.41
0.52
0.56
0.52
0.56
0.38
0.41
0.55
0.945
0.55
0.60
HALECHBETAaVV
1.00
36.77
0144
0.00
0.43
1.01
0.48
0.04
0.36
0.56
0.40
0.23
0.25
0.63
0:40
0.17
0.43
0.12
HorPERV..
1.00
22.00
0.58
0.80
0.90
0.71
0.75
0.44
0.5%
0.99
0.733
0.48
0.65
0.51
0.73
0.21
0.66
0.30
HUMANNIDDVL
1.00
38.42
0.72
0.76
0.72
2.38
0.75
0:00
0.68
0.82
0.21
0.01
0.64
0.01
0.68
0.80
0.71
0.93
SWINMMERVLL
1.00
6.61
0.57
1.1]
0.66
0.84
0.91
0.92
0.54
0.90
0.51
0.82
0.50
0.68
0.59
0.95
0.47
1.30
WALKERZDVEL
1.00
36.11
0.44
0.20
0.51
0.77
0.27
0.36
0.49
0.15
0.39
0.11
0.27
0.71
0.44
0.18
0.50
0.16
Reward and Cost. Episodic reward and cost exhibit a trade-off relationship. Unconstrained algorithms aim to maximize reward through risky behaviors. HAPPO [37] achieves higher rewards compared to MAPPO (38) across 8 velocity- tasks, accompanied by a simultaneous increase in average costs. SafeRL algorithms tend to maximize reward while adhering to constraints. As depicted in Table 2 in the velocity task, compared to PPO 43 PPO-Lag [18) achieves reduction of 98% in cost while only experiencing a decrease of 45% in reward.
Randomness and Oscillation. The randomness of tasks is correlated with the oscillation of algo- rithms' performance, All SafeRL algorithms achieve average episodic costs within the cost_lim for velocity tasks. The divergence in episodic rewards between algorithms is negligible, and the distribution of optimal policies is tightly clustered. However, pronounced oscillations are present in navigation tasks characterized by high stochasticity. Out of the 20 navigation tasks examined,

---

optimal policies are spread out more, leading to observable differences in algorithm performance across various tasks.
Lagrangian vs. Projection.
contrast
projection-based methods, the Lagrangian- based methods tend to display more oscillation.
Table 3: The normalized performance of SafePO's multi-agent algorithms on Safety-Gymnas ium. ium.
notable disparity becomes apparent upon ex-
amining the oscillatory patterns in the episodic
MAPPO
HAPPO
MAPPO-Lag
MACIO
cost around the designated safety constraints dur-
Safsty Velecity
J
y
1P
J
J
JG
2SSANTVEL
1.00
35.76
1.26
98.12
0.57
0.00
0.51
0.14
ing training, as presented in Figure 8))). Both
4XEANT VEL
1.00
35.01
34.34
8.50
0.00
0.01
CPO [30] and PPO-Lag [88 demonstrate oscil- lations; however, those exhibited by PPO-Lag are more conspicuous. This discrepancy man- ifested in a higher proportion of instances clas- sified as Strongly Unsafe and Strongly Safe for PPO-Lag, while CPO maintains a more centered distribution. Nevertheless, excessively cautious policy has the potential to undermine performance. In contrast, the projection-based method PCPO [3] exhibits lower average costs and rewards in navigation and velocity tasks than CPO. This distinction is further accentuated when examining the contrast between MACPO and MAPPO-Lag.
ESSHALSCHETTTANN
1.00
3xHHommaNe SEUURAANOIVVE
1.00
23.05
0100
100
1.00
6.34
2.79
17.18
0.54
0.84
0.53
1.30
2x SAS.AARRTTOON
1.00
77.59
1.55
33.67
0.00
0.01
0.37
1.21
Lagrangian vs. PID-Lagrangian Incorporating PID controller within the Lagrangian based framework proves to effective mitigating inherent oscillations. As shown Figure 8] CPPO-PID 44 displays episodic rewards during training that closely resemble those of PPO-Lag. However, CPPO-PID demonstrates reduced frequency of instances entering the Strongly Unsafe region, resulting in more significant proportion of Safe states and improved safety performance.
7 Limitations and Future Works
Ensuring safety remains a paramount concern. Across various tasks, safety concerns can be trans- formed into corresponding constraints. However, limitation of this study is its inability encompass all forms of constraints. For instance, safety constraints related to human-centre considerations paramount in human-Al collaboration, yet these considerations have not been fully integrated within the scope of this study. This work focuses on safety tasks within simulated environment and introduces extensive testing component. However, the transferability of the results to complex real-world safety-critical applications may be limited. A promising work for the future involves transferring policy refined within the Safety Meety-yymnssium physical robotic platforms, which holds profound implications.
10

---

References
[1] Tom Carlson and Yiannis Demiris. Increasing robotic wheelchair safety with collaborative control: Evidence from secondary task experiments. In 2010 IEEE International Conference Robotics and Automation, pages 5582-5587. IEEE, 2010.
[2] Zhu Ming Bi, Chaomin Luo, Zhonghua Miao, Bing Zhang, WJ Zhang, and Lihui Wang. Safety assurance mechanisms of collaborative robotic systems in manufacturing. Robotics and Computer-Integratte Manufacturing, 67:102022, 2021.
[3] Tsung- T's Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. arXiv preprint arXiv:2010.03/52 2020.
[4] Fengshuo Bai, Hongming Zhang, Tianyang Tao, Zhiheng Wu, Yanna Wang. and Bo Xu. Picor: Multi-task deep reinforcement learning with policy correction. Proceedings of the AAAI Conference on Artificial Intelligence, (7(6):6728-6) 267768,,,,, Jun. 2023.
[5] Keith W Ross and Ravi Varadarajan. Markov decision processes with sample path constraints: the communicating case. Operations Research, 37(5):780-790, 1989.
[6] Eitan Altman. Constrained Markov decision processes. Routledge, 2021.
[7] Linrui Zhang, Qin Zhang, Li Shen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Evaluating model-free reinforcement learning toward safety-critical tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37(12), pages 15313-15321, 2023.
[8] Jiaming Ji. Jiayi Zhou, Borong Zhang, Juntao Dai, Xuchai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research, 2023.
[9] Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang. Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O' D'Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo, and Wen Gao. Ai alignment: comprehensive survey, 2024.
[10] Shuo Feng, Haowei Sun, Xintao Yan, Haojie Zhu, Zhengxia Zou, Shengyin Shen, and Henry X Liu. Dense reinforcement learning for safety validation of autonomous vehicles. Nature, 615(7953)x620-627, 2023.
[11] Yafei Ou and Mahdi Tavakoli. Towards safe and efficient reinforcement learning for surgical robots using real-time human supervision and demonstration. In 2023 International Symposium on Medical Robotics (ISMR), pages 1-7. IEFF, 2023.
[12] Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang. and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via -umaa-preeereece dataset, 2023.
[13] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback, 2023.
[14] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: arXiv:1606.000 1606.0154, 2016.
[15] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXiv:1312.5602, 2013.
[16] Saran Tunyasuvunakool. Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm iim_control, Software and tasks for continuous control. Software Impacts, 6: 100022, 2020.
[17] Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq. Laurent Orseau, and Shane Legg, Ai safety gridworlds. arXiv preprint arXiv: 2017.

---

[18] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep rein- forcement learning. arXiv preprint arXiv:9110.01708. 7(1):2. 2019.
[19] Zhaocong Yuan, Adam W Hall, Siqi Zhou, Lukas Brunke, Melissa Greeff, Jacopo Panerati, and Angela P Schoellig. Safe-control-gyy A unified benchmark suite for safe learning- based control and reinforcement learning in robotics. IEEE Robotics and Automation Letters, ([4):11142-1 11149, 2022.
[20] Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 45(3).3461-3475 2022.
[21] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, al. Isaac gym: High performance gpu- gpu-based physics simulation for robot learning. arXiv preprint (xXiv:2108.10477 2021.
[22] Farama Foundation. A standard api for single-agent reinforcement learning environments, with popular reference environments and related utilities (formerly gym). https:///gthhi: com/F
arama- arama-Foundation Foundationn /aton//Gymnnnnuunnn 2022.
[23] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[24] Mengdi Xu, Zuxin Liu, Peide Huang, Wenhao Ding, Zhepeng Cen, Bo Li, and Ding Zhao Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability. arXiv preprint arXiv:2209.000 00020, 2022.
[25] Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 10330, 2022.
[26] Lodewijk CM Kallenberg. Linear programming and finite markovian control problems. MC Tracts, 1983.
[27] Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
[28) Juntao Dai, Jiaming Ji, Long Yang, Qian Zheng, and Gang Pan. Augmented proximal policy optimization for safe reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6))7288-77777 -795, Jun. 2023.
[29] Weidong Huang, Jiaming Ji, Borong Zhang, Chunhe Xia, and Yaodong Yang. Safedreamer: Safe reinforcement learning with world models, 2023.
[30] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization.
In International conference on machine learning, pages 22-31. PMLR, 2017.
[31] Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 1534598--1334, 2020.
[32] Long Yang, Jiaming Ji, Juntao Dai, Yu Zhang, Pengfei Li. and Gang Pan. Cup: A conservative update policy algorithm for safe reinforcement learning, 2022.
[33] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[34] Patrick L Jacobs, Stephen E Olvey, Brad M Johnson, and Kelly Cohn. Physiological responses to high-speed, open-wheel racecar driving. Medicine and science in sports and exercise, 34(12): 34(12):2085-2090 2002.
[35] Johannes Betz, Alexander Wischnewski, Alexander Heilmeier, Felix Nobis, Tim Stahl, Leonhard Hermansdorfer, and Markus Lienkamp. software architecture for an autonomous racecar. In 2019 IEEE 89th Vehicular Technology Conference VVTC001--Spring pages 1-6. IEEE, 2019.
12

---

[36] Christian Schroeder de Witt, Bei Peng, Vierre-lexandrr Kamienny, Philip Torr. Wendelin Böhmer, and Shimon Whiteson. Deep multi- reinforcement learning for decentralized continuous cooperative control. arXiv preprini arXiv:2003.067009 19, 2020.
[37] Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in mulli--gent reinforcement learning, arXiv preprint arXiv: 11251, 2021.
[38] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 1-22444,,, 2022.
[39] Shangding Gu, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang, Zheng Tian, Jun Wang, Alois Knoll, and Yaodong Yang. Multi-agent constrained policy optimisation. arXiv preprint 10.00:21 2021.
[40] Yecheng Jason Ma, Andrew Shen, Osbert Bastani, and Jayaraman Dinesh. Conservative and adaptive penalty for model-based safe reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 36(5), pages 5404-5412, 2022.
[41] Yarden As, Ilnura Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy opti- mization via bayesian world models. arXiv preprint (rxXix2201.09000 2022.
[42] Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen McAleer, Hao Dong, Song-Chun Zhu, and Yaodong Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning. Advances in Neural Information Processing Systems, 35:5150-5163, 2022.
[43] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
[44] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pages PMLR, 2020.
[45] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 150.00550,,2233,, 2015.
[46] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: l0t.::412.69900 6980, 2014.
13

---

A Details of Experimental Results
A.1 Hyperparameters Analysis
This section presents the disclosure of SafePO hyperparameters settings and their rationales. We employed the Generalized Advantage Estimation (5 method to estimate the values of rewards and cost ad advantages and used Adam [4] for learning the neural network parameters.
Single-agent Algorithm Settings. The models employed in the single-agent algorithms were 3-layer MLPs with Tanh activation functions and hidden layer sizes of [64, 64), for more intricate navigation agents Ant and Doggo, hidden layers of [256, 256] were employed.
Multi-agent Algorithms Settings. The models employed in the multi-agent algorithms were 3-layer MIPs with ReLU activation functions and hidden layer sizes of [128, 128).
Table 4: Hyperparameters of SafePO algorithms in Safet Safety-Gymant G-m--aiinininnn um tasks. Second-order algo- rithms set the parameters to the actor model directly, instead of iterative gradient descent, so the Actor Learning Rate of them are marked Gray.
PL. PLOPPILPOO
Vadao
TRPOOERRO
Valer
CPPO PIB
Valer
SPG PO
Take
Take
E37
Floarr
EPP:
GAll
0L5
dud Sae
CAS
035
CAE.1
Toaky Back Soz
Real Site
2000
Tralk Sioe
Destine
25.00
Corll
26.00
2403
2500
208
Laginapan bisal Valae
Lagttagiin Sn Valae
2005
411
8.5
ChO
Take
PCPO
Valae
CUP
Vale
FOCOPS
Take
MACIP
Tale
D19
CAS
E.55
CAS
D45
CAF
Tranngg
33000
Sac
Sach
Thimin Blach
19:00
heir
June
01011
Cod leet
1516
15.00
Tage
Styp.
Starchin Seis.
Skyy Raike
(.90
Vay Pratees
0.55
CEP
2/6
3.00
0153
Lagrangian Multiplier Settings. Lagrangian- based methods are sensitive to hyperparameters. We present the following detailed description of the settings for both the naive and the PID-controlled Lagrangian multiplier.
Lagrangian Initial Value: The initial value of the Lagrangian multiplier. It impacts the ctrly-stage performance of the Lagrangian based methods. A higher initial value promotes safer exploration but may impede task completion. Conversely, a lower initial value delays the agent's exploration of safe policies.
Lagrangian Learning Rate: The learning rate of the Lagrangian multiplier. A high learning rate induces excessive oscillations, impedes convergence speed, and hinders the algorithm's ability to attain the desired solution. Conversely, a low learning rate slows down convergence and adversely affects training.
PID Controller Kp: The PID controller's proportional gain determines the output's response to changes in the episodic costs. If pid_kp is too large, the Lagrangian multiplier oscillates, and performance deteriorates. If pid_kp k_ is too small, the Lagrangian multiplier updates slowly, also impacting performance negatively.
PID Controller Kd: The PID controller's derivative gain governs the output's response to changes in the episodic costs. If fidppikkt kd is too large, the Lagrangian multiplier becomes excessively sensitive to noise or changes in the episodic costs, leading to instability or oscillations. If pid_kd kd is too small, the Lagrangian multiplier may not respond quickly or accurately enough to changes in the episodic costs.
PID Controller Ki: The PID controller's 's integral gain determines the controller's 'ss ability to eliminate the steady-state error by integrating the episodic costs over time. If pid_ki is too large, the Lagrangian multiplier may become overly responsive to previous emrors, adversely affecting performance.
14

---

A.2 Performance Table of Safety-Gymnasine
Table 5: The performance of SafePO algorithms on Safety-Gymnasiur All experimental outcomes were derived from 10 assessment iterations encompassing multiple random seeds and under the experimental setting of cost cost_limit= 1:2..=25.00. 00. The indicates higher rewards are better, while the I indicates lower costs (when beyond the threshold of 25.00) are better. Gray and Black depicts breach and compliance with the cost_lime t, while Green represents the optimal policy, maximizing reward within safety constraints.
(a) The performance of SafePD single-agent algorithms on Safety- -amoaas-mmaaiii i ium.
PPO
P70-Lag
CPPO-PED
TRPO-Eag
RCPO
CPO
PCPO
CUP
FOCOPS
Safety Novigations
JG
1
Ast TTTB
18.20
100.00
3.83
11.00
4.06
17:45
3.50
5.10
$1-70
4.30
100.30
1.27
23.35
1.26
4.25
11:23
33.55
ANTEINCI
54.01
11:31
13:00
64.90
67.50
36.00
36.30
26.47
56.35
30.50
ANTDiic x2
84.83
73200
05:33
23.45
63.00
10:11 15
66.75
63.04
13
53.70
03.00
16.41
1385
42.65
10.80
4.78
66.33 30
DiALL
0282
45.30
21.35
23.00
38.29
48.35
20.44
13.50
23.38
39.00
13:35
13.30
7.31
10.30
27.05
6.99
36:75
0.46
17.55
6.96
1100
DC6
0.00
0.16
0.00
0.15
0.50
D.28
0.00
0.03
0.00
0.09
0.50
-0.14
0.70
A
0.77
130
0.1
0.00
0.01
0.01
0.00
0.10
0.00
D.25
0.00
0.00
D.00
0.07
0.10
0.07
0.30
CARBUTTONN
19.32
52
23.00
40.00
35.50
0.40
0.50
154.50
01.00
$3.49
CARCIRCLE
21.92
255
12.51
30.02
35.71
31.42
50.18
33.25
10
79.23
11.33
23.30
16.30
25.00
19.03
23395
CARCIRCLL2
38.43
1.00
30.64
6.38
15.07
13.58
33.39
31.59
0.90
16:81
813
23.00
1.7
15
7.01
3.55
37.63
1.85
57.50
3.45
3.36
33.01
CasPaall
002
11.03
0.59
0.08
59.00
0.34
23.06
0.31
8.56
DoccoB-T:ON:
25.23
199.50
6.33
0.50
0.2
1,67
0.01
31.35
0.0
2.25
023
3.50
-0.0
5,30
0.67
11.17
1.53
91.90
191.0
0.10
1.00
016
3.70
4000
17.00
0.55
1.40
8.01
0:35
0120
5.10
14.56
1350
30.55
13:44
15.00
55.05
50.34
117.3
DoccodnAz)
43.10
$7.10
2000
1.00
0.2
17:25
5:33
$2.05
12:75
12
0.16
1000
0.16
22.45
1.55
31.90
0.25
0.56
DoeesPuw
12:30
0.16
0.16
1.00
63.70
0.00
DecesPisn
0.90
12:70
0120
0.7
11:05
0.18
m
0.98
18:50
D411
0.00
0.3
1000
0.77
17 17
0.55
76.10
151
-1.3
33.36
3.30
10-16
2.18
54.38
4.70
0.50
Poss PoSTCCRCCLE]
$4.57
202 54
47.00
13.28
03.54
52.33
50.51
33.55
90.05
35.33
10
43.50
12.31
56.53
46.5
15.50
45.05
21.36
Pres (C
51:34
3167
$2.07
5,03
K3.10
7.50
31.33
19-33
27.87
47.35
3046
Poss (aaat
26-41
159.25
11.55
2.55
1.30
PoesTPusul
0.82
51.80
0.90
33.18
0.29
3.50
0.70
24.05
4.3
33.07
1.82
19.00
1,41
31.53
100
19.06
0.55
53.64
Poss
1.01
25.87
1.05
1.50
23.57
KAmbABBTTOO
343.15
0.15
130
$4.20
5.77
259
-054
33.25
1154
31.00
0.19
05.00
0.31
18.25
1000
12.00
1180
51.00
-0.05
72.55
-1.0
5766
81.02
331
17.00
18.05
01.68
03.77
35.00
11.29
105.40
2.90
12.70
0.433
36.37
13.40
19.20
5.90
35.70
13.30
04.50
3.72
5.00
1.47
30.57
3.47
15.40
0.61
158
0.00
$1.00
0.55
50.50
0.40
14.30
0.5
15.00
1.19
109.85
0.60
41.00
4.09
02.53
0.17
33,05
213.95
41.88
35.07
-0.51
50.00
Sufety Vilecle
1"
10
1
1
35
27
ASTVaL
5899.64
07
3021
9070
10:23
3157.40
3106.77
2976.19
10.18
5200 20
23.56
3291.50
15,07
RAI RALFCIRTTAIVEE
3013.92
033 18
3025 42
0.00
3336
100
295208
25.35
2530 50
13.05
3738
5.68
174577
15644
2265.2
4.28
2873.14
2.58
2319:11
543
138111
11:11
1397.9
17.0
128122
1319300
1230
1710.3
5.37
1518:54
7.45
121.25
171
00.30
106344
79A1
96
06.13
03.00
30.80
50.45
12.31
70.55
22.00
13.30
12.62
6312.07
099.80 82
61
4.50
174.06
3.00
3200.78
19.15
3472 07
3.72
548082
20.15
1666.3
13.73
2750.50
50
4.39
311600
3.55
(b) The performance of SafePO multi agent algorithms on Safety-Gymaasee
MARPO
HAPPO
MAPPIALaR
MALPO
Safct Vducty
___
IX DNAANTVE:
4359-52
504.05
5353.61 61
675.10 m3
2423.47
0.07
3160-23
3,39
402ANTV
4329.0
401140
2171.40
1000
2172-11
E.17
415.90
3583.5
543
1490.38
0.1
1850U5
0.33
974.59
138.81
2218.45
ml
$35.99
00
512.59
32.50
2x 20.
32.05
101.99
$1.55
27.84
59.73
-11G
3133
Experimental Results Analysis.
During the observation of the experimental results, we have discovered some Insightful findings that presented below:
The Lagrangian method is a promising yet constrained baseline approach, successfully optimizing rewards while adhering to constraints. However, its effectiveness heavily relies hyperparameters configuration, as discussed in Table A. Consequently, despite being dependable baseline, the Lagrangian method is not exempt from limitations.
Second-order algorithms perform worse in achieving higher rewards in the MuloCo velocity series but better in navigation series tasks that require higher safety standards, ie., achieving similar or approximate rewards while minimizing the number and smoothness of cost violations.
15

---

B Details Documentation of Gymnasium-basse Learning Environments
B.1 Single-agent Specification
(8) Form
(d) four ritt
Figure 9: A different view of the robot: Point.
Table 6: The overall information of Point
Specific Action Space
Box(-1.0, 1.0, (2.), Bloa644)
Specific Obscrvation Space
(12.)
Observation High
inf
Observation Low
-Inf
Table 7: The specific observation space of Point
Size Observation
Min
Max
Name (in XML tile)
Joint/Sitc
Unit
accelerometer
-inf inf accelerometer
site
acceleration (m/s*2)
velocimeter
-inf int velocimeter
site
velocity (m/s)
-inf int
gyro
site anglular velocity (rad/s)
magnetometer
inf inf
magnetometer
site magnetic flux (Wb)
Table 8: The specific action space of Point
Num
Action
Control Min
Control Max
Name (in XML file)
Joint/Site
Unit
force applied on the agent move forward backward velocity of the agent, which is around the 7-axis
-1
site
force (N)
-1
z
hinge
velocity (m/s)
Point: As shown in Figure 9 Point operating within 2D plane equipped with two distinct
actuators: one for rotation and another for forward/backward movement. This decomposed control system greatly facilitates the navigation of the robot. Moreover, there is small square positioned in front of the robot, aiding in the visual identification of its orientation. Additionally, this square plays a crucial role assisting the robot, named Point, to effectively push any boxes encountered during its tasks. The overall information of Point, the specific action and observation space of Point is shown in Table Table
Table 7
03 Car
(bjCr
(63Cr
(6 right
Figure 10: different view of the robot: Car.
Table 9: The overall information of Car
Specific Action Space
Box 1.0, (2.), floa164)
Specific Observation Space
Observation High
inf
Observation Low
in
16

---

Table 10: The specific action space of Car
Num
Action
Control Min
Control Max
Name (in XML file)
Joint/Site
Unit
force to applied on left wheel
left
hinge
force (N)
force to applied on right wheel
right
hinge
force (N)
Table 11: The specific observation space of Car
Size
Observation
Min
Max
Name (in XMI. file)
Joint/Site
Unit
Quaternions
which are
iinf
inf
ballquat_ rear
ball
unitless
turned into 3x3 rotation matrices Angle velocity of the rear wheel accolerometer
inf
inf hallangvel rear
rear
ball
anglular velocity (rad/s)
inl accelerometer
site
acceleration (m/s*2)
velocimeter
inf velocimeter
site
velocity (m/5)
8Y10
inf gyro
anglular velocity (rad/s)
magnetometer
inl
inl
magnetometer
sile
magnetic flux (Wh)
Car: As shown in Figure 10 the robot in question operates in three dimensions and features two independently driven parallel wheels, along with a freely rolling rear wheel. This design requires coordinated operation of the drives for both steering and forward/backwarc movement. While the robot shares similarities with basic Point robot, possesses added complexity. The overall information of Car, the specific action and observation space of Car is shown in Table Table 10
Table 11
rile
Figure 11: A different view of the robot: Racecar.
Table 12: The overall information of Racear
Specific Action Space
Box(F220. -0.7851, [20. 0. 0.785J, (2,), floa664)
Specific Observation Space
(12,)
Observation High
inl
Observation Low
-inl
Table 13: The specific action space of Racecar
Num
Action
Control Min
Control Max
Name (in XML file)
JointSite
Unit
Velocity of
-20
20
diff_ring
hinge
velocity (m/s)
rear wheels.
Angle the front
-0.785
0.785
hinge
hinge
angle (rad)
wheel.
Table 14:
The specific observation space of Racecar
Size
Observation
Min
Max Name (in XML file)
Joint/Site
Unit
accelerometer
-inf inf accelerometer
site acceleration (m/s*2)
site
velocimeter
gyro inf inf velocimeter gyro
inf inf
velocity (m/s)
site anglular velocity (rad/s)
magnetometer inf inf magnetometer
site magnetic flux (Wb)
Racecar. As shown in Figure 11] the robot closer to realistic car dynamics, moving three
dimensions, it has one velocity servo and one position servo, one to adjusts the rear wheel speed to
17

---

the target speed and the other to adjust the front wheel steering angle to the target angle. Racecar references the widely known MIT Racecar project's dynamics model. For to accomplish the specified goal, it must coordinate the relationship between the steering angle of the tires and the speed, just like human driving car. The overall information of Racecar, the specific action and observation space of Racecar is shown in Table 12
Table 13
Table 14
Table 15: The overall information of Ant
Specifii Action Space
Box(-L.0, 1.0, (8.), Noa164)
Specific Observation Space
Obscrvation High
inf
Observation Low
Table 16: The specific action space of Ant
Num
Action
Control Min
Control Max
Name (in XML lile)
Joint/Site
Unit
torque applied on the
rotor between torso and front left hip
hip_ (front_lcff leg)
hinge
torque (N m)
torque applicd the
rotor between the front
angle_ Jeft
hinge
torque (N
left two links
torque applied on the
2
rotor between the torso and front right hip
hip_2 (front_right right_
hinge
torque (N m)
torque applied on rotor between front
angle_2 [troon_riggt_lege
hinge
torque (N
right two links
torque applied on the
4 rotor between the torso and back left hip
hip (back_
hinge
torque (N m)
torque applied on the
5 rotor between back left two links
angle_ _ (hack_ leg)
hinge
torque (N m)
torque applicd the rotor hetween the torso and back right hip torque applied on the
hip_4 begk_leg)
hinge
torque (N
7 rotor between the back right two links
angle backhh_bacck__u log)
hinge
torque (N m)
Ant. As depicted Figure 12 the quadrupedal robot, inspired by the model proposed in 455 consists of a torso and four interconnected legs. Each leg is composed of two hinged connecting limbs, which, in turn, are connected the torso via hinges. To achieve movement in the desired direction, coordination of the four legs is required by applying moments to the eight hinge drivers. For comprehensive understanding of the robot, please refer to Table 15 Table 16 and Table 17 which provide an overview of the Ant robot, its specific action space, and observation space, respectively.
Figure 12: A different view of the robot: Ant.
18

---

Table 17: The specific observation space of Ant
Size
Observation
Min
Max
Name XML file)
Join/Site
Unit
accelorometer
accelerometer
sitc
acceleration (m/s"2)
velocimeter
inn
inf velocimeter
site
velocity (m/s)
gyro
-inf
inf
gyro magnetometer
site
anglular velocity (radis)
magnetometer
site
magnetic flux (Wh)
angular velocity angle
hip I left__e hinge
angle (rad)
between torso and front left link
angular velocity of the angle
inn
inf
ankle_ I (front_left left_
hinge
angle (rad)
between front left links
angular velocity of angle
inn
inf
hip_ 2 (front_right_legg
hinge
angle (rad)
between torso and front right
angular velocity of the angle
ankle_2 (front_right_eeg
hinge
angle (rad)
between front right links
angular velocity angle
between torso and back left link
inf
inf
hip_3 (back_legg
hinge
angle (rad)
angular velocity of the angle
hetween hack links
inf
inf
ankle_3 (back_leg) leg)
hinge
angle (rad)
angular velocity angle
between torso and back right link
hip_4 (tightttbbckkkeeeg back_leg)
hinge
angle (rad)
angular velocity
angle
inf
inf
ankle_4 (right (right_back back_ leg)
hinge
angle (rad)
between back right links
z-coordinatc of the torso
(centre).
inn
inf
site
position (m)
cor-donnee angular velocity the tors.
torso
site
angular velocity (rad/s)
sin() and cos() of angle
-inf
inf
hip_ (front_left Jegt_ee)))
hinge
unitless
between torso and first link front left
sim() and cosi) of angle
inf
inf
ankle_ I (front_left _eft__ee))
hinge
unitless
front left
sin() and cos() of angle
hip_2 fifoontrighh_llgg
hinge
unitless
between torso and first link on front left
sin() cos() angle
-in
inf
ankle_2 2 (frent_right reght__gggt
hinge
unitless
between torso and first link front left
sin() and cos() of angle
between and link front left
iin
inf
hip_3 (back_leg)
hinge
unitless
sin() and cos() of angle
ankle_3 (back
hinge
unitless
hetween torso and first link on front left
sin() und cos() of angle
-inf
inf
hip_4 (right_ Bright_buckkler
hinge
unitless
between torso and first link front left
sim() and cosi) of ungle
inf
inf
ankle_4 (right_back back__acggll
hinge
unitless
hetween
link front left
B.2 Multi-agents Specification
(a) 2-ant: Render
(b) 2-ant-diag: Render
(c) 4-ant: Render
(d) Dynamics
Figure 13: A different view of the MA-Ant.
2-ant. The Ant is partitioned into parts, the front part (containing the front legs) and the back part (containing the back legs). The action space of agent-O and agent-1 as shown Table and Table 19
19

---

Table 18: The specific action space of 2-ant: agent-0
Num
Action
Control Min
Control Max
Name (in XML lile)
Joint
Unit
Torque applied on the rotor
between the torso front
hip Jeft
hinge
torque (N m)
left hip
Torque upplied the rolor between the front left
angle_ (front_leff lefont_leff_eggggee___
hinge
torque (N m)
Two links
Torque applied on the rotor between the torso and front
2
hip (front_rightt regght))
hinge
torque (N m)
right hip
Torque applied between the front right
rolor
angle_ 2 (front_ (front_right__ee right__og)
hinge torque
Two links
Table 19: The specific action space of 2-ant: agent-l
Num
Action
Control Min
Control Max
Name (in XML file)
Joint
Unit
Torque applied the rotor hetween the torso and front
0
hip_1 Jeft_t__eft__egg
hinge
torque (N m)
left hip
Torque applied the rotor
between the front
angle_ (front_left_
hinge
torgue (N m)
two links
Torque applied the rolor between the torso and front
2
hip (front (front_right__eee
hinge
torque (N m)
right hip
Torque applied on the rotor between the front right
3
angle_2 (fronL_right_ee)
hinge
torque (N m)
two links
2-ant-diag, The Ant is partitioned into 2 parts, split diagonally, the front part (containing the front legs) and the back part (containing the back legs). The action space agent-0 and agent-1 as shown in Table and Table 21
Table 20: The specific action space of 2-ant-diag: agent-0
Num
Action
Control Min Control Max
Name XML nile
Joint
Unit
Torque applied on the rotor
between the torso and front left hip Torque applied
hip_ leftt__eftt_eeggg
hinge
torque m)
the between the front left two links
-1
angle_ Iffoot_left_le
hinge torque (N m)
Torque applicd on the rotor
2
hetween torso
right hip
-1
(right back
hinge torque m)
Torque applied the
3
between the back right two links
-1
angle_4 (right_ back_
hinge tonque (N m)
Table 21: The specific action space of 4-ant: agent-1
Num
Action
Control Min Control Max
Name (in XML file)
Joint
Unit
Torque applied on the rotor
between the
front right hip
-1
hip (front_rrihhh
hinge
torque (N m)
Torque applied on the sotor
between the front right two links
-1
angle_2 (front_right_leg)
hinge
torque m)
Torgue applied
the rotor
2 between the torso and back left hip
-1
hip_3 (back_leg)
hinge torque (N m)
Torque applied on the rotor
3
between the back two
-1
angle_3 3 (back_lep
hinge
torque m)
4-ant. The Ant is partitioned into parts, with each part corresponding leg of the ant. The action space of agent-O, agent-1, agent-2, 22, and agent-3 3 shown in Table 22 Table 23 Table 24 and Table 25
20

---

Table 22: The specific action space of 4-ant: agent-0
Num
Action
Control Min
Control Max
Name (in XML nlle
Joint
Unit
Torque applied the
0
between the torso and front left hip
-1
hip_1 (front left Jeft leg)
hinge
torque (N m)
Torque applied on the rotor
between front left two links
angle_ (frontt lert__leee__e_________ leg)
hinge
torque m)
Table 23: The specific action space of 2-ant-diag: agent-1
Num
Action
Control Min
Control Max
Name XMe)lle)
Joint
Unit
Torgue app lied the
hip_2 (front_right_kes
hinge
torque (N m)
between the torso and front right hip
Torque applied on the rotor
between the front right links
-1
angle_ (front_right_lee)
hinge
tonque (N m)
Table 24: The specific action space of 4-ant: agent-2
Num
Action
Control Min
Control Max
Name (in XML lile)
Joint
Unit
Torque applied the rotor
hip_3 (back_leg)
hinge
torque m)
between the torso and back left hip
Torque upphed on the rotor
between back two links
angle_ (back_
hinge
lorque (N m)
Table 25: The specific action space of 4-ant: agent-3
Num
Action
Control Min Control
Name XML hile)
Joint
Unit
Torque applied the
between the torso and back right hip
-1
hip_4 (right (right_ back_
hinge
torque m)
Torque applied on the rotor between the back right links
-1
ungle_4 4 (right_ buck_le
hinge
torque (N m)
In addition the robots mentioned in this paper, also provide other multi-agent versions of robots. Due to space constraints, we did not elaborate on them extensively in the paper. However, you can http://wwww..ff.feee// seafett-gymmasium gywggsiunna.mm...mmmmm.. com/ for more detailed information.
refer htttss://www.....
B.3 Task Representation
(a) Velocity
(b) Run
(b) Circle
(d) Goal
(e) Button
(f) Push
Figure 14: Tasks Gymnasium based Environments.
As shown in Figure 14
the Gymnasium-based learning environments support the following tasks:
Velocity: the robot aims to facilitate coordinated leg movement of the robot in the forward (right) direction by exerting torques on the hinges.
Run: the robot starts with a random initial direction and specific initial speed as it embarks on journey reach the opposite side of the map.
Circle: the reward is maximized by moving along the green circle, and the agent is not allowed to enter the outside of the red region, its optimal constrained path follows the line segments AD and
T --
BC. The reward function: R(s) = T+|| the cost function is C(s) = 1 [|<]| > Diim), where
I, are the coordinates in the plane, the velocity, and d, Zlim environmental parameters.
21

---

Goal: the robot navigates to multiple goal positions. After successfully reaching goal, its location is randomly reset while maintaining the overall layout. Achieving goal position, indicated by entering the goal circle, yields sparse reward. Additionally, dense reward encourages the robot's progress by rewarding proximity to the goal.
the objective is to move box to series of goal positions. Like the goal task, a new random goal location is generated after each successful achievement. The sparse reward is earned when the yellow box enters the designated goal circle. The dense reward consists of two components: one for moving the agent closer the box and another for bringing the box closer to the final goal.
Push:
the objective to activate a series goal buttons distributed throughout the environment. The agent's goal is to navigate towards and make contact with the currently highlighted button, known as the goal button. Once the correct button is pressed, a new goal button is selected and highlighted while preserving the rest of the environment. The sparse reward is earned upon successfully pressing the current goal button, while the dense reward component provides a bonus for progressing toward the highlighted goal button.
Button:
B.4 Constraint Specification
(4) Velocity Constraints
(b) Pillare
Hazards
(d) Sigwalls
(e) Veses
(1) Gremiins
Figure 15: Constraints of Gymnasium based Environments.
In these tasks, agents, such as Ant, HalfCheetah, and Humanoid, are trained to move faster for higher rewards, while also being imposed a velocity constraint for safety considerations. Formally, for an 02 + og: for an agent moving two-dimensional plane, the velocity is calculated v(ss a) V
Telocity-Constrain consists of a series safety tasks based on MuJoCo agents 23
agent moving along straight line, the velocity is caleulated v(s, a) |uz], where Mzy Bl the velocities of the agent in the x and y directions respectively. Then, cost(s, a) = [b(ss (s, a) > Uismit), Here, [P] denotes notation where the value is if the proposition P true, and 0 otherwise.
Pillars are employed to represent large cylindrical obstacles within the environment. In the general setting, contact with a pillar incurs costs.
Hazards utilized model areas within the environment that pose a risk, resulting in costs when an agent enters such areas.
Sigwalls designed specifically for Circle tasks. They serve visual representations of two four solid walls, which limit the circular area to smaller region. Crossing the wall from inside the safe area to the outside incurs costs.
Vases are specifically designed for Goal tasks. They represent static and fragile objects within the environment. Touching displacing these objects incurs costs for the agent.
Gremlins specifically employed in the Button tasks. They represent moving objects within the environment that can interact with the agent.
B.5 Vision-only Tasks
In recent years, vision-only SafeRL has gained significant attention as a focal point of research. primarily due to its applicability in real-world contexts [40 41]. While the initial iteration of Safety Gym offered rudimentary visual input support, there is room enhancing the realism and complexity of its environment. To effectively evaluate vision-based safe reinforcement learning algorithms, we have devised some more realistic visual tasks utilizing MuJoCo. This enhanced environment facilitates the incorporation of both RGB and RGB-d inputs. More details can be
22

---

referred to our online documentation: https: www.www.......o............ww www.saffty.. gymnaslum com/ en
ham/ennlse latest/tettttt env
ironments/safe fiiieeeiiiio... hmmmmlmll
(a) BuitonngButtonO
(b) BuildingButton!
BuildingButton2
Figure 16: Overview of BuildingButton tasks.
The Level 0 of BuildingButton requires the agent to operate multiple machines within construction site.
The Level 1 of BuildingButton requires the agent to proficiently and accurately operate multiple machines within a construction site, while concurrently evading other robots and obstacles present in the area.
The Level 2 of BuildingButton requires the agent to proficiently and accurately operate multiple machines within a construction site, while concurrently evading a heightened number of other robots and obstacles in the area.
(a) BoildingGoal0
(b) BuildingGoall
(c) BuildingGoal2
Figure 17: Overview of BuildingGooa tasks.
The Level 0 of BuildingGoal requires the agent to dock at designated positions within construction site.
The Level 1 of BuildingGoal requires the agent to dock at designated positions within a construction site while ensuring to avoid entry into hazardous areas.
The Level of BuildingGoal requires the agent to dock designated positions within a construction site, while ensuring avoid entry into hazardous areas and circumventing the site's 's exhaust fans.
23

---

(a) BuildingPushO
BuildingPushl
BuildingPush2.
Figure 18: Overview of BuildingPush tasks.
The Level BuildingPush requires the agent relocate the box to designated locations within construction site.
The Level of BuildingPush requires the agent relocate the box designated locations within construction site while avoiding areas demarcated as restricted.
The Level of BuildingPush requires the agent relocate the box to designated locations within construction while avoiding numerous hazardous fuel drums and areas demarcated as restricted.
(a) RaceO
(b) Racel
Race2
Figure 19: Overview of Race tasks.
The Level of Race requires the agent to reach the goal position.
The Level of Race requires the agent to reach the goal position while ensuring it avoids straying into the grass and prevents collisions with roadside objects.
The Level Race requires the agent to reach the goal position from distant starting point while ensuring it avoids straying into the grass and prevents collisions with roadside objects.

---

333
(a) FormulaOne0
(b) FormulaOnel
(c) FormulaOne2
Figure 20: Overview of FormulaOne tasks.
The Level FormulaOne requires the agent to maximize its reach to the goal position. For each episode, the agent is randomly initialized at one of the seven checkpoints.
The Level 1 of FormulaOne requires the agent maximize its reach to the goal position while circumventing barriers and racetrack fences. For each episode, the agent is randomly initialized at one of the seven checkpoints.
The Level 2 of FormulaOne requires the agent to maximize its reach to the goal position while circumventing barriers and racetrack fences. For each episode, the agent is randomly initialized at one of the seven checkpoints. Notably, the barriers surrounding the checkpoints are denser.
B.6 Some Issues about Safety Gym
a Safety-Gymmasium
(b) Safety- -ym
Figure 21: The difference between Safety-Gymnasin um and Safety Gym.
25

---

The bug of Natural Lidar. As shown in Figure 21 the original Natural Lidar in Safe-Gymt Fite-Gymm' has problem of not being able detect low-lying objects, which may affect comprehensive environmental observations.
The problem of observation space. In Safety Gym, by default, the observation space is presented as a one-dimensional array. The implementation leads to all ranges in observation space to be 1-00++00 as shown the following code:
if self Observation_flatteee
. obs oba_ffat_size self babs_sppce_ddte dict values ()])
sum ( ([np.prod(( (i shape) for in
2
self alas.bbborvation.spaee seeaff.obs_flatttiee
gym. Box (-np. inf, np.inf,
dtype-np. float32)
While this representation does not lead to behavioral errors in the environment, can be somewhat misleading for users. To address this issue, we have implemented the Gymnasium's 's flatten mechanism in the Safety Gym to handle the representation the observation space. This mechanism reorganizes the observation space into more intuitive and easily understandable format, enabling users to process and analyze the observation data more effectively.
self. obs_ info...foooo obs_space_dic dict gymnasium. spaces Dict (obs_space_dice
if self observation,flaatenn self observation_ gymnasium. www.aassssssaccs........ utils. flacten ols_.obs_infee sef..oa..ifo.oossspeeed.d.
else:
self wtw..obbervatton.spccc
caff.obs_infe obs sps_.ooe_dspcccii.c
assert self obs info obs_ space_dict contains(
obs
), B'B obs {obs} {self obs info. obs discee
self observation
obs
14
gymnasium spacess utils flatten (self. obs_ info. sbs_space_dict, obs)
return obs
Missing cost information. Safety Gym, by default, there are only two possible outputs for the cost: and 1, representing whether cost is incurred or not.
Optionally remove shaping from reward functions.
self constrain_ inteeiitnidicatoor.
for k in list (cost. keys ()); cost [k] float (cost [k] > 0.0) Indicator function
We believe that this representation method loses some information. For example, when the robot collides with a vase and causes the vase to move at different velocities, there should be different cost values associated with it indicate subtle differences in violating constraint behaviors. Additionally, these costs incurred by the actions are accumulated into the total cost. In typical cases, algorithms the total cost update the policy if the total cost generated by different obstacles is limited to only two states 0 and 1, the learning potential for multiple constraints is lost when multiple costs are triggered simultaneously.
Neglected dependency maintenance leads to conflicts.
The numpy =1.17.4 will cause the following problems:
ValueError: numpy ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 80 from PyObject
AttributeError: module 'numpy' has no attribute 'comple''.
httpg://gttabb..cooooee////////
26

---

C Details of Isaac Gym-based Learning Environments
C.1 Supported Agents
Safery- Seeeyy-DeeteroudHHande is based on Bi-DexHands (refer [42 for more details). Bi-DexHands aims to establish a comprehensive learning framework for two Shadow Hands, enabling them to possess a wide range of skills similar those humans. The Shadow Hand's joint limitations are follows (refer to Table 26). The thumb exhibits 5 degrees of freedom with 5 joints, while the other fingers have 3 degrees of freedom and joints each. The joints located the fingertips are not controllable. Similar to human fingers, the distal joints of the fingers are interconnected, ensuring that the angle of the middle joint always greater than or equal that of the distal joint. This design allows the middle phalange to be curved while the distal phalange remains straight. Additionally, an extra joint (LF5) located at the end of the little finger, enabling to rotate the same direction as the thumb. The wrist comprises two joints, facilitating a complete 360-degree rotation of the entire hand.
Table 26: Finger range of motion.
Joints
Corresponds to the number of Figure 22
Min
Max
Finger Distal (FFI.MFI (FFF.,,,FFLFFFD
15. 11, 7.
0
90
Finger Middle (FF2,MF2,RF2,1F))
16, 12, 8,
0
90"
Finger Base Abduction
17. 13, 9,
-15
90
Finger Base Lateral (FF.,M44RRR (F4,LF44
18, 14, 10,
-20
20
Little Finger Rotation (L.F)
19
0"
45
Thumb Distal (THI)
20
-15
90
Thumb Middle (TH2)
21
-30"
30
Thumb Base Abduction (TH3)
22
-12
12"
Thumb Base Lateral (TH4)
23
0"
70
Thumb Base Rotation (TH5)
24
-60"
60
Hand Wrist Abduction (WR1)
-40
28
Hand Wrist Lateral (WR2)
2
-28
8"
Stiffness, damping, friction, and armature are also important physical parameters in robotics. For each Shadow Hand joint, we show our DoF properties in Table 27 This part be adjusted in the Isaac Gym simulator.
Table 27: DoF properties of Shadow Hand.
Joints
Stiffness
Damping
Friction
Armature
WRI
100
4.78
0
0
WR2
100
2.17
0
0
FF2
100
3.4e+38
0
0
FF3
100
0.9
0
0
FF4
100
0.725
0
0
MF2
100
3.4e+38
0
0
MF3
100
0.9
0
MF4
100
0.725
0
RF2
100
3.4e+38
RF3
100
0.9
RF4
100
0.725
0
0
LF2
100
3.4c+38
0
LF3
100
0.9
LF4
100
0.725
0
TH2
100
3,4c+38
0
0
TH3
100
0.99
TH4
100
0.99
0
0
TH5
100
0.81
(
27

---

Figure 22: Degree-of-Freedon (DOF) configuration of the Shadow Hand similar to the skeleton of human hand.
Table 28: Observation space dual Shadow Hands.
Index
Description
0- 23
right Shadow Hand dof position
24 47
right Shadow Hand dof velocity
48-71 71
right Shadow Hand dof force
72 136
right Shadow Hand fingertip pose. linear velocity, angle velocity (5 x 13)
137 166
right Shadow Hand fingertip force, torque (5 6)
167 169
right Shadow Hand base position
170 172
right Shadow Hand base rotation
173 198
right Shadow Hand actions
199 222
left Shadow Hand dof position
223 246
left Shadow Hand dof velocity
247 270
left Shadow Hand dof force
271 335
left Shadow Hand fingertip pose, linear velocity, angle velocity (5 13)
336 365
left Shadow Hand fingertip force, torque (5 6)
366 368
left Shadow Hand base position
369 371
left Shadow Hand base rotation
372 397
left Shadow Hand actions
C.2 Task Representation
Hand Over
This scenario encompasses specific environment comprising two Shadow Hands positioned opposite each other, with their palms facing upwards. The objective is to pass an object between these hands. Initially, the object will randomly descend within the area of the Shadow Hand on the right side. The hand on the right side then grasps the object and transfers it to the other hand. It is important to note that the base of each hand remains fixed throughout the process. Furthermore, the hand initially holding the object cannot directly make contact with the target hand or roll the object towards it. Hence, the object must be thrown into the air, maintaining its trajectory until reaches the target hand.
28

---

In this task, there are 398-dimensional observations and 30-dimensional actions. The reward function closely tied to the positional discrepancy between the object and the target. As the pose error diminishes, the reward increases significantly. The detailed observation space for each agent can be found in Table 29 while the corresponding action space outlined in Table 30
Observations The observational space for the Hand Over task consists of 398 dimensions, as indicated in Table 29) However, important highlight that in this particular task, the base of the dual hands remains fixed. Therefore, the observation for the dual hands is compared to reduced 24- dimensional space, as described in Table 28
Table 29: Observation space of Hand Over.
Index
Description
373
dual hands observation shown in Table
374 380
object pose
381 - 383
object linear velocity
384 386
object angle velocity
387 393
goal pose goal rot - object rot
394 397
Actions The action space for a single hand in the Hand Over task comprises 40 dimensions, as illustrated in Table 30
Table 30: Action space of Hand Over.
Index
Description
0- 19
right Shadow Hand actuated joint
39
left Shadow Hand actuated joint
Rewards Let the positions of the object and the goal be denoted To and Vy respectively. The translational position difference between the object and the goal, represented as de, can be define the angular position difference between
computed as de || Z-ll2- Similarly, the object and the goal as da. The rotational difference, denoted as dr, is then calculated as d, arcsin (clamp(Idaalaa max- 1.0)).
The rewards for the Hand Over task are determined using the following formula: exp( 00.2(ad, + d,))
(2)
Here, a represents a constant that balances the rewards between translational and rotational aspects.
Hand Over Catch
This environment is made up of a half Hand Over, and Catch Underarm 42 the object needs to be thrown from the vertical hand to the palm-up hand.
Observations The observational space for this combined task encompasses 422 dimensions, as illustrated in Table 31
Table 31: Observation space of Hand Over Catch.
Index
Description
0- 397
dual hands observation shown in Table
398 404
object pose
405 407
object linear velocity
408 410
object angle velocity
411 417
goal pose
418 421
goal rot - object rot
Actions The action space, consisting of dimensions, illustrated comprchensive representation of the available actions.
Table 32 providing
29

---

Table 32: Action space of Hand Over Catch.
Index
Description
0- 19
right Shadow Hand actuated joint
20 22
right Shadow Hand base translation
23 3 25
right Shadow Hand base rotation
26-45
left Shadow Hand actuated joint
46 48
left Shadow Hand base translation
49 51
left Shadow Hand base rotation
Rewards Let's denote the positions of the object and the goal as &o and zgg. respectively. The translational position difference between the object and the goal denoted as d,, can be caleu- lated as de ||z Ezll2- Additionally, we define the angular position difference between the object and the goal da. The rotational difference, denoted d,, is given by the formula d, = 2 arcsin (alainnccmm((dldl.a 2,, max = 1.0)). Finally, the rewards are determined using the specific formula:
r = expl 0.2(ad + d,)]
Here, a represents a constant that balances the translational and rotational rewards.
C.3 Constraint Specification
(a) Hand Catch Over
(b): Hand Over
(c): Dynamics
(d): Safety Joint
(e): Safety Finger
Figure 23: Tasks of Selety-DexttrrouHHand..
Safety Joint constrains the freedom of joint @ of the forefinger (please refer to Figure 23 (c) and (d)). Without the constraint, joint @ has freedom of [-20%, 20"). The safety tasks restrict joint @ within [-10%, 10%)
Let ang_4 4 be the angle of joint @, and the cost is defined as:
C = [[(ang 4 & [-10%, 10%))
Safety Finger constrains the freedom of joints @, @ and @ of forefinger (please refer to Figure 23 (c) and (c)). Without the constraint, joints @ and @ have freedom of [0,, 900 and joint @ of -20 20 The safety tasks restrict joints @, @, and @ within [22.5%, 67.5 67.5%), [22.5 67.5%). and 1- 10 respectively. Let ang 2, ang 3, ang be the angles of joints @, @, 4 and the cost is defined as:
C I[ang [22.5%, 22.5%.67.59 67.5%), or ang 3 & [22.5%, 67.5%), ang & [-100, 109)).
(5)
30


Reviews:
[{"review_id": "vFMCYg3kcE", "reviewer": "Reviewer_2t22", "summary": "", "questions": "", "limitations": "The key contribution of this paper should be clearly stated. \nIt would be great if authors include more information in documentation explaining how someone can reproduce results in the paper with the scripts provided.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "Availability of dataset environment through Gym and MuJoCo interface and easy-installation.\nComprehensive validation and comparison of 16 state-of-the-art reinforcement learning algorithms. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "rBZUPz8nxz", "reviewer": "Reviewer_w71G", "summary": "", "questions": "", "limitations": "There is no real discussion of limitations except for a few technical ones mentioned in the appendix. While it is useful to note some of the issues with dependencies, the technical details of observations etc., it would have been more informative to discuss the limitations of the tasks (i.e., do these sufficiently represent realistic tasks? Are they complex enough to push the limits of current Safe RL algorithms), as well as sensor models (sensor noise, fidelity etc.). It would also be good to discuss how easy it is to implement new algorithms  within the SafepO library. ", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Given the high importance of safe reinforcement learning and the development of new algorithms, a safe RL focused environment and algorithm library is a good contribution for rapid experimentation and benchmarking. The proposed platform contains state and vision observations (proposing both RGB and RGBD sensors  for example) and a higher variety of environments/constraints that can be programmed. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "HznKdBl0pQ", "reviewer": "Reviewer_vCTw", "summary": "", "questions": "", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "(1) The benchmark contains a diverse and rich set of safety-critical tasks, constraints, and robot models that require solving challenging SafeRL problems. This diversity enables rigorous evaluation.\n\n(2) The procedural generation and Gymnasium framework allow for easy customization and expansion to new tasks, making this a flexible long-term platform.\n\n(3) Seamless installation, comprehensive documentation, and simple usage examples make Safety-Gymnasium welcoming to new users and researchers.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "VviRceGMkh", "reviewer": "Reviewer_7227", "summary": "", "questions": "", "limitations": "No. I do not think the authors discuss the limitations of the work. I would suggest the authors to add a detailed discussion about the limitations and the ways to mitigate it.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The suite of environments are diverse\n2. The implementations of algorithms are clean and correct\n3. The experiment results are extensive", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "2ERsgnCLuC", "reviewer": "Reviewer_EjZD", "summary": "", "questions": "", "limitations": "The paper primarily focuses on benchmarking SafeRL algorithms using the proposed environment suite. However, the transferability of the results to complex real-world safety-critical applications may be limited. Future work should explore the adaptability and performance of SafePO in more challenging and diverse real-world scenarios.\nThe paper could benefit from a more extensive evaluation of the performance metrics used to assess the algorithms' safety and efficiency. Incorporating additional evaluation criteria, such as worst-case scenario analysis or robustness to adversarial perturbations, would provide a more comprehensive understanding of the SafeRL algorithms' capabilities.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The availability of Safety-Gymnasium and SafePO library, as benchmark tools and validation resources, respectively, promotes collaborative research and empowers researchers to work collectively towards safer AI development.\nThe broad scope of tasks covered in Safety-Gymnasium and its acceptance of different input types make it suitable for diverse applications, enhancing its relevance to various research domains.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This submission proposes a safe RL benchmark called Safety Gymnasium. A range of safety-critical autonomy tasks in single and multi-agent scenarios are encompassed. A library of algorithms called SafePO with 16 SOTA safe RL algorithms is also provided as the baselines. The proposed benchmark can facilitate the evaluation and comparison of safe RL algorithms. Detailed documentation and Github repo are provided. The Github repo already has a couple of hundred stars, demonstrating the usage of the proposed benchmark to the community. AC has checked all of them. 

After the discussion between reviewers and the authors, the submission received scores of 8, 7, 6, 6, 5, so overall very positive evaluation. AC did some homework in the discussion session to examine the library and raised an issue about duo submission. After clarification from the authors, the issue is resolved. But authors are still suggested to be mindful of the submission policy and what can be put on a personal webpage, to avoid confusion. 

Overall, AC appreciates the contribution of the library, the detailed documentation, and the timely responses of the authors in the discussions. AC believes this is a solid contribution to the community. Thus, an acceptance is recommended.
Decision: Accept (Poster)