SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiaming Ji $^{1,*}$ , Borong Zhang $^{1,*}$ , Jiayi Zhou $^{1,*}$ , Xuehai Pan $^{1}$ , Weidong Huang $^{1}$ , Ruiyang Sun $^{1}$ , Yiran Geng $^{1}$ , Yifan Zhong $^{1,2}$ , Juntao Dai $^{1}$ , Yaodong Yang $^{1,\dagger}$
$^{1}$  Institute for AI, Peking University  $^{2}$  Beijing Institute for General Artificial Intelligence (BIGAI) {jiamg.ji, borongzh}@gmail.com, gaiejj@outlook.com yaodong.yang@pku.edu.cn
Artificial intelligence (AI) systems possess significant potential to drive societal progress. However, their deployment often faces obstacles due to substantial safety concerns. Safe reinforcement learning (SafeRL) emerges as a solution to optimize policies while simultaneously adhering to multiple constraints, thereby addressing the challenge of integrating reinforcement learning in safety-critical scenarios. In this paper, we present an environment suite called Safety-Gymnasium, which encompasses safety-critical tasks in both single and multi-agent scenarios, accepting vector and vision-only input. Additionally, we offer a library of algorithms named Safe Policy Optimization (SafeP0), comprising 16 state-of-the-art SafeRL algorithms. This comprehensive library can serve as a validation tool for the research community. By introducing this benchmark, we aim to facilitate the evaluation and comparison of safety performance, thus fostering the development of reinforcement learning for safer, more reliable, and responsible real-world applications. The website of this project can be accessed at https://sites.google.com/view/safety-gymnasium.
AI systems possess enormous potential to spur societal progress. However, their deployment is frequently hindered by substantial safety considerations [1; 2; 3; 4]. Distinct from pure reinforcement learning (RL), Safe reinforcement learning (SafeRL) seeks to optimize policies while concurrently adhering to multiple constraints, addressing the challenge of employing RL in scenarios with critical safety implications [5; 6; 7; 8; 9]. This strategy proves particularly pertinent in real-world applications such as autonomous vehicles [10] and healthcare [11], where system failures or unsafe actions can result in grave consequences, such as accidents or harm to individuals. In large language models (LLMs), some studies have also shown that the toxicity of the models can be reduced through SafeRL [12; 13]. Incorporating safety constraints ensures adherence to predefined boundaries and regulatory standards, fostering trust and enabling exploration in environments with high-risk potential. Overall, SafeRL is instrumental in guaranteeing the dependable operation of intelligent systems in intricate and high-stake domains.
*Equal Contribution. †Corresponding author.  
Work done when Jiayi Zhou visited Peking University.
Simulation environments have become instrumental in fostering the advancement of RL. Eminent examples such as Gym [14], Atari [15], and dm-control [16] underline their importance. These versatile platforms permit researchers to swiftly design and execute varied tasks, thus enabling efficient evaluation of algorithmic effectiveness and intrinsic limitations. However, within the sphere of SafeRL, there is a notable dearth of dedicated simulation environments, which impedes comprehensive exploration of SafeRL. In recent years, there have been strides to address this gap. DeepMind presented AI-Safety-Gridworlds, a suite of RL environments showcasing various safety properties of intelligent agents [17]. Afterward, OpenAI introduced the Safety Gym benchmark suite, a collection of high-dimensional continuous control environments incorporating safety-robot tasks [18]. Over the past two years, several additional environments have been developed by researchers, including safe-control-gym [19], MetaDrive [20], etc.
Compared to Safety Gym<sup>1</sup> Safety-Gymnasium inherits and expands the settings of some tasks of Safety Gym, aiming to bolster the community's growth further. Compared with Safety Gym, we have made the following major improvements:
- Refactoring of the physics engine. Safety Gym utilizes mujoco-py to enable Python-based customization of MuJoCo components. However, mujoco-py stopped updates and support after 2021. In contrast, Safety-Gymnasium supports MuJoCo directly, eliminating the reliance on mujoco-py. This facilitates access to the latest MuJoCo features (e.g., rendering speed and accuracy improved, etc.) and lowers the entry barrier, particularly due to mujoco-py's dependency on specific GCC versions and more.
- Extension of Agent and Task Components. Safety Gym initially supports only three agents and tasks. On this basis, Safety-Gymnasium has been further expanded, introducing more diverse agents and task components and expanding safety tasks to cover multi-agent domains. Finally, Safety-Gymnasium launched a high-dimensional test component based on Issac-Gym [21], further enriching the benchmark.
- Enhanced Visual Task Support. The visual components of Safety Gym are simplistic (consisting of basic geometric shapes), and mujoco-py relies on OpenGL for visual rendering, which results in significant virtualization performance loss on headless servers. In contrast, SafetyGymnasium, built on MuJoCo, achieves rendering speeds on CPU that are twice as fast as the former. Additionally, it offers more comprehensive visual component support.
- Easy Installation and High Customization. Safety Gym is cumbersome to install and relies heavily on the underlying software. One of the design motivations of Safety-Gymnasium is the ease of use so that everyone can focus on algorithm design. Safety-Gymnasium can be easily installed with one simple command pip install safety-gymnasium. While benefiting from the highly integrated framework, Safety-Gymnasium only needs 100 lines of code to customize the required environment.
In this work, we introduce Safety-Gymnasium, a collection of environments specifically for SafeRL, built upon the Gymnasium [14; 22] and MuJoCo [23]. Enhancing the extant Safety Gym framework [18], we address various concerns and expand the task scope to include vision-only and multi-agent scenarios. Additionally, we released SafeP0, a single-file style algorithm library containing over 16 state-of-the-art algorithms. Collectively, our contributions are enumerated as follows:
- Environmental Components. We provide various safety-oriented tasks under the umbrella of Safety-Gymnasium. These tasks encompass single-agent, multi-agent, and vision-based challenges, each with varying constraints. Our environments are categorized into two primary types: Gymnasium-based, featuring agents of escalating complexity for algorithm verification and comparison, and Issac-Gym-based, incorporating sophisticated agents that harness the parallel processing power of Issac-gym's GPU. This empowers researchers to explore SafeRL algorithms in complex scenarios. Further details can be found in Section 4.
- Algorithm Components. We offer the Safe ePO algorithm library, which comprises a single-file style housing 16 diverse algorithms. These algorithms encompass both single-agent and multi-agent approaches, along with first-order and second-order variants, as well as Lagrangian-based
<sup>1</sup>Again, we have no intention of attacking Safety Gym; the contribution of Safety Gym to the SafeRL community cannot be ignored, and Safety Gym also inspired this work. We hope that through our efforts, Safety-Gymnasium can further promote the development of SafeRL and give back to the entire RL community.
and Projection-based methods. Through meticulous decoupling, each algorithm's code resides in an individual file. A more in-depth exploration of SafeP0 is presented in Section 5.
- Insights and Analysis. Combining Safety-Gymnasium and SafePO, we conduct a detailed analysis of existing algorithms. Our analysis encompasses 16 algorithms across 54 distinct environments, covering various scenarios such as single-agent and multi-agent setups with varying constraint complexities. This analysis delves into each algorithm's strengths, constraints, and avenues for enhancement. We provide access to all metadata, fostering community verification and encouraging further research. Further details can be found in Section 6.
Safety Environments In RL, agents need to explore environments to learn optimal policies by trial and error. It is currently typical to train RL agents mostly or entirely in simulation, where safety concerns are minimal. However, we anticipate that challenges in simulating the complexities of the real world (e.g., human-AI collaborative control [1; 2]) will cause a shift towards training RL agents directly in the real world, where safety concerns are paramount [20; 24; 25]. OpenAI includes safety requirements in the Safety Gym [18], which is a suite of high-dimensional continuous control environments for measuring research progress on SafeRL. Safe-control-gym [19] allows for constraint specification and disturbance injection onto a robot's inputs, states, and inertial properties through a portable configuration system. DeepMind also presents a suite of RL environments, AI-Safety-Gridworlds [17], illustrating various safety properties of intelligent agents.
SafeRL Algorithms CMDPs have been extensively studied for different constraint criteria [26; 27; 28; 29]. With the rise of deep learning, CMDPs are also moving to more high-dimensional continuous control problems. CPO [30] proposes the first general-purpose policy search algorithm for SafeRL with guarantees for near-constraint satisfaction at each iteration. However, CPO's policy updates hinge on Taylor approximations and the inversion of high-dimensional Fisher information matrices. These approximations can occasionally lead to inappropriate policy updates. FOCOPS [31] applies a primal-dual approach to solve the constrained trust region problem directly and subsequently projects the solution back into the parametric policy space. Similarly, CUP [32] offers non-convex implementations through a first-order optimizer, thereby not requiring a strong approximation of the convexity of the objective.
3.1 Constrained Markov decision process
SafeRL [6; 33] is often formulated as a Constrained Markov decision process (CMDP) [6], which is a tuple  $\mathcal{M} = (\mathcal{S},\mathcal{A},\mathbb{P},R,\mathcal{C},\mu ,\gamma)$ . Here  $\mathcal{S}$  and  $\mathcal{A}$  are the state space and action space correspondingly.  $\mathbb{P}(s^{\prime}|s,a)$  is the probability of state transition from  $s$  to  $s^{\prime}$  after taking action  $a$ .  $R(s^{\prime}|s,a)$  denotes the reward obtained by the agent performing action  $a$  in state  $s$  and transitioning to state  $s^{\prime}$ . The set  $\mathcal{C} = \{(c_i,b_i)\}_{i = 1}^m$ , where  $c_{i}$  are cost functions:  $c_{i}:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$  and the cost thresholds are  $b_{i},i = 1,\dots ,m.$ $\mu (\cdot):S\rightarrow [0,1]$  is the initial state distribution and the discount factor  $\gamma \in [0,1)$ .
A stationary parameterized policy  $\pi_{\theta}$  is a probability distribution defined on  $S\times \mathcal{A}$ ,  $\pi_{\theta}(a|s)$  denotes the probability of taking action  $a$  in state  $s$ . We use  $\Pi_{\theta} = \{\pi_{\theta}:\theta \in \mathbb{R}^{p}\}$  to denote the set of all stationary policies and  $\theta$  is the network parameter needed to be learned. Let  $P_{\pi_{\theta}}\in \mathbb{R}^{|S|\times |S|}$  denotes a state transition probability matrix and the components are:  $P_{\pi_{\theta}}[s,s^{\prime}] = \mathbb{P}_{\pi_{\theta}}(s^{\prime}|s) = \sum_{a\in \mathcal{A}}\pi_{\theta}(a|s)\mathbb{P}(s^{\prime}|s,a)$ , which denotes one-step state transition probability from  $s$  to  $s^{\prime}$  by executing  $\pi_{\theta}$ . Finally, we let  $d_{\pi_{\theta}}^{s_0}(s) = (1 - \gamma)\sum_{t = 0}^{\infty}\gamma^t\mathbb{P}_{\pi_{\theta}}(s_t = s|s_0)$  to be the stationary state distribution of the Markov chain starting at  $s_0$  induced by policy  $\pi_{\theta}$  and  $d_{\pi_{\theta}}^{\mu}(s) = \mathbb{E}_{s_0\sim \mu (\cdot)}[d_{\pi_{\theta}}^{\mu}(s)]$  to be the discounted state visitation distribution on initial distribution  $\mu$ .
The objective function is defined via the infinite horizon discounted reward function where for a given  $\pi_{\theta}$ , we have  $J^{R}(\pi_{\theta}) = \mathbb{E}[\sum_{t=0}^{\infty} \gamma^{t} R(s_{t+1}|s_{t},a_{t})|s_{0} \sim \mu, a_{t} \sim \pi_{\theta}]$ . The cost function is similarly specified via the following infinite horizon discount cost function:  $J_{i}^{C}(\pi_{\theta}) = \mathbb{E}[\sum_{t=0}^{\infty} \gamma^{t} C_{i}(s_{t+1}|s_{t},a_{t})|s_{0} \sim \mu, a_{t} \sim \pi_{\theta}]$ .
Then, we define the feasible policy set  $\Pi_{\mathcal{C}}$  as:  $\Pi_{\mathcal{C}} = \cap_{i=1}^{m}\{\pi_{\theta} \in \Pi_{\theta} \text{ and } J_{i}^{C}(\pi_{\theta}) \leq b_{i}\}$ . The goal of CMDP is to search the optimal policy  $\pi_{\star}: \pi_{\star} = \arg \max_{\pi_{\theta} \in \Pi_{\mathcal{C}}} J^{R}(\pi_{\theta})$ .
3.2 Constrained Markov Game
Safe multi-agent reinforcement learning is often formulated as a Constrained Markov Game  $(\mathcal{N},\mathcal{S},\mathcal{A},\mathbb{P},\mu ,\gamma ,R,C,\boldsymbol {b})$ . Here,  $\mathcal{N} = \{1,\dots ,n\}$  is the set of agents,  $\mathcal{S}$  and  $\mathcal{A} = \prod_{i = 1}^{n}\mathcal{A}^i$  are the state space and the joint action space (i.e., the product of the agents' action spaces),  $\mathbb{P}:S\times A\times S\to \mathbb{R}$  is the probabilistic transition function,  $\mu$  is the initial state distribution,  $\gamma \in [0,1)$  is the discount factor,  $R:S\times A\rightarrow \mathbb{R}$  is the joint reward function,  $C = \left\{C_j^i\right\}_{1\leq j\leq m^i}$  is the set of sets of cost functions (every agent  $i$  has  $m^i$  cost functions) of the form  $C_j^i:\mathcal{S}\times \mathcal{A}^i\to \mathbb{R}$ , and finally the set of corresponding cost threshold is given by  $\pmb {b} = \left\{b_j^i\right\}_{1\leq j\leq m^i}$ . At time step  $t$ , the agents are in a state  $\mathrm{s}_t$ , and every agent  $i$  takes an action  $\mathrm{a}_t^i$  according to its policy  $\pi^i (\mathbf{a}^i\mid \mathbf{s}_t)$ . Together with other agents' actions, it gives a joint action  $\mathbf{a}_t = (\mathbf{a}_t^1,\ldots ,\mathbf{a}_t^n)$  and the joint policy  $\pi (\mathbf{a}\mid \mathbf{s}) = \prod_{i = 1}^{n}\pi^{i}(\mathbf{a}^{i}\mid \mathbf{s})$ . The agents receive the reward  $R(\mathrm{s}_t,\mathbf{a}_t)$ , meanwhile each agent  $i$  pays the costs  $C_j^i (\mathrm{s}_t,\mathrm{a}_t^i)$ ,  $\forall j = 1,\dots ,m^i$ . The environment then transits to a new state  $\mathrm{s}_{t + 1}\sim \mathbb{P}(\cdot |\mathrm{s}_t,\mathbf{a}_t)$ .
The objective of reward function are  $J(\pi) \triangleq \mathbb{E}_{\mathrm{s}_0 \sim \rho^0, \mathbf{a}_{0:\infty} \sim \pi, \mathrm{s}_{1:\infty} \sim \mathrm{p}}[\sum_{t=0}^{\infty} \gamma^t R(\mathbf{s}_t, \mathbf{a}_t)]$ , and costs function are  $J_j^i(\pi) \triangleq \mathbb{E}_{\mathrm{s}_0 \sim \rho^0, \mathbf{a}_{0:\infty} \sim \pi, \mathrm{s}_{1:\infty} \sim \mathrm{p}}\left[\sum_{t=0}^{\infty} \gamma^t C_j^i(\mathbf{s}_t, \mathbf{a}_t^i)\right] \leq c_j^i$ ,  $\forall j = 1, \ldots, m^i$ .
We are examining a fully cooperative setting where all agents share a common reward function. Consequently, the goal of safe multi-agent RL is to identify the optimal policy that maximizes the expected total reward while simultaneously ensuring that the safety constraints of each agent are satisfied. Then we define the feasible joint policy set  $\pi_{\mathcal{C}} = \cap_{i=1}^{n}\{\pi_{\theta} \in \Pi_{\theta} \text{ and } J_{j}^{i}(\pi) \leq c_{j}^{i}, \forall j = 1, \ldots, m^{i}\}$ . The goal of CMG is to search the optimal policy  $\pi_{\star} = \arg \max_{\pi_{\theta} \in \Pi_{\mathcal{C}}} J(\pi_{\theta})$ .
4 Safety Environments: Safety-Gymnasium
Safety-Gymnasium provides a seamless installation process and minimalistic code snippets to basic examples, as shown in Figure 1. Due to the limited space of the paper, we provide a more detailed description (e.g., detailed instructions, the composition of the robot's observation space and action space, dynamic structure, physical parameters, etc.) in Appendix B and Online Documentation<sup>2</sup>.
Figure 1: Using Safety-Gymnasium to create, step, render a specific safety-task environment.
4.1 Gynasium-based Learning Environments
In this section, we introduce Gymnasium-based environment components from three aspects: (1) the robots (both single-agent and multi-agent); (2) the tasks that are supported within the environment; (3) the safety constraints that are upheld.
2Online Documentation: www.safety-gymnasium.com
Supported Robots As shown in Figure 2, Safety-Gymnasium inherits three pre-existing agents from Safety Gym [18], namely Point, Car, and Doggo. By meticulously adjusting the model parameters, we have successfully mitigated the issue of excessive oscillations during the runtime of Point and Car agents. Building upon this foundation, we have introduced two additional robots: racecar [34; 35], and ant [23], to enrich the single-agent scenarios. As for multi-agent robots, we have leveraged certain configurations from multi-agent MuJoCo [36], deconstructing the original single-agent structure and enabling multiple agents to control distinct body segments. This design choice has been widely adopted in various research works [37; 38; 39].
(a). Point
(a). Ant
Figure 2: Upper: The Single-Agent Robots of Gymnasium-based Environments. Lower: The Multi-Agent Robots of Gymnasium-based Environments.
(b). Car
(b). Half Cheetah
(c). Racecar
(c). Humanoid
(d). Doggo
(d). Hopper
(e). Ant
(e). Walker2D
Supported Tasks As shown in Figure 3, the Gymnasium-based learning environments support the following tasks. For a more detailed task specification, please refer to our online documentation<sup>3</sup>.
- Velocity. The robot aims to facilitate coordinated leg movement of the robot in the forward (right) direction by exerting torques on the hinges.
- Run. The robot starts with a random initial direction and a specific initial speed as it embarks on a journey to reach the opposite side of the map.
- Circle. The reward is maximized by moving along the green circle and not allowed to enter the outside of the red region, so its optimal path follows the line segments  $AD$  and  $BC$ .
- Goal. The robot navigates to multiple goal positions. After successfully reaching a goal, its location is randomly reset while maintaining the overall layout.
- Push. The objective is to move a box to a series of goal positions. Like the goal task, a new random goal location is generated after each achievement.
- Button. The objective is to activate a series of goal buttons distributed throughout the environment. The agent's goal is to navigate towards and contact the currently highlighted button, known as the goal button.
Supported Constraints As shown in Figure 3, the Gymnasium-based environments support the following constraints. For a more detailed task specification, please refer to our online documentation.
- Velocity-Constraint involves safety tasks using MuJoCo agents [23]. In these tasks, agents aim for higher reward by moving faster, but they must also adhere to velocity constraints for safety. Specifically, in a two-dimensional plane, the cost is computed as the Euclidean norm of the agent's velocities  $(v_{x}$  and  $v_{y})$ .
3Task Specification Documentation: https://www.safety-gymnasium.com/en/latest/components_of_environments/tasks.html
(e).Button
(f).Push
(a). Velocity
(a). Velocity Constraints
(b). Run
(b). Pillars
Figure 3: Upper: Tasks of Gymnasium-based Environments; Lower: Constraints of Gymnasium-based Environments.
(c).Circle
(c). Hazards
(d). Goal
(d). Sigwalls
(e). Vases
(f).Gremlins
- Pillars are employed to represent large cylindrical obstacles within the environment. In the general setting, contact with a pillar incurs costs.
- Hazards are utilized to model areas within the environment that pose a risk, resulting in costs when an agent enters such areas.
- Sigwalls are designed specifically for Circle tasks. Crossing the wall from inside the safe area to the outside incurs costs.
- Vases represent static and fragile objects within the environment. Touching or displacing these objects incurs costs for the agent.
- Gremlins represent moving objects within the environment that can interact with the agent.
4.1.1 Vision-only tasks
Vision-only SafeRL has gained significant attention as a focal point of research, primarily due to its applicability in real-world contexts [40; 41]. While the initial iteration of Safety Gym offered rudimentary visual input support, there is room for enhancing the realism of its environment. To effectively evaluate vision-based SafeRL algorithms, we have devised a more realistic visual environment utilizing MuJoCo. This enhanced environment facilitates the incorporation of both RGB and RGB-D inputs (as shown in Figure 5). An exemplar of this environment is depicted in Figure 4, while comprehensive descriptions are available in Appendix B.5.
(a) Race
Figure 4: Vision-only Tasks of Gymnasium-based Environments.
(b) The Vision Input of Race
(c) FormulaOne
(d) The Vision Input of FormulaOne
4.2 Issac-Gym-based Learning Environments
In this section, we introduce Safety-DexterousHands, a collection of environments built upon DexterousHands [42] and the Isaac Gym engine [21]. Leveraging GPU capabilities, Safety-DexterousHands enables large-scale parallel sample collection, significantly accelerating the training process. The environments support both single-agent and multi-agent settings. These environments involve two robotic hands (refer to Figure 6 (a) and (b)). In each episode, a ball randomly descends near the right hand. The right hand needs to grasp and launch the ball toward the left hand, which subsequently catches and deposits it at the target location.
Figure 5: The RGB and RGB-D input of Gymnasium-based Environments.
(a) Hand Catch Over
(b): Hand Over
(c): Dynamics
(d): Safety Joint
Figure 6: Tasks of Safety-DexterousHands.
(e): Safety Finger
For timestep  $t$ , let  $x_{b,t}, x_{g,t}$  to be the position of the ball and the goal,  $d_{p,t}$  to denote the positional distance between the ball and the goal  $d_{p,t} = \| x_{b,t} - x_{g,t}\|_2$ . Let  $d_{a,t}$  denote the angular distance between the object and the goal, and the rotational difference is  $d_{r,t} = 2\arcsin \min \{|d_{a,t}|, 1.0\}$ . The reward is defined as follows,  $r_t = \exp \{-0.2(\alpha d_{p,t} + d_{r,t})\}$ , where  $\alpha$  is a constant balance of positional and rotational reward.
Safety Joint constrains the freedom of joint  $④$  of the forefinger (refer to Figure 6 (c) and (d)). Without the constraint, joint  $④$  has freedom of  $[-20^{\circ}, 20^{\circ}]$ . The safety tasks restrict joint  $④$  within  $[-10^{\circ}, 10^{\circ}]$ . Let ang_4 be the angle of joint  $④$ , and the cost is defined as:  $c_{t} = \mathbb{I}(\mathrm{ang}_{-}4 \notin [-10^{\circ}, 10^{\circ}])$ .
Safety Finger constrains the freedom of joints ②, ③ and ④ of forefinger (refer to Figure 6 (c) and (e)). Without the constraint, joints ② and ③ have freedom of  $[0^{\circ}, 90^{\circ}]$  and joint ④ of  $[-20^{\circ}, 20^{\circ}]$ . The safety tasks restrict joints ②, ③, and ④ within  $[22.5^{\circ}, 67.5^{\circ}]$ ,  $[22.5^{\circ}, 67.5^{\circ}]$ , and  $[-10^{\circ}, 10^{\circ}]$  respectively. Let ang_2, ang_3, ang_4 be the angles of joints ②, ③, ④, and the cost is defined as:
$$
c _ {t} = \mathbb {I} (\text {a n g} _ {- 2} \notin [ 2 2. 5 ^ {\circ}, 6 7. 5 ^ {\circ} ], \text {o r a n g} _ {- 3} \notin [ 2 2. 5 ^ {\circ}, 6 7. 5 ^ {\circ} ], \text {o r a n g} _ {- 4} \notin [ - 1 0 ^ {\circ}, 1 0 ^ {\circ} ]). \tag {1}
$$
5 Safe Policy Optimization Algorithms: SafePO
This section provides a detailed discussion of the design of SafeP0. Features such as strong performance, extensibility, customization, visualization, and documentation are all presented to demonstrate the advantages and contributions of SafeP0.