SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 16: The specific action space of Ant
Num	Action	Control Min	Control Max	Name (in XML file)	Joint/Site	Unit
0	torque applied on the rotor between the torso and front left hip	-1	1	hip_1 (front_left leg)	hinge	torque (N m)
1	torque applied on the rotor between the front left two links	-1	1	angle_1 (front_left leg)	hinge	torque (N m)
2	torque applied on the rotor between the torso and front right hip	-1	1	hip_2 (front_right leg)	hinge	torque (N m)
3	torque applied on the rotor between the front right two links	-1	1	angle_2 (front_right leg)	hinge	torque (N m)
4	torque applied on the rotor between the torso and back left hip	-1	1	hip_3 (back leg)	hinge	torque (N m)
5	torque applied on the rotor between the back left two links	-1	1	angle_3 (back leg)	hinge	torque (N m)
6	torque applied on the rotor between the torso and back right hip	-1	1	hip_4 (right_back leg)	hinge	torque (N m)
7	torque applied on the rotor between the back right two links	-1	1	angle_4 (right_back leg)	hinge	torque (N m)
Ant. As depicted in Figure 12, the quadrupedal robot, inspired by the model proposed in [45]. It consists of a torso and four interconnected legs. Each leg is composed of two hinged connecting limbs, which, in turn, are connected to the torso via hinges. To achieve movement in the desired direction, coordination of the four legs is required by applying moments to the eight hinge drivers. For a comprehensive understanding of the robot, please refer to Table 15, Table 16, and Table 17, which provide an overview of the Ant robot, its specific action space, and observation space, respectively.
(a) Ant: front
(b) Ant: back
(c) Ant: left
Figure 12: A different view of the robot: Ant.
(d) Ant: right
Table 17: The specific observation space of Ant
Size	Observation	Min	Max	Name (in XML file)	Joint/Site	Unit
3	accelerometer	-inf	inf	accelerometer	site	acceleration (m/s^2)
3	velocimeter	-inf	inf	velocimeter	site	velocity (m/s)
3	gyro	-inf	inf	gyro	site	angular velocity (rad/s)
3	magnetometer	-inf	inf	magnetometer	site	magnetic flux (Wb)
1	angular velocity of angle between torso and front left link	-inf	inf	hip_1 (front_left leg)	hinge	angle (rad)
1	angular velocity of the angle between front left links	-inf	inf	ankle_1 (front_left leg)	hinge	angle (rad)
1	angular velocity of angle between torso and front right link	-inf	inf	hip_2 (front_right leg)	hinge	angle (rad)
1	angular velocity of the angle between front right links	-inf	inf	ankle_2 (front_right leg)	hinge	angle (rad)
1	angular velocity of angle between torso and back left link	-inf	inf	hip_3 (back leg)	hinge	angle (rad)
1	angular velocity of the angle between back left links	-inf	inf	ankle_3 (back leg)	hinge	angle (rad)
1	angular velocity of angle between torso and back right link	-inf	inf	hip_4 (right_back leg)	hinge	angle (rad)
1	angular velocity of the angle between back right links	-inf	inf	ankle_4 (right_back leg)	hinge	angle (rad)
1	z-coordinate of the torso (centre).	-inf	inf	torso	site	position (m)
3	xyz-coordinate angular velocity of the tons.	-inf	inf	torso	site	angular velocity (rad/s)
2	sin() and cos() of angle between torso and first link on front left	-inf	inf	hip_1 (front_left leg)	hinge	unitless
2	sin() and cos() of angle between torso and first link on front left	-inf	inf	ankle_1 (front_left leg)	hinge	unitless
2	sin() and cos() of angle between torso and first link on front left	-inf	inf	hip_2 (front_right leg)	hinge	unitless
2	sin() and cos() of angle between torso and first link on front left	-inf	inf	ankle_2 (front_right leg)	hinge	unitless
2	sin() and cos() of angle between torso and first link on front left	-inf	inf	hip_3 (back leg)	hinge	unitless
2	sin() and cos() of angle between torso and first link on front left	-inf	inf	ankle_3 (back leg)	hinge	unitless
2	sin() and cos() of angle between torso and first link on front left	-inf	inf	hip_4 (right_back leg)	hinge	unitless
2	sin() and cos() of angle between torso and first link on front left	-inf	inf	ankle_4 (right_back leg)	hinge	unitless
B.2 Multi-agents Specification
(a) 2-ant: Render
(b) 2-ant-diag: Render
(c) 4-ant: Render
(d) ant: Dynamics
Figure 13: A different view of the MA-Ant.
2-ant. The Ant is partitioned into 2 parts, the front part (containing the front legs) and the back part (containing the back legs). The action space of agent-0 and agent-1 as shown in Table 18 and Table 19.
Table 18: The specific action space of 2-ant: agent-0
Num	Action	Control Min	Control Max	Name (in XML file)	Joint	Unit
0	Torque applied on the rotor between the torso and front left hip	-1	1	hip_1 (front_left leg)	hinge	torque (N m)
1	Torque applied on the rotor between the front left two links	-1	1	angle_1 (front_left leg)	hinge	torque (N m)
2	Torque applied on the rotor between the torso and front right hip	-1	1	hip_2 (front_right leg)	hinge	torque (N m)
3	Torque applied on the rotor between the front right two links	-1	1	angle_2 (front_right leg)	hinge	torque (N m)
Table 19: The specific action space of 2-ant: agent-1
Num	Action	Control Min	Control Max	Name (in XML file)	Joint	Unit
0	Torque applied on the rotor between the torso and front left hip	-1	1	hip_1 (front_left leg)	hinge	torque (N m)
1	Torque applied on the rotor between the front left two links	-1	1	angle_1 (front_left leg)	hinge	torque (N m)
2	Torque applied on the rotor between the torso and front right hip	-1	1	hip_2 (front_right leg)	hinge	torque (N m)
3	Torque applied on the rotor between the front right two links	-1	1	angle_2 (front_right leg)	hinge	torque (N m)
2-ant-diag. The Ant is partitioned into 2 parts, split diagonally, the front part (containing the front legs) and the back part (containing the back legs). The action space of agent-0 and agent-1 as shown in Table 20 and Table 21.
Table 20: The specific action space of 2-ant-diag: agent-0
Num	Action	Control Min	Control Max	Name (in XML file)	Joint	Unit
0	Torque applied on the rotor between the torso and front left hip	-1	1	hip_1 (front_left leg)	hinge	torque (N m)
1	Torque applied on the rotor between the front left two links	-1	1	angle_1 (front_left leg)	hinge	torque (N m)
2	Torque applied on the rotor between the torso and back right hip	-1	1	hip_4 (right_back leg)	hinge	torque (N m)
3	Torque applied on the rotor between the back right two links	-1	1	angle_4 (right_back leg)	hinge	torque (N m)
Table 21: The specific action space of 4-ant: agent-1
Num	Action	Control Min	Control Max	Name (in XML file)	Joint	Unit
0	Torque applied on the rotor between the torso and front right hip	-1	1	hip_2 (front_right leg)	hinge	torque (N m)
1	Torque applied on the rotor between the front right two links	-1	1	angle_2 (front_right leg)	hinge	torque (N m)
2	Torque applied on the rotor between the torso and back left hip	-1	1	hip_3 (back leg)	hinge	torque (N m)
3	Torque applied on the rotor between the back left two links	-1	1	angle_3 (back leg)	hinge	torque (N m)
4-ant. The Ant is partitioned into 4 parts, with each part corresponding to a leg of the ant. The action space of agent-0, agent-1, agent-2, and agent-3 as shown in Table 22, Table 23, Table 24 and Table 25.
Table 22: The specific action space of 4-ant: agent-0
Num	Action	Control Min	Control Max	Name (in XML file)	Joint	Unit
0	Torque applied on the rotor between the torso and front left hip	-1	1	hip_1 (front_left leg)	hinge	torque (N m)
1	Torque applied on the rotor between the front left two links	-1	1	angle_1 (front_left leg)	hinge	torque (N m)
Table 23: The specific action space of 2-ant-diag: agent-1
Num	Action	Control Min	Control Max	Name (in XML file)	Joint	Unit
0	Torque applied on the rotor between the torso and front right hip	-1	1	hip_2 (front_right-legged)	hinge	torque (N m)
1	Torque applied on the rotor between the front right two links	-1	1	angle_2 (front_right-legged)	hinge	torque (N m)
Table 24: The specific action space of 4-ant: agent-2
Num	Action	Control Min	Control Max	Name (in XML file)	Joint	Unit
0	Torque applied on the rotor between the torso and back left hip	-1	1	hip_3 (back leg)	hinge	torque (N m)
1	Torque applied on the rotor between the back left two links	-1	1	angle_3 (back leg)	hinge	torque (N m)
Table 25: The specific action space of 4-ant: agent-3
Num	Action	Control Min	Control Max	Name (in XML file)	Joint	Unit
0	Torque applied on the rotor between the torso and back right hip	-1	1	hip_4 (right_back-legged)	hinge	torque (N m)
1	Torque applied on the rotor between the back right two links	-1	1	angle_4 (right_back-legged)	hinge	torque (N m)
In addition to the robots mentioned in this paper, we also provide other multi-agent versions of robots. Due to space constraints, we did not elaborate on them extensively in the paper. However, you can refer to https://www.safety-gymnasium.com/ for more detailed information.
B.3 Task Representation
(a) Velocity
(b) Run
(b) Circle
(d) Goal
Figure 14: Tasks of Gymnasium-based Environments.
(e) Button
(f)Push
As shown in Figure 14, the Gymnasium-based learning environments support the following tasks:
Velocity: the robot aims to facilitate coordinated leg movement of the robot in the forward (right) direction by exerting torques on the hinges.
Run: the robot starts with a random initial direction and a specific initial speed as it embarks on a journey to reach the opposite side of the map.
Circle: the reward is maximized by moving along the green circle, and the agent is not allowed to enter the outside of the red region, so its optimal constrained path follows the line segments  $AD$  and  $BC$ . The reward function:  $R(s) = \frac{v^T[-y,x]}{1 + ||[x,y]||_2 - d|}$ , the cost function is  $C(s) = 1\left[||x| > x_{\mathrm{lim}}\right]$ , where  $x,y$  are the coordinates in the plane,  $v$  is the velocity, and  $d, x_{\mathrm{lim}}$  are environmental parameters.
Goal: the robot navigates to multiple goal positions. After successfully reaching a goal, its location is randomly reset while maintaining the overall layout. Achieving a goal position, indicated by entering the goal circle, yields a sparse reward. Additionally, a dense reward encourages the robot's progress by rewarding proximity to the goal.
Push: the objective is to move a box to a series of goal positions. Like the goal task, a new random goal location is generated after each successful achievement. The sparse reward is earned when the yellow box enters the designated goal circle. The dense reward consists of two components: one for moving the agent closer to the box and another for bringing the box closer to the final goal.
Button: the objective is to activate a series of goal buttons distributed throughout the environment. The agent's goal is to navigate towards and make contact with the currently highlighted button, known as the goal button. Once the correct button is pressed, a new goal button is selected and highlighted while preserving the rest of the environment. The sparse reward is earned upon successfully pressing the current goal button, while the dense reward component provides a bonus for progressing toward the highlighted goal button.
B.4 Constraint Specification
(a) Velocity Constraints
(b) Pillars
(c) Hazards
(d) Sigwalls
Figure 15: Constraints of Gymnasium-based Environments.
(e) Vases
(f) Gremlins
Velocity-Constraint consists of a series of safety tasks based on MuJoCo agents [23]. In these tasks, agents, such as Ant, HalfCheetah, and Humanoid, are trained to move faster for higher rewards, while also being imposed a velocity constraint for safety considerations. Formally, for an agent moving on a two-dimensional plane, the velocity is calculated as  $v(s,a) = \sqrt{v_x^2 + v_y^2}$ ; for an agent moving along a straight line, the velocity is calculated as  $v(s,a) = |v_x|$ , where  $v_x, v_y$  are the velocities of the agent in the  $x$  and  $y$  directions respectively. Then,  $cost(s,a) = [v(s,a) > v_{limit}]$ . Here,  $[P]$  denotes a notation where the value is 1 if the proposition  $P$  is true, and 0 otherwise.
Pillars are employed to represent large cylindrical obstacles within the environment. In the general setting, contact with a pillar incurs costs.
Hazards are utilized to model areas within the environment that pose a risk, resulting in costs when an agent enters such areas.
Sigwalls are designed specifically for Circle tasks. They serve as visual representations of two or four solid walls, which limit the circular area to a smaller region. Crossing the wall from inside the safe area to the outside incurs costs.
Vases are specifically designed for Goal tasks. They represent static and fragile objects within the environment. Touching or displacing these objects incurs costs for the agent.
Gremlins are specifically employed in the Button tasks. They represent moving objects within the environment that can interact with the agent.
B.5 Vision-only Tasks
In recent years, vision-only SafeRL has gained significant attention as a focal point of research, primarily due to its applicability in real-world contexts [40; 41]. While the initial iteration of Safety Gym offered rudimentary visual input support, there is room for enhancing the realism and complexity of its environment. To effectively evaluate vision-based safe reinforcement learning algorithms, we have devised some more realistic visual tasks utilizing MuJoCo. This enhanced environment facilitates the incorporation of both RGB and RGB-d inputs. More details can be
referred to our online documentation: https://www.safety-gymnasium.com/en/latest/environments/safe_vision.html.
(a) BuildingButton0
(b) BuildingButton1
(c) BuildingButton2
Figure 16: Overview of BuildingButton tasks.
The Level 0 of BuildingButton requires the agent to operate multiple machines within a construction site.
The Level 1 of BuildingButton requires the agent to proficiently and accurately operate multiple machines within a construction site, while concurrently evading other robots and obstacles present in the area.
The Level 2 of BuildingButton requires the agent to proficiently and accurately operate multiple machines within a construction site, while concurrently evading a heightened number of other robots and obstacles in the area.
(a) BuildingGoal0
Figure 17: Overview of BuildingGoal tasks.
(b) BuildingGoal1
(c) BuildingGoal2
The Level 0 of BuildingGoal requires the agent to dock at designated positions within a construction site.
The Level 1 of BuildingGoal requires the agent to dock at designated positions within a construction site while ensuring to avoid entry into hazardous areas.
The Level 2 of BuildingGoal requires the agent to dock at designated positions within a construction site, while ensuring to avoid entry into hazardous areas and circumventing the site's exhaust fans.
(a) BuildingPush0
(b) BuildingPush1
(c) BuildingPush2
Figure 18: Overview of BuildingPush tasks.
The Level 0 of BuildingPush requires the agent to relocate the box to designated locations within a construction site.
The Level 1 of BuildingPush requires the agent to relocate the box to designated locations within a construction site while avoiding areas demarcated as restricted.
The Level 2 of BuildingPush requires the agent to relocate the box to designated locations within a construction while avoiding numerous hazardous fuel drums and areas demarcated as restricted.
(a) Race0
Figure 19: Overview of Race tasks.
(b) Race1
(c) Race2
The Level 0 of Race requires the agent to reach the goal position.
The Level 1 of Race requires the agent to reach the goal position while ensuring it avoids straying into the grass and prevents collisions with roadside objects.
The Level 2 of Race requires the agent to reach the goal position from a distant starting point while ensuring it avoids straying into the grass and prevents collisions with roadside objects.
(a) FormulaOne0
(b) FormulaOne1
(c) FormulaOne2
Figure 20: Overview of FormulaOne tasks.
The Level 0 of FormulaOne requires the agent to maximize its reach to the goal position. For each episode, the agent is randomly initialized at one of the seven checkpoints.
The Level 1 of FormulaOne requires the agent to maximize its reach to the goal position while circumventing barriers and racetrack fences. For each episode, the agent is randomly initialized at one of the seven checkpoints.
The Level 2 of FormulaOne requires the agent to maximize its reach to the goal position while circumventing barriers and racetrack fences. For each episode, the agent is randomly initialized at one of the seven checkpoints. Notably, the barriers surrounding the checkpoints are denser.
B.6 Some Issues about Safety Gym
(a) Safety-Gymnasium
(b) Safety-Gym
Figure 21: The difference between Safety-Gymnasium and Safety Gym.
The bug of Natural Lidar. As shown in Figure 21, the original Natural Lidar in Safe-Gym<sup>7</sup> has a problem of not being able to detect low-lying objects, which may affect comprehensive environmental observations.
The problem of observation space. In Safety Gym, by default, the observation space is presented as a one-dimensional array. The implementation leads to all ranges in observation space to be  $[- \infty, + \infty]$ , as shown in the following code:
if selfobservation Flatten: 1   
self obs flatten size  $=$  sum([np.prod(i.shape) for i in 2   
self(obs_space_dict_values())   
selfobservation_space  $\equiv$  gymspaces.Box(-np.inf，np.inf, 3   
(self(obsflat_size,)，dtype=np.float32)
While this representation does not lead to behavioral errors in the environment, it can be somewhat misleading for users. To address this issue, we have implemented the Gymnasium's flatten mechanism in the Safety Gym to handle the representation of the observation space. This mechanism reorganizes the observation space into a more intuitive and easily understandable format, enabling users to process and analyze the observation data more effectively.
self obs_info(obs_space_dict = gymnasium.spaces.Dict(obs_space_dict) 1   
if selfobservation Flatten: 2   
selfobservation_space  $\equiv$  gymnasium.spaces.utils Flatten_space( 3   
self(obs_info(obs_space_dict 4   
else: 7   
selfobservation_space  $\equiv$  self(obs_info(obs_space_dict 8   
assert self(obs_info(obs_space_dict.contains( 9   
obs 10   
),f'Bad obs {obs} {self(obs_info(obs_space_dict)}' 11   
if selfobservation Flatten: 12   
obs  $=$  gymnasium.spaces.utils Flatten(self(obs_info(obs_space_dict, 14   
obs)   
return obs
Missing cost information. In Safety Gym, by default, there are only two possible outputs for the cost: 0 and 1, representing whether a cost is incurred or not.
#Optionally remove shaping from reward functions. 1   
if self.constrain_indicator: 2   
for k in list(cost.keys(): 3   
cost[k]  $=$  float(cost[k]  $\rightharpoondown$  0.0)#Indicator function 4
We believe that this representation method loses some information. For example, when the robot collides with a vase and causes the vase to move at different velocities, there should be different cost values associated with it to indicate subtle differences in violating constraint behaviors. Additionally, these costs incurred by the actions are accumulated into the total cost. In typical cases, algorithms use the total cost to update the policy if the total cost generated by different obstacles is limited to only two states 0 and 1, the learning potential for multiple constraints is lost when multiple costs are triggered simultaneously.
Neglected dependency maintenance leads to conflicts.
The numpy =1.17.4 will cause the following problems:
ValueError: numpy. ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 80 from PyObject AttributeError: module 'numpy' has no attribute 'complex'.
<sup>7</sup>https://github.com/openai/safety-gym
C Details of Isaac Gym-based Learning Environments
C.1 Supported Agents
Safety-DexteroudsHand is based on Bi-DexHands (refer to [42] for more details). Bi-DexHands aims to establish a comprehensive learning framework for two Shadow Hands, enabling them to possess a wide range of skills similar to those of humans. The Shadow Hand's joint limitations are as follows (refer to Table 26). The thumb exhibits 5 degrees of freedom with 5 joints, while the other fingers have 3 degrees of freedom and 4 joints each. The joints located at the fingertips are not controllable. Similar to human fingers, the distal joints of the fingers are interconnected, ensuring that the angle of the middle joint is always greater than or equal to that of the distal joint. This design allows the middle phalange to be curved while the distal phalange remains straight. Additionally, an extra joint (LF5) is located at the end of the little finger, enabling it to rotate in the same direction as the thumb. The wrist comprises two joints, facilitating a complete 360-degree rotation of the entire hand.
Table 26: Finger range of motion.
Joints	Corresponds to the number of Figure 22	Min	Max
Finger Distal (FF1,MF1,RF1,LF1)	15, 11, 7, 3	0°	90°
Finger Middle (FF2,MF2,RF2,LF2)	16, 12, 8, 4	0°	90°
Finger Base Abduction (FF3,MF3,RF3,LF3)	17, 13, 9, 5	-15°	90°
Finger Base Lateral (FF4,MF4,RF4,LF4)	18, 14, 10, 6	-20°	20°
Little Finger Rotation(LF5)	19	0°	45°
Thumb Distal (TH1)	20	-15°	90°
Thumb Middle (TH2)	21	-30°	30°
Thumb Base Abduction (TH3)	22	-12°	12°
Thumb Base Lateral (TH4)	23	0°	70°
Thumb Base Rotation (TH5)	24	-60°	60°
Hand Wrist Abduction (WR1)	1	-40°	28°
Hand Wrist Lateral (WR2)	2	-28°	8°
Stiffness, damping, friction, and armature are also important physical parameters in robotics. For each Shadow Hand joint, we show our DoF properties in Table 27. This part can be adjusted in the Isaac Gym simulator.
Table 27: DoF properties of Shadow Hand.
Joints	Stiffness	Damping	Friction	Armature
WR1	100	4.78	0	0
WR2	100	2.17	0	0
FF2	100	3.4e+38	0	0
FF3	100	0.9	0	0
FF4	100	0.725	0	0
MF2	100	3.4e+38	0	0
MF3	100	0.9	0	0
MF4	100	0.725	0	0
RF2	100	3.4e+38	0	0
RF3	100	0.9	0	0
RF4	100	0.725	0	0
LF2	100	3.4e+38	0	0
LF3	100	0.9	0	0
LF4	100	0.725	0	0
TH2	100	3.4e+38	0	0
TH3	100	0.99	0	0
TH4	100	0.99	0	0
TH5	100	0.81	0	0
Figure 22: Degree-of-Freedom (DOF) configuration of the Shadow Hand similar to the skeleton of a human hand.
Table 28: Observation space of dual Shadow Hands.
Index	Description
0 - 23	right Shadow Hand dof position
24 - 47	right Shadow Hand dof velocity
48 - 71	right Shadow Hand dof force
72 - 136	right Shadow Hand fingertip pose, linear velocity, angle velocity (5 x 13)
137 - 166	right Shadow Hand fingertip force, torque (5 x 6)
167 - 169	right Shadow Hand base position
170 - 172	right Shadow Hand base rotation
173 - 198	right Shadow Hand actions
199 - 222	left Shadow Hand dof position
223 - 246	left Shadow Hand dof velocity
247 - 270	left Shadow Hand dof force
271 - 335	left Shadow Hand fingertip pose, linear velocity, angle velocity (5 x 13)
336 - 365	left Shadow Hand fingertip force, torque (5 x 6)
366 - 368	left Shadow Hand base position
369 - 371	left Shadow Hand base rotation
372 - 397	left Shadow Hand actions
C.2 Task Representation
Hand Over
This scenario encompasses a specific environment comprising two Shadow Hands positioned opposite each other, with their palms facing upwards. The objective is to pass an object between these hands. Initially, the object will randomly descend within the area of the Shadow Hand on the right side. The hand on the right side then grasps the object and transfers it to the other hand. It is important to note that the base of each hand remains fixed throughout the process. Furthermore, the hand initially holding the object cannot directly make contact with the target hand or roll the object towards it. Hence, the object must be thrown into the air, maintaining its trajectory until it reaches the target hand.
In this task, there are 398-dimensional observations and 40-dimensional actions. The reward function is closely tied to the positional discrepancy between the object and the target. As the pose error diminishes, the reward increases significantly. The detailed observation space for each agent can be found in Table 29, while the corresponding action space is outlined in Table 30.
Observations The observational space for the Hand Over task consists of 398 dimensions, as indicated in Table 29. However, it is important to highlight that in this particular task, the base of the dual hands remains fixed. Therefore, the observation for the dual hands is compared to a reduced 24-dimensional space, as described in Table 28.