SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Correctness For a benchmark, it is critical to ensure its correctness and reliability. Firstly, each algorithm is implemented strictly according to the original paper (e.g., ensuring consistency with the gradient flow of the original paper, etc.). Secondly, we compare our implementation with those line by line for algorithms with a commonly acknowledged open-source code base to double-check the correctness. Finally, we compare SafeP0 with existing benchmarks (e.g., Safety-Startup-Agents $^4$  and RL-Safety-Algorithms $^5$ ) and SafeP0 outperforms or achieves comparable performance with other existing implementations, as shown in Table 1.
<sup>4</sup>Safety-Starter-Agents: https://github.com/openai/safety-starter-agents
$^{5}$ RL-Safety-Algorithms: https://github.com/SvenGronauer/RL-Safety-Algorithms
Figure 7: The Architecture of SafePO
Extensibility SafeP0 enjoys high extensibility thanks to its architecture (as shown in Figure 7). New algorithms can be integrated into SafeP0 by inheriting from base algorithms and only implementing their unique features. For example, we integrate PPO by inheriting from policy gradient and only adding the clip ratio variable and rewriting the function that computes the loss of policy  $\pi$ . Similarly, algorithms can be easily added to SafeP0.
Logging and Visualization Another necessary functionality of SafePO is logging and visualization. Supporting both TensorBoard and WandB, we offer code for visualizing more than 40 parameters and intermediate computation results to inspect the training process. Standard parameters and metrics such as KL-divergence, SPS (step per second), and cost variance are visualized universally. Special features of algorithms are also reported, such as the Lagrangian multiplier of Lagrangian-based methods,  $g^T H^{-1}g$ ,  $g^T H^{-1}b$ ,  $\nu^{*}$ , and  $\lambda^{*}$  of CPO, proportional, integral, and derivative of PID-Lagrangian algorithms, etc. During training, users can inspect the changes of every parameter, collect the log file, and obtain saved checkpoint models. The complete and comprehensive visualization allows easier observation, model selection, and comparison.
Documentation In addition to its code implementation, SafeP0 comes with an extensive documentation<sup>6</sup>. We include detailed guidance on installation and propose solutions to common issues. Moreover, we provide instructions on simple usage and advanced customization of SafeP0. Official information concerning maintenance, ethical, and responsible use are stated clearly for reference.
Table 1: A comparison between SafePO and other implementations. Results are based on 10 evaluation iterations using over 3 seeds under cost_limit=25.00.  $\bar{J}^{R}$  stands for normalized reward from PPO's performance,  $\bar{J}^{C}$  signifies normalized cost relative to cost_limit, and AvgR/AvgC represents the ratio of the means of both across 10 environments. The  $\uparrow$  indicates higher rewards are better, while the  $\downarrow$  indicates lower costs (when beyond the threshold of 1.00) are better. Gray and Black depicts violation and compliance with the cost_limit.
CPO	TRPO-Lag	PPO-Lag	FOCOPS
SafePO (Ours)	Safety Starter Agents	RL-Safety-Algorithms	SafePO (Ours)	Safety Starter Agents	RL-Safety-Algorithms	SafePO (Ours)	Safety Starter Agents	SafePO (Agents)	SafePO (Ours)	Original Implementation
Safety Navigation	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓
CARBUTTON1	0.08	1.75	0.34	3.65	-0.06	3.30	-0.04	1.08	0.02	0.78	-0.05	0.63	0.01	0.47	0.02	0.67	0.04	1.21	0.53	6.02
CARGOLD1	0.78	1.63	0.94	2.49	0.46	1.25	0.82	1.09	0.72	1.04	0.72	0.91	0.43	0.39	0.52	0.52	0.52	0.93	0.79	2.45
POINTBUTTON1	0.12	1.61	0.70	3.01	0.03	3.25	0.27	1.29	0.21	0.92	0.04	0.87	0.22	1.32	0.17	0.96	0.25	1.53	0.70	3.74
POINTGOLD1	0.78	1.10	0.81	1.99	0.28	2.05	0.72	0.91	0.65	0.94	0.33	0.72	0.47	1.50	0.66	0.77	0.56	1.32	0.81	1.53
Safety Velocity	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓
ANTVEL	0.52	0.56	0.31	0.93	0.40	1.09	0.53	0.15	0.32	0.76	0.44	0.70	0.54	0.22	0.31	0.61	0.55	0.60	0.52	0.39
HALFCHEETAHELVEL	0.40	0.23	0.30	1.13	0.31	0.97	0.43	1.01	0.25	0.79	0.43	0.67	0.44	0.04	0.30	0.93	0.42	0.12	0.44	0.04
HOPPERVEL	0.73	0.48	0.35	0.93	0.26	0.68	0.59	0.71	0.41	1.11	0.24	0.57	0.58	0.89	0.29	1.20	0.66	0.30	0.74	0.53
HUMANOIDVEL	0.71	0.01	0.05	0.19	0.36	0.83	0.72	2.38	0.05	0.01	0.71	0.79	0.72	0.76	0.07	0.09	0.71	0.93	0.73	0.43
SWIMMERVEL	0.51	0.82	0.38	1.11	0.41	0.82	0.66	0.84	0.43	1.67	0.41	1.02	0.57	1.11	0.38	1.18	0.47	1.30	0.68	0.71
WALKER2DVEL	0.39	0.81	0.44	1.85	0.05	0.67	0.51	0.77	0.46	0.67	0.51	1.34	0.44	0.20	0.47	0.81	0.50	0.68	0.48	0.74
AvgR/AvgC	0.56	0.27	0.17	0.51	0.46	0.64	0.52	0.39
<sup>6</sup>SafePO's Documentation: https://safe-policy-optimization.readthedocs.io
(a) Average Episodic Reward of Algorithms
(b) Bar Chart Categorizing Algorithms into Four Classes Based on Average Episodic Cost
Figure 8: A bar chart analyzing the performance of different algorithms. The left graph compares episodic reward with PPO-Lag [18] (or MAPPO-Lag [39] for multi-agent). The right graph shows episodic costs proportionally under varying constraints. Single-agent data is from 40 navigation and 6 velocity tasks, and multi-agent data is from all 8 velocity tasks in Safety-Gymnasium.
Reward and Cost. Episodic reward and cost exhibit a trade-off relationship. Unconstrained algorithms aim to maximize reward through risky behaviors. HAPPO [37] achieves higher rewards compared to MAPPO [38] across 8 velocity-based tasks, accompanied by a simultaneous increase in average costs. SafeRL algorithms tend to maximize reward while adhering to constraints. As depicted in Table 2, in the velocity task, compared to PPO [43], PPO-Lag [18] achieves a reduction of  $98\%$  in cost while only experiencing a decrease of  $45\%$  in reward.
Randomness and Oscillation. The randomness of tasks is correlated with the oscillation of algorithms' performance. All SafeRL algorithms achieve average episodic costs within the cost_limit for velocity tasks. The divergence in episodic rewards between algorithms is negligible, and the distribution of optimal policies is tightly clustered. However, pronounced oscillations are present in navigation tasks characterized by high stochasticity. Out of the 20 navigation tasks examined,
optimal policies are spread out more, leading to observable differences in algorithm performance across various tasks.
Lagrangian vs. Projection. In contrast to projection-based methods, the Lagrangian-based methods tend to display more oscillation. A notable disparity becomes apparent upon examining the oscillatory patterns in the episodic cost around the designated safety constraints during training, as presented in Figure 8(b). Both CPO [30] and PPO-Lag [18] demonstrate oscillations; however, those exhibited by PPO-Lag are more conspicuous. This discrepancy is manifested in a higher proportion of instances clas
Table 3: The normalized performance of SafePO's multi-agent algorithms on Safety-Gymnasium.
MAPPO	HAPPO	MAPPO-Lag	MACPO
Safety Velocity	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓	JR↑	JC↓
2x4ANTVEL	1.00	35.76	1.26	39.12	0.57	0.00	0.51	0.14
4x2ANTVEL	1.00	38.01	1.07	34.34	0.50	0.00	0.50	0.01
2x3HALFCHEETAHVEL	1.00	39.02	1.11	37.70	0.35	0.01	0.49	1.28
6x1HALFCHEETAHVEL	1.00	39.23	1.09	37.74	0.28	0.02	0.36	0.37
3X1HOPPERVEL	1.00	22.58	1.04	22.05	0.47	0.00	0.22	1.03
98HUMANOIDVEL	1.00	6.34	2.79	17.18	0.54	0.84	0.53	1.30
2x3WALKER2DVEL	1.00	22.99	1.55	33.67	0.60	0.01	0.27	1.21
sified as Strongly Unsafe and Strongly Safe for PPO-Lag, while CPO maintains a more centered distribution. Nevertheless, an excessively cautious policy has the potential to undermine performance. In contrast, the projection-based method PCPO [3] exhibits lower average costs and rewards in navigation and velocity tasks than CPO. This distinction is further accentuated when examining the contrast between MACPO and MAPPO-Lag.
Lagrangian vs. PID-Lagrangian. Incorporating a PID controller within the Lagrangian-based framework proves to be effective in mitigating inherent oscillations. As shown in Figure 8, CPPO-PID [44] displays episodic rewards during training that closely resemble those of PPO-Lag. However, CPPO-PID demonstrates a reduced frequency of instances entering the Strongly Unsafe region, resulting in a more significant proportion of Safe states and improved safety performance.
7 Limitations and Future Works
Ensuring safety remains a paramount concern. Across various tasks, safety concerns can be transformed into corresponding constraints. However, a limitation of this study is its inability to encompass all forms of constraints. For instance, safety constraints related to human-centric considerations are paramount in human-AI collaboration, yet these considerations have not been fully integrated within the scope of this study. This work focuses on safety tasks within a simulated environment and introduces an extensive testing component. However, the transferability of the results to complex real-world safety-critical applications may be limited. A promising work for the future involves transferring policy refined within the Safety-Gymnasium to physical robotic platforms, which holds profound implications.
[36] Christian Schroeder de Witt, Bei Peng, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint arXiv:2003.06709, 19, 2020.
[37] Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint arXiv:2109.11251, 2021.
[38] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness ofppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611-24624, 2022.
[39] Shangding Gu, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang, Zheng Tian, Jun Wang, Alois Knoll, and Yaodong Yang. Multi-agent constrained policy optimisation. arXiv preprint arXiv:2110.02793, 2021.
[40] Yecheng Jason Ma, Andrew Shen, Osbert Bastani, and Jayaraman Dinesh. Conservative and adaptive penalty for model-based safe reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 36(5), pages 5404-5412, 2022.
[41] Yarden As, Ilnura Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy optimization via bayesian world models. arXiv preprint arXiv:2201.09802, 2022.
[42] Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen McAleer, Hao Dong, Song-Chun Zhu, and Yaodong Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning. Advances in Neural Information Processing Systems, 35:5150-5163, 2022.
[43] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[44] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pages 9133-9143. PMLR, 2020.
[45] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.
[46] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
A Details of Experimental Results
A.1 Hyperparameters Analysis
This section presents the disclosure of SafePO hyperparameters settings and their rationales. We employed the Generalized Advantage Estimation (GAE)[45] method to estimate the values of rewards and cost advantages and used Adam[46] for learning the neural network parameters.
Single-agent Algorithm Settings. The models employed in the single-agent algorithms were 3-layer MLPs with Tanh activation functions and hidden layer sizes of [64, 64], for more intricate navigation agents Ant and Doggo, hidden layers of [256, 256] were employed.
Multi-agent Algorithms Settings. The models employed in the multi-agent algorithms were 3-layer MLPs with ReLU activation functions and hidden layer sizes of [128, 128].
Table 4: Hyperparameters of SafeP0 algorithms in Safety-Gymnasium tasks. Second-order algorithms set the parameters to the actor model directly, instead of iterative gradient descent, so the Actor Learning Rate of them are marked Gray.
PG/PPO/PPO-Lag	Value	TRPO/TRPO-Lag	Value	CPPO-PID	Value	NPG/RCPO	Value	HAPPO/MAPPO/MAPPO-Lag	Value
Discount Factor γ	0.99	Discount Factor γ	0.99	Discount Factor γ	0.99	Discount Factor γ	0.99	Discount Factor γ	0.99
Target KL	0.02	Target KL	0.01	Target KL	0.02	Target KL	0.01	Target KL	0.016
GAE λ	0.95	GAE λ	0.95	GAE λ	0.95	GAE λ	0.95	GAE λ	0.95
Number of SGD Iterations	40	Number of SGD Iterations	10	Number of SGD Iterations	10	Number of SGD Iterations	10	Number of SGD Iterations	5
Training Batch Size	20000	Training Batch Size	20000	Training Batch Size	20000	Training Batch Size	20000	Training Batch Size	10000
Actor Learning Rate	0.0003	Actor Learning Rate	None	Actor Learning Rate	0.0003	Actor Learning Rate	None	Actor Learning Rate	0.0005
Critic Learning Rate	0.0003	Critic Learning Rate	0.001	Critic Learning Rate	0.0003	Critic Learning Rate	0.001	Critic Learning Rate	0.0005
Cost Limit	25.00	Cost Limit	25.00	Cost Limit	25.00	Cost Limit	25.00	Cost Limit	25.00
Clip Coefficient	0.20	Conjugate Gradient Iterations	15	Clip Coefficient	0.20	Conjugate Gradient Iterations	15	Clip Coefficient	0.20
Lagrangian Initial Value	0.001	Lagrangian Initial Value	0.001	PID Controller Kp	0.10	Lagrangian Initial Value	0.001	Lagrangian Initial Value	0.00001
Lagrangian Learning Rate	0.035	Lagrangian Learning Rate	0.035	PID Controller KI	0.01	Lagrangian Learning Rate	0.035	Lagrangian Learning Rate	0.78
Lagrangian Optimizer	Adaan	Lagrangian Optimizer	Adaan	PID Controller Kd	0.01	Lagrangian Optimizer	Adaan	Lagrangian Optimizer	SGD
CPO	Value	PCPO	Value	CUP	Value	FOCOPS	Value	MACPO	Value
Discount Factor γ	0.99	Discount Factor γ	0.99	Discount Factor γ	0.99	Discount Factor γ	0.99	Discount Factor γ	0.99
Target KL	0.01	Target KL	0.01	Target KL	0.02	Target KL	0.02	Target KL	0.01
GAE λ	0.95	GAE λ	0.95	GAE λ	0.95	GAE λ	0.95	GAE λ	0.95
Number of SGD Iterations	10	Number of SGD Iterations	10	Number of SGD Iteration	10	Number of SGD Iteration	10	Number of SGD Iteration	15
Training Batch Size	20000	Training Batch Size	20000	Training Batch Size	20000	Training Batch Size	20000	Training Batch Size	10000
Actor Learning Rate	None	Actor Learning Rate	None	Actor Learning Rate	0.0003	Actor Learning Rate	0.0003	Actor Learning Rate	None
Critic Learning Rate	0.001	Critic Learning Rate	0.001	Critic Learning Rate	0.0003	Critic Learning Rate	0.0003	Critic Learning Rate	0.0005
Cost Limit	25.00	Cost Limit	25.00	Cost Limit	25.00	Cost Limit	25.00	Cost Limit	25.00
Conjugate Gradient Iterations	15	Conjugate Gradient Iterations	15	Clip Coefficient	0.20	Clip Coefficient	0.20	Conjugate Gradient Iterations	10
LPO Searching Steps	15	PCPO Searching Steps	200	CUP λ	0.95	FOCOPS λ	1.50	MACPO Searching Steps	10
Step Fraction	0.80	Step Fraction	0.80	CUP ρ	2.00	FOCOPS ρ	2.00	Step Fraction	0.50
Lagrangian Multiplier Settings. Lagrangian-based methods are sensitive to hyperparameters. We present the following detailed description of the settings for both the naive and the PID-controlled Lagrangian multiplier.
- Lagrangian Initial Value: The initial value of the Lagrangian multiplier. It impacts the early-stage performance of the Lagrangian-based methods. A higher initial value promotes safer exploration but may impede task completion. Conversely, a lower initial value delays the agent's exploration of safe policies.
- Lagrangian Learning Rate: The learning rate of the Lagrangian multiplier. A high learning rate induces excessive oscillations, impedes convergence speed, and hinders the algorithm's ability to attain the desired solution. Conversely, a low learning rate slows down convergence and adversely affects training.
- PID Controller Kp: The PID controller's proportional gain determines the output's response to changes in the episodic costs. If pid_kp is too large, the Lagrangian multiplier oscillates, and performance deteriorates. If pid_kp is too small, the Lagrangian multiplier updates slowly, also impacting performance negatively.
- PID Controller Kd: The PID controller's derivative gain governs the output's response to changes in the episodic costs. If pid_kd is too large, the Lagrangian multiplier becomes excessively sensitive to noise or changes in the episodic costs, leading to instability or oscillations. If pid_kd is too small, the Lagrangian multiplier may not respond quickly or accurately enough to changes in the episodic costs.
- PID Controller Ki: The PID controller's integral gain determines the controller's ability to eliminate the steady-state error by integrating the episodic costs over time. If pid_ki is too large, the Lagrangian multiplier may become overly responsive to previous errors, adversely affecting performance.
A.2 Performance Table of Safety-Gymnasium
Table 5: The performance of SafePO algorithms on Safety-Gymnasium. All experimental outcomes were derived from 10 assessment iterations encompassing multiple random seeds and under the experimental setting of cost_limit=25.00. The  $\uparrow$  indicates higher rewards are better, while the  $\downarrow$  indicates lower costs (when beyond the threshold of 25.00) are better. Gray and Black depicts breach and compliance with the cost_limit, while Green represents the optimal policy, maximizing reward within safety constraints.
(a) The performance of SafePO single-agent algorithms on Safety-Gymnasium.
(b) The performance of SafePO multi-agent algorithms on Safety-Gymnasium.
MAPPO	HAPPO	MAPPO-Lag	MACPO
Safety Velocity	JR	JC	JR	JC	JR	JC	JR	JC
2x4ANTVEL	4259.52	894.06	5368.61	978.06	2423.47	0.00	2169.23	3.39
4x2ANTVEL	4309.05	950.33	4613.69	858.50	2171.40	0.00	2172.31	0.17
2x3HALFCHEETAHVEL	5057.63	975.50	5605.98	942.56	1750.96	0.33	2470.29	32.06
6x1HALFCHEETAHVEL	5061.53	980.67	5540.57	943.56	1439.38	0.61	1830.65	9.33
3x1HOPPERVEL	2115.35	564.56	2207.50	551.33	1002.01	0.00	461.25	25.78
9HUMANOIDVEL	974.50	158.61	2718.48	429.61	526.69	21.00	512.29	32.50
2x1SWIMMERVEL	39.88	101.89	51.95	267.00	27.89	59.73	-4.02	20.83
2x3WALKER2VEL	2691.41	574.72	4183.34	841.83	1618.98	0.33	714.18	30.22
Experimental Results Analysis.
During the observation of the experimental results, we have discovered some Insightful findings that are presented below:
- The Lagrangian method is a promising yet constrained baseline approach, successfully optimizing rewards while adhering to constraints. However, its effectiveness heavily relies on hyperparameters configuration, as discussed in Table A.1. Consequently, despite being a dependable baseline, the Lagrangian method is not exempt from limitations.
- Second-order algorithms perform worse in achieving higher rewards in the MuJoCo velocity series but better in navigation series tasks that require higher safety standards, i.e., achieving similar or approximate rewards while minimizing the number and smoothness of cost violations.
B Details Documentation of Gymnasium-based Learning Environments
B.1 Single-agent Specification
(a) Point: front
(b) Point: back
(c) Point: left
Figure 9: A different view of the robot: Point.
(d) Point: right
Table 6: The overall information of Point
Specific Action Space	Box(-1.0, 1.0, (2.), float64)
Specific Observation Space	(12,)
Observation High	inf
Observation Low	-inf
Table 7: The specific observation space of Point
Size	Observation	Min	Max	Name (in XML file)	Joint/Site	Unit
3	accelerometer	-inf	inf	accelerometer	site	acceleration (m/s^2)
3	velocimeter	-inf	inf	velocimeter	site	velocity (m/s)
3	gyro	-inf	inf	gyro	site	angular velocity (rad/s)
3	magnetometer	-inf	inf	magnetometer	site	magnetic flux (Wb)
Table 8: The specific action space of Point
Num	Action	Control Min	Control Max	Name (in XML file)	Joint/Site	Unit
0	force applied on the agent to move forward or backward	-1	1	x	site	force (N)
1	velocity of the agent, which is around the z-axis	-1	1	z	hinge	velocity (m/s)
Point: As shown in Figure 9, Point operating within a 2D plane is equipped with two distinct actuators: one for rotation and another for forward/backward movement. This decomposed control system greatly facilitates the navigation of the robot. Moreover, there is a small square positioned in front of the robot, aiding in the visual identification of its orientation. Additionally, this square plays a crucial role in assisting the robot, named Point, to effectively push any boxes encountered during its tasks. The overall information of Point, the specific action and observation space of Point is shown in Table 6, Table 8, Table 7.
(a) Car: front
Figure 10: A different view of the robot: Car.
(b) Car: back
(c) Car: left
(d) Car: right
Table 9: The overall information of Car
Specific Action Space	Box(-1.0, 1.0, (2.), float64)
Specific Observation Space	(24,)
Observation High	inf
Observation Low	-inf
Table 10: The specific action space of Car
Num	Action	Control Min	Control Max	Name (in XML file)	Joint/Site	Unit
0	force to applied on left wheel	-1	1	left	hinge	force (N)
1	force to applied on right wheel	-1	1	right	hinge	force (N)
Table 11: The specific observation space of Car
Size	Observation	Min	Max	Name (in XML file)	Joint/Site	Unit
3	Quaternions of the rear wheel which are turned into 3x3 rotation matrices	-inf	inf	ballquat_rear	ball	unitless
3	Angle velocity of the rear wheel	-inf	inf	ballangvel_rear	ball	angular velocity (rad/s)
3	accelerometer	-inf	inf	accelerometer	site	acceleration (m/s^2)
3	velocimeter	-inf	inf	velocimeter	site	velocity (m/s)
3	gyro	-inf	inf	gyro	site	angular velocity (rad/s)
3	magnetometer	-inf	inf	magnetometer	site	magnetic flux (Wb)
Car: As shown in Figure 10, the robot in question operates in three dimensions and features two independently driven parallel wheels, along with a freely rolling rear wheel. This design requires coordinated operation of the two drives for both steering and forward/backward movement. While the robot shares similarities with a basic Point robot, it possesses added complexity. The overall information of Car, the specific action and observation space of Car is shown in Table 9, Table 10, Table 11.
(a) Racecar: front
(b) Racecar: back
(c) Racecar: left
Figure 11: A different view of the robot: Racecar.
(d) Racecar: right
Table 12: The overall information of Racear
Specific Action Space	Box([-20. -0.785], [20. 0.785], (2.), float64)
Specific Observation Space	(12,)
Observation High	inf
Observation Low	-inf
Table 13: The specific action space of Racecar
Num	Action	Control Min	Control Max	Name (in XML file)	Joint/Site	Unit
0	Velocity of the rear wheels.	-20	20	diff_ring	hinge	velocity (m/s)
1	Angle of the front wheel.	-0.785	0.785	steering_hinge	hinge	angle (rad)
Table 14: The specific observation space of Racecar
Size	Observation	Min	Max	Name (in XML file)	Joint/Site	Unit
3	accelerometer	-inf	inf	accelerometer	site	acceleration (m/s^2)
3	velocimeter	-inf	inf	velocimeter	site	velocity (m/s)
3	gyro	-inf	inf	gyro	site	angular velocity (rad/s)
3	magnetometer	-inf	inf	magnetometer	site	magnetic flux (Wb)
Racecar. As shown in Figure 11, the robot is closer to realistic car dynamics, moving in three dimensions, it has one velocity servo and one position servo, one to adjusts the rear wheel speed to
the target speed and the other to adjust the front wheel steering angle to the target angle. Racecar references the widely known MIT Racecar project's dynamics model. For it to accomplish the specified goal, it must coordinate the relationship between the steering angle of the tires and the speed, just like a human driving a car. The overall information of Racecar, the specific action and observation space of Racecar is shown in Table 12, Table 13, Table 14.
Table 15: The overall information of Ant
Specific Action Space	Box(-1.0, 1.0, (8.), float64)
Specific Observation Space	(40,)
Observation High	inf
Observation Low	-inf