SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Require: number of agents  $n$ , number of updates  $K$ , minibatch size  $B$ , temperature  $\{\lambda_j\}_{1 \leq j \leq m^i}$ , initial cost constraint parameter  $\{\nu_j\}_{1 \leq j \leq m^i}$ , cost constraint parameter bound  $v_{max}$ , learning rate for cost constraint parameter  $\alpha_{\nu}$ , trust region bound  $\delta$ , cost bound  $b_j$
1: Initialize, policy networks  $\{\pi_{\theta_0}^i,i\in \mathcal{N}\}$ , global value network  $\{\phi_0\}$  and cost value networks  $\{\phi_{j,0}^{i}\}_{1\leq j\leq m^{i}}$ , replay buffer  $\mathcal{B}$
2: for  $k = 0,1,\ldots$  do
3: Generate trajectories  $\tau \sim \pi_{\theta_k}$ , save the data into the buffer and sample a batch of data;
4: Estimate the C-returns  $\hat{J}_C$  by averaging over the cost return for all episodes.
5: Compute the advantage functions  $\hat{A}_{\pi_{\theta_k}}(s,\pmb {a})$  and  $\hat{A}_{j,\pi_{\theta_k}}^i (s,a^i)$  using GAE;
6: Draw a permutation  $i_{1:n}$  of agents at random.
7: Set  $M^{i_1}(s, \pmb{a}) = \hat{A}_{\pmb{\pi}_{\theta_k}}(s, \pmb{a})$
8: for agent  $i_h = i_1, i_2, \dots, i_n$  do
9: Update  $\nu_{j}$  by  $\nu_{j}$  by  $\nu_{j} \gets \text{proj}_{\nu_{j}}[\nu_{j} - \alpha(c_{j}^{i_{h}} - \bar{J}_{C,j}^{i_{h}}(\pi_{\pmb{\theta}_{k}}))], \forall j = 1, \dots, m^{i_{h}}$
10: for  $K$  epochs do
11: for each minibatch data of size  $B$  do
12: Update value networks (and cost value networks analogously) by minimizing the MSE loss  $\phi_{k + 1} = \arg\min_{\phi}\sum_{t = 0}^{T}(V_{\phi_k}(s_t) - \hat{R}_t)^2$ , where  $\hat{R}$  is the target return.
13: Update policy network by the derived equation of  $\nabla_{\theta}L(\theta)$  (Eq. 11), where  $\hat{\eta}_{\pi_{\theta_k}}(s,a^{i_h})$  is estimated by  $M^{i_{1:h}}(s,\pmb{a})$ .
14: end for
15: if  $\bar{D}_{KL}(\pi^{i_h},\pi_{\theta_k}^{i_h})\geq \delta$  then
16: Break
17: end if
18: end for
19: Compute  $M^{i_{1:h+1}}(s, \boldsymbol{a}) = \frac{\pi_{\theta_{k+1}^i h}^{i_h}(a^{i_h}|o^{i_h})}{\pi_{\theta_k^{i_h}}^{i_h}(a^{i_h}|o^{i_h})} M^{i_{1:h}}(s, \boldsymbol{a})$ , unless  $h = n$ .
20: end for
21: end for
F Experiment Environment Introduction
In this section, we introduce the environments that we adopt in the experiments.
(b)
Figure 5: Specific tasks in Safe MAMuJoCo. (a): Ant Task: Ant 4x2 with three folding Jagged  $(30^{\circ})$  line walls, (b): HalfCheetah Task: HalfCheetah 2x3 with the moving obstacles, (c): ManyAgent Ant Task: ManyAgent Ant 2x3 inside one folding line walls (corridor width is  $9\mathrm{m}$ ).
F.1 Safe MAMuJoCo
This environment is an extension of MAMuJoCo [38], maintaining the background environment, agents, physics simulator, and reward function. However, in the Safe MAMuJoCo setting, additional obstacles such as walls or pitfalls are introduced, and the environment emits cost with the increasing risk of an agent stumbling upon an obstacle. Here, we mainly introduce the scenarios that we employ in our work and present them in Figure 5.
ManyAgent Ant task & Ant task The corridor in the environment is bounded by two walls, with a width of  $9\mathrm{m}$  for ManyAgent Ant and  $10\mathrm{m}$  for Ant. The environment emits the cost of 1 for an agent, if the distance between the robot and the wall is less than  $1.8\mathrm{m}$ , or when the robot topples over, which can be described as
$$
c _ {t} = \left\{ \begin{array}{l l} 1, & 0. 2 \leq \boldsymbol {z} _ {\text {t o r s o}, t + 1} \leq 1. 0, z _ {\text {r o t}} > - 0. 7, \left\| \boldsymbol {x} _ {\text {t o r s o}, t + 1} - \boldsymbol {x} _ {\text {w a l l}} \right\| _ {2} \geq 1. 8 \\ 0, & \text {o t h e r w i s e} \end{array} , \right. \tag {36}
$$
where  $\mathbf{z}_{torso,t+1}$  and  $\mathbf{x}_{torso,t+1}$  is the robot's torso's z-coordinate and x-coordinate at time  $t + 1$ ,  $z_{rot}$  is the robot's rotation's z-coordinate and  $\mathbf{x}_{wall}$  denotes the x-coordinate of the wall.
HalfCheetah task In these maps, the HalfCheetah agents move inside a corridor (which constrains their movement, but does not induce costs). Concurrently, there are pitfalls within the corridor that also move. When an agent is too close to a pitfall, specifically when the distance between an agent and a pitfall is less than  $9\mathrm{m}$ , a cost of 1 will be emitted.
$$
c _ {t} = \left\{ \begin{array}{l l} 1, & \left\| \boldsymbol {y} _ {\text {t o r s o}, t + 1} - \boldsymbol {y} _ {\text {o b s t a c l e}} \right\| _ {2} \geq 9 \\ 0, & \text {o t h e r w i s e} \end{array} , \right. \tag {37}
$$
where the y-coordinate of the robot's torso is represented by  $\mathbf{y}_{\text{torso},t+1}$  and  $\mathbf{y}_{\text{obstacle}}$  denotes the y-coordinate of the moving obstacles.
F.2 Safe Multi-Agent Isaac Gym
This environment builds upon Issac Gym platform [39], renowned for its GPU-accelerated capabilities, and leverages the powerful Nvidia PhysX engine. Extending from the existing framework of DexterousHands [40], Safe MAIG requires agents to control the robot hands while optimizing both the reward and safety performance. Similarly, we also give an introduction of the specific scenarios in our experimental evaluations.
ShadowHandOver This task revolves around a dual-hand setup, with each hand occupying a fixed position. The primary objective entails the first hand, holding an object, navigating a suitable trajectory to transfer the item to the second hand while the second hand aims to acquire a successful grasp of the object. To be noted, this task incorporates safety constraints pertaining to the range of motion of one of the fingers on the first hand. Formally, the cost function can be expressed as follows:
$$
c _ {t} = \left\{ \begin{array}{l l} 1, & \left\| \boldsymbol {F} _ {a 4, t + 1} \right\| \geq 0. 1 \\ 0, & \text {o t h e r w i s e} \end{array} , \right. \tag {38}
$$
where  $F_{a4,t+1}$  is the first hand's fourth fingers's motion degree.
ShadowHandReOrientation Within the context of this task, both hands are equipped with two items. The fundamental objective for the agents is to execute rotational movements between these two items around each other and the safety constraints remain the same as Equation 38.
Figure 6: Performance comparisons on tasks of Ant 2x4, 2x4d, 4x2 and 8x1. The safety bound is 50, except for Ant 8x1 whose cost threshold is set as 70. The solid line shows the median performance across 5 seeds and the shaded areas correspond to the  $25 - 75\%$  percentiles.
G Performance on Safe MAMuJoCo
In this section, we present the comprehensive results of experiments in Safe MAMuJoCo environment in Figure 6-9. It can be observed that our proposed MAFOCOPS consistently demonstrates superior overall performance across all tasks. Even when our method achieves similar performance compared to the other two algorithms in HalfCheetah scenarios, it still exhibits faster learning, demonstrating the advantages of our approach. As is discussed in the Experiment section, MAPPO-L algorithm always achieves the similar performance as MAPPO, except in HalfCheetah scenarios where the cost threshold is significantly smaller compared to cost achieved by HAPPO and MAPPO. This may be due to that MAPPO-Lagrangian being built upon Lagrangian multiplier combined with standard MARL algorithms, leading to a performance more similar to safety-unaware MARL algorithms. Regarding the other two hard constraint algorithm, their performance would degrade with the increasing number of agents. However, MAFOCOPS consistently outperforms MACPO, proving the effectiveness of our method. What's more, we provide additional videos of the trained policies of both our algorithm and MACPO in supplementary materials.
Figure 7: Performance comparisons on tasks of HalfCheetah  $2 \times 3$ ,  $3 \times 2$  and  $6 \times 1$ . The safety bound is 30. The solid line shows the median performance across 5 seeds and the shaded areas correspond to the  $25 - 75\%$  percentiles.
Figure 8: Performance comparisons on tasks of ManyAgent Ant  $2 \times 3$ ,  $3 \times 2$  and  $6 \times 1$ . The safety bound is 25. The solid line shows the median performance across 5 seeds and the shaded areas correspond to the  $25 - 75\%$  percentiles.
Figure 9: Performance comparisons on tasks of ManyAgent Ant  $2 \times 4$ ,  $4 \times 2$  and  $8 \times 1$ . The safety bound is 80. The solid line shows the median performance across 5 seeds and the shaded areas correspond to the  $25 - 75\%$  percentiles.
H Efficiency Analysis
In this section, we evaluate the training efficiency, which is measured by frame per second (FPS), and the memory cost between MACPO and our MAFOCOPS. To be specific, we record the time and samples spent for each update to calculate average FPS and employ memory monitor tools to track memory utilization after 200000 samples. To ensure a fair comparison, both algorithms are executed on the same GPU device, thereby minimizing the influence of other variables. The results obtained from these evaluations are presented in Table 1 and Table 2 with a precision of two decimal places. Based on the obtained results, it is evident that an increase in the number of agents leads to a noticeable escalation in computational cost for MACPO. Whereas, our algorithm showcases substantial improvement in computational efficiency and demonstrates the ability to effectively conserve memory resources, especially in scenarios involving a larger number of agents.
I Sensitivity Analysis
We test the sensitivity of our algorithm to hyperparameters, i.e.,  $\lambda_{j}$  and  $\nu_{max}$ , as well as the safety bound. To be noted, because the benchmarks that we adopt only involve a single cost, we only need to set one value for  $\lambda_{j}$  and  $\nu_{max}$ . In future works, we may explore the performance of our method in
Scenarios	Ant Task	HalfCheetah Task
Config FPS	2x4d	2x4	4x2	8x1	2x3	3x2	6x1
MACPO	231	218	130	73	298	192	106
MAFOCOPS	322	270	160	115	340	229	162
Improvement(%)	39.39	23.85	23.08	57.53	14.09	19.27	52.83
Scenarios	ManyAgent Ant Task
Config FPS	2x3	3x2	6x1	-	2x4	4x2	8x1
MACPO	244	167	98	-	232	135	73
MAFOCOPS	271	249	149	-	253	193	115
Improvement(%)	11.07	49.10	52.04	-	9.05	42.96	57.53
Table 1: Average FPS between MACPO and MAFOCOPS and the bold results demonstrate the improvement brings by our algorithm.
Scenarios	Ant Task	HalfCheeath Taks
Config Memory (MiB)	2x4d	2x4	4x2	8x1	2x3	3x2	6x1
MACPO	18.85	23.60	31.24	66.25	16.54	30.20	52.08
MAFOCOPS	18.97	21.82	24.23	56.99	19.34	27.26	39.15
Saved Memory	-0.12	1.77	7.01	9.26	-2.80	2.93	12.93
Scenarios	ManyAgent Ant Task
Config Memory (MiB)	2x3	3x2	6x1	-	2x4	4x2	8x1
MACPO	25.32	32.64	55.27	-	27.31	38.90	65.02
MAFOCOPS	24.45	30.88	44.38	-	24.62	34.73	60.71
Saved Memory	0.87	1.76	10.89	-	2.69	4.17	4.31
Table 2: Memory cost of MACPO and MAFOCOPS and the bold results demonstrate the memory saved by our algorithm.
environments with multiple costs. We choose several scenarios in Safe MAMuJoCo to conduct the ablation studies.
The sensitivity to the hyperparameters is evaluated across several different values for  $\lambda_{j}$  and  $\nu_{max}$  while keeping all other parameters fixed. For ease of comparison, we normalized the results based on the return and cost achieved by [18], namely if our method yields a return of  $x$  and HAPPO achieves a return of  $y$ , the normalized result is reported as  $\frac{x}{y}$ . The results report the final performance of the models after training for 10 million steps and are showcased in Table 3 and Table 4 with a precision of three decimal places. Given the complexity inherent in multi-agent environments, it is difficult to delineate the correlation between the performance of our method and the hyperparameters  $\lambda_{j}$  and  $\nu_{max}$ . It can be observed that different scenarios have different sensitivity to these hyperparameters from the results. Overall, the reward achieved under different settings is relatively insensitive as even setting these parameters across a broad range only leads to an average degradation of less than  $10\%$ . On the other hand, the cost may be more sensitive to these parameters, highlighting the inherent challenges in ensuring safety guarantees in safe multi-agent reinforcement learning to some degree.
Furthermore, we select some maps to examine the sensitivity of our algorithm to the safety bound. To be mentioned, hyperparameters in this experiment remain unchanged as the main experiments. From the results depicted in Figure 10, we can see that setting the safety bound too high could lead to increased oscillations in cost performance, although it may yield better reward performance. This observation is reasonable since higher safety bounds may allow agents to explore actions with potentially higher returns but less safety. If the safety bound is low enough, the agents will take actions that are definitely safe, leading to less vibration. Whereas, from a global perspective, the effectiveness of our algorithm remains consistent across these different safety levels. In general, we
Ant 2x4	HalfCheetah 2x3	ManyAgent Ant 2x3	ManyAgent Ant 2x4	All envs
λ	Reward	Cost	Reward	Cost	Reward	Cost	Reward	Cost	Reward	Cost
1	0.913	0.212	0.599	0.049	0.889	0.137	0.781	0.082	0.796	0.120
2	0.975	0.188	0.658	0.056	0.946	0.131	0.882	0.212	0.865	0.147
2.2	0.964	0.091	0.668	0.049	0.947	0.090	0.802	0.166	0.845	0.099
3	0.983	0.183	0.699	0.073	0.958	0.059	0.879	0.178	0.880	0.123
5	1.004	0.113	0.694	0.078	0.871	0.070	0.784	0.267	0.838	0.132
Figure 10: Performance comparisons on Ant 2x4, 2x4d, 4x2 with different safety bound.
need to strike a balance between ensuring safety and achieving a good reward performance when applying safe RL algorithms.
Table 3: Performance of MAFOCOPS for different  $\lambda$  and the "all envs" column presents the averaged performance across these four scenarios.
Ant 2x4	HalfCheetah 2x3	ManyAgent Ant 2x3	ManyAgent Ant 2x4	All envs
νmax	Reward	Cost	Reward	Cost	Reward	Cost	Reward	Cost	Reward	Cost
1	1.042	0.074	0.680	0.029	0.981	0.067	0.859	0.328	0.891	0.125
1.3	0.964	0.091	0.668	0.049	0.947	0.090	0.802	0.166	0.845	0.099
2	0.908	0.222	0.627	0.040	0.962	0.137	0.910	0.206	0.852	0.151
3	0.923	0.097	0.579	0.042	0.936	0.077	0.844	0.123	0.821	0.085
5	0.793	0.175	0.606	0.063	1.013	0.102	0.800	0.256	0.803	0.149
∞	0.873	0.209	0.588	0.048	0.959	0.100	0.749	0.146	0.792	0.126
Table 4: Performance of MAFOCOPS for different  $\nu_{max}$  and the "all envs" column presents the averaged performance across these four scenarios.
J Details of Settings for Experiments
The majority of settings have been described in detail in the Experiments section; however, we provide some additional information here. As our implementation is based on the codebase provided by MACPO [24], and thus most hyperparameters remain consistent with their original values. For MAFOCOPS, the Lagrange multipliers, namely  $\lambda$  and  $\nu_{max}$ , we utilize are 2.2 and 1.3, respectively, which can found in Table 3 and 4. For the other two safe MARL algorithms, MACPO and MAPPO-L, we modify the relevant hyperparameters to ensure their compatibility with the safety bound As mentioned in the Experiments section, for the two benchmarks, we adopt distinct hyperparameters
for MAPPO-L in different categories of tasks, as the safety bound is relative to the cost achieved by standard MARL algorithms. However, MAFOCOPS and MACPO both use unchanged parameters, indicating the robustness of these two methods. We present the specific hyperparameters that we use in our experiments in Table 5 (as most parameters are unchanged, we only report the changed ones or unique parameters in our algorithm).
Safe MAMuJoCo	MACPO	MAPPO-L	MAFOCOPS
kl-threshold	0.008	/	0.0125
lambda lagr	/	[0.38a, 0.46b, 0.59c, 0.52d]	/
λ	/	/	2.2
νmax	/	/	1.3
νlr	/	/	0.00005
fraction coef	0.3	/	/
minibatch size	/	/	256
update numbers	/	/	5
Safe MAIG	MACPO	MAPPO-L	MAFOCOPS
kl-threshold	0.009	/	0.01
lambda lagr	/	[0.14a, 0.68b]	/
lagrangian coef rate	/	[1e-7a, 9e-7b]	/
λ	/	/	2
νmax	/	/	1.4
νlr	/	/	0.001
fraction coef	0.26	/	/
minibatch size	/	/	8192
update numbers	/	/	3
Table 5: Different hyperparameters used for MACPO, MAPPO-L and MAFOCOPS. As MAPPO-L employs different hyperparameters, the changed ones are represented in the list. In Safe MAMuJoCo domains, a means Ant tasks, b corresponds to HalfCheetah tasks, c represents ManyAgent Ant 2x3 tasks and d represents denotes ManyAgent Ant 2x4 tasks. In Safe MAIG domains, a represents ShadowHandOver task and b denotes ShadowHandReOrientation task.