SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Figure 1, we demonstrate the performance of Algorithm 1 in the 20-agent Pistonball environment under entropy constraints. We observe that, while the truncation with  $\kappa = 3$  converges in fewer iterations, truncation with  $\kappa = 1$  also yields comparable performance. This underscores the efficiency of Algorithm 1 as employing a smaller communication radius can significantly reduce the computation.
Finally, we compare Algorithm 1 with three baselines based on the MAPPO-Lagrangian method [31].
- MAPPO-L: the original algorithm introduced in [31]. Note that each agent has access to global information.
- Decentralized MAPPO-L: decentralized version of MAPPO-L, where each agent only has access to information in the local neighborhood. However, since each agent is trained to greedily maximize its individual reward, its behaviors might sacrifice the performance of other agents.
- Decentralized Aggregate MAPPO-L: decentralized version of MAPPO-L, where we address the aforementioned issue by redefining each agent's reward to be the sum of rewards of all agents in its local neighborhood.
For a fair comparison, we consider two standard safe MARL problems, where both objectives and constraints are shaped by cumulative rewards (see Appendix H.4). The results in Table 1 demonstrate that our method consistently outperforms both the centralized and decentralized variants of MAPPO-Lagrangian. We refer the readers to Appendix H for the comprehensive experimental results that fully answer the three questions raised at the beginning of this section.
In this work, we study the safe MARL with general utilities, with a focus on the setting of distributed training without global observability. To address the challenge of scalability and incorporating general utilities, we propose a primal-dual actor-critic method with shadow reward and  $\kappa$ -hop policy. Taking advantage of the spatial correlation decay property of the transition dynamics, we show that the proposed method achieves an  $\mathcal{O}\left(T^{-2/3}\right)$  convergence rate to the FOSP of the problem in the exact setting and achieves an  $\widetilde{\mathcal{O}}\left(\epsilon^{-3.5}\right)$  sample complexity, with high probability, in the sample-based setting. Finally, the effectiveness of our model and approach is verified by numerical studies. For future research, it would be interesting to develop scalable safe MARL algorithms with adaptive communication of agents' information [51] and intelligent sampling of agents' trajectories.
[88] Jonas Moritz Kohler and Aurelien Lucchi. Sub-sampled cubic regularization for non-convex optimization. In International Conference on Machine Learning, pages 1895–1904. PMLR, 2017.
[89] Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research, 70(4):2563-2578, 2022.
[90] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.