SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Scenarios An ideal MARL benchmark must provide a high degree of flexibility, enabling users to construct custom scenarios that encompass a diverse range of configurations. This ranges from simple one vs one setups to expansive environments involving thousands of potentially heterogeneous agents. Gigastep, embraces this principle by offering a flexible platform for custom scenario generation. It allows users to craft unique maps, define custom agent types, build teams of size one to over a thousand, and specify multiple objectives, among other customizations. This robust, user-oriented framework encourages exploration across the vast design space inherent to MARL problems, promoting creativity, adaptability, and thoroughness in the pursuit of MARL advancements.
4 Environment Performance
In this section, we run a performance evaluation to test the rollout speed of Gigastep. In particular, we run the environment in a closed loop with a neural network policy using jax.jit and jax.vmap while increasing the batch size of the vmap dimension. We repeat this experiment on several commonly used consumer GPUs and for both observation space types, i.e., feature vector and RGB image. The results are shown in Figure 3. They demonstrate that a sufficient amount of experience data can be generated with Gigastep on affordable hardware within a short runtime.
5 MARL Baselines with Gigastep
To evaluate the developed environment for multi-agent reinforcement learning research, we adopted a handful of MARL algorithms such as multi-agent evolutionary strategies and multi-agent reinforcement learning algorithms. We discovered the emergence of a diverse set of behaviors using the proposed methods, validating the suitability of Gigastep for MARL research. We want to highlight that our objective is to validate Gigastep and not to run a comprehensive comparison of different MARL algorithms. Additional details and results of multiple MARL baselines can be found in the supplementary materials.
Self-play has shown to be the most effective method to train agents in adversarial settings [56]. We, therefore, adopted self-play in the following algorithms, but keep the policy parameters of both teams separated to allow asymmetric team structures.
Figure 3: Performance of Gigastep in a 2-vs-2 agent scenario. Number of agent steps that can be computed within a second.
Multi-agent PPO. We tested multi-agent proximal policy optimization (PPO) [72] using the standard PPO implementation [52] with centralized value function  $V_{\phi}(S) = f(V_{\phi_i}(s_i)_{i=1,\dots,n})$  and decentralized action policy  $a_i = \pi_i(s_i)$ . During training, the value function was centralized and trained with the composition of the individual value functions, where each individual value function  $V_{\phi_i}$  uses local observation from individual agent  $s_i$ . During the execution phase, the action policy  $\pi_i(s_i)$  is fully decentralized. The individual value function  $V_{\phi_i}$  and  $\pi_i$  share the same structure and parameters among each agent. In Figure 4 we visualize example behaviors that emerged during the MARL training procedure on some Gigastep environments.
Multi-agent evolution strategy. We interfaced Gigastep with evosax [24] to facilitate streamlined training of various evolution strategies. This formulation enables scanning through entire episodes, further reducing communication overhead and runtime requirements by only updating based on episode returns. We specifically evaluate multi-team OpenAI-ES [48], separable NES [68], and Discovered Evolution Strategy [26], which all successfully learn diverse behaviors in Gigastep.
6 Benchmark Challenges on top of Gigastep
Evaluation is one of the issues of MARL for Dec-POMDP, especially in an adversarial setting and self-play. Specifically, reward is only defined relative with respect to a particular adversary policy, i.e., an old copy of the policy in case of self-play, but not an absolute measure as found in single-agent RL.
MOBA or board games such as chess solve this issue by using an Elo-based [16] ranking system, which defines a numerical skill ranking over a set of players or policies.
A key assumption required to adapt such an Elo-ranking for Gigastep is a large pool of fixed policies that reflect a wide range of skill levels. To obtain this set, we define manually coded and MARL-trained baseline policies for Gigastep. The manually coded policies implement only elementary behaviors, such as a team moving in circles. We observed that such simple behavior already poses a decent baseline due to two properties: First, the circling behavior strictly outperforms random policies due to random agents eventually crashing at the boundary of the environment while the circling team moves on a stable orbit indefinitely. Second, for tracking objectives, the view of a circling team can cover a large part of the maps and simultaneously implement some rudimentary evasion strategy. Consequently, the performance of a policy against our manually coded baseline allows us to measure if the policy has learned the objective of the task beyond random behavior. For instance, in Figure 5a, we show the progress of the win rate of multi-agent PPO-trained policies against the manually coded baselines. At the beginning of the training, we see that the manually coded agents strictly outperform the RL agents at an 80:20 win-loose ratio. Nonetheless, after a certain amount of training iterations, the RL agents are able to win nearly all engagements with the coded baselines.
Our second baseline policy set is neural networks trained via multi-agent PPO and self-play. To obtain policies across diverse skill levels, we save checkpoints throughout the training process, resulting in
A) Line formation
B) Climbing (gaining potential energy)
C) Encircling
D) Grouping
Figure 4: Visualization of some examples of the team behavior emerging during the MARL training process on Gigastep environments.
weaker baselines at the beginning of the training and stronger ones toward the end. An example of a MARL training progress using an Elo-ranking system is shown in Figure 5b.
In the future, we may extend the pool of baseline policies in case advances in multi-agent RL algorithms are able to train much stronger policies that render the current pool inaccurate.
We have provided demonstrative algorithms to train agents for the Gigastep that show the emergence of competitive agents. However, we did not perform extensive tuning for all game configurations. Consequently, we may update certain scenarios if necessary, e.g., a scenario that turns out to be too simplistic or difficult to allow the emergence of complex agent behaviors.
Additionally, we want to distinguish between Gigastep as a research platform to create scalable and complex MARL environments and the built-in benchmark challenge defined in Section 6. The flexibility and openness of Gigastep allow future research to create additional challenges with Gigastep, including custom scenarios, other objectives, and different agent capabilities and dynamics.
Finally, while Gigastep provides a wide range of objectives and other environmental parameters, we highlight that all evaluations with Gigastep share common agent dynamics. This can potentially
Wining rate during training
Figure 5: Agent performance over the course of a MARL learning process with Gigastep.
introduce bias into MARL experimentation if Giagstep is used as the sole evaluation platform. Consequently, we note that proper MARL experimentation requires a diverse set of environments for the robustness and transferability of the observed results.
In this work, we introduced Gigastep, a novel, fully vectorizable multi-agent reinforcement learning (MARL) environment designed in JAX. Gigastep distinguishes itself by its ability to carry out up to one billion steps per second on consumer-grade hardware, a feature that significantly broadens the accessibility of MARL research. This environment encompasses complex, high-dimensional domains, integrating 3D dynamics, stochasticity, and partial observability. It also supports both collaborative and adversarial tasks, making it a comprehensive platform for a wide range of MARL applications.
Our experiments demonstrate that Gigastep can play a pivotal role in the development of MARL algorithms with high efficiency. By enabling researchers to conduct advanced simulations on affordable hardware, Gigastep addresses a key challenge in the field - the computational divide that often impedes underrepresented researchers from engaging in cutting-edge MARL research.
As we continue to explore the potential of Gigastep, we anticipate that its capacity to democratize access to MARL research promote a more diverse, inclusive, and innovative scientific community. Future work will focus on extending Gigastep's capabilities, including the integration of more complex task domains and additional functionality to further enhance the user experience and expand the range of feasible MARL experiments. Furthermore, we aim to make use of Gigastep's RGB observation space capabilities for research on human-AI collaboration by adding user-controlled agents along RL agents. We hope that the introduction of Gigastep will inspire new directions in MARL research and contribute to the growth and diversification of our scientific community.
Research was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. The research was also funded in part by the AI2050 program at Schmidt Futures (Grant G-22-63172) and Capgemini SE.
[65] Paul Tylkin, Tsun-Hsuan Wang, Tim Seyde, Kyle Palko, Ross Allen, Alexander Amini, and Daniela Rus. Autonomous flight arcade challenge: Single-and multi-agent learning environments for aerial vehicles. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pages 1744-1746, 2022.
[66] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Dop: Off-policy multi-agent decomposed policy gradients. In International Conference on Learning Representations, 2020.
[67] Peter Werner, Tim Seyde, Paul Drews, Thomas Matrai Balch, Igor Gilitschenski, Wilko Schwarting, Guy Rosman, Sertac Karaman, and Daniela Rus. Dynamic multi-team racing: Competitive driving on 1/10-th scale vehicles via learning in simulation. In 7th Annual Conference on Robot Learning, 2023.
[68] Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. The Journal of Machine Learning Research, 15(1):949-980, 2014.
[69] Wei Xiao, Tsun-Hsuan Wang, Ramin Hasani, Mathias Lechner, Yutong Ban, Chuang Gan, and Daniela Rus. On the forward invariance of neural odes. In International Conference on Machine Learning (ICML), 2023.
[70] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5571-5580. PMLR, 2018.
[71] Lianhao Yin, Tsun-Hsuan Wang, Makram Chahine, Tim Seyde, Mathias Lechner, Ramin Hasani, and Daniela Rus. Cooperative flight control using visual-attention-air-guardian. arXiv preprint arXiv:2212.11084, 2022.
[72] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games, 2022.
[73] Lianmin Zheng, Jiacheng Yang, Han Cai, Ming Zhou, Weinan Zhang, Jun Wang, and Yong Yu. Agent: A many-agent reinforcement learning platform for artificial collective intelligence. In Proceedings of the AAAI conference on artificial intelligence, volume 32, No 1, 2018.
Gigastep is published under the open-source MIT license on Github https://github.com/mlech261/gigastep. Further development and potential updates discussed in the Limitations section will take place on the Gigastep GitHub page.
Accountability framework
Our team is committed to maintaining an open and accountable framework throughout the development of Gigastep. Our work is being carried out with the utmost transparency and we welcome the participation and insights of the broader AI community. The primary objective of our project is to democratize access to efficient and affordable MARL experimentation. To this end, we ensure that every code commit, model modification, and development update is documented and shared publicly, to provide insights into the ongoing development process. We also uphold rigorous software testing standards and frequent internal code reviews to maintain code quality and reliability. Moreover, we actively encourage feedback, constructive criticism, and contribution from other researchers and developers, and have implemented mechanisms for easy reporting of issues and suggestions. By maintaining this level of accountability, we strive to build a resource that genuinely serves the needs of the MARL community and advances the field of AI research.