UMAP: A Highly Extensible and Physics-Based Simulation Environment for Multi-agent Reinforcement Learning

Authors: Tianyi Hu, Qingxu Fu, Zhiqiang Pu, Yuan Wang, Tenghai Qiu
Primary area: datasets and benchmarks
Keywords: multi-agent reinforcement learning, simulation environment, reinforcement learning

Abstract:
Existing simulation environments in the field of multi-agent reinforcement learning (MARL) either lack authenticity or complexity. The data generated by these environments significantly deviate from the requirements of the real world, hindering the practical application of MARL. To address this issue, we propose Unreal Multi-Agent Playground (UMAP), a highly extensible, physics-based 3D simulation environment implemented on the Unreal Engine. UMAP is user-friendly in terms of deployment, modification, and visualization, and all its components are open-sourced. Based on UMAP, we design a series of MARL tasks featuring heterogeneous agents, large-scale agents, multiple teams, and sparse team rewards.
We also develop an experimental framework compatible with algorithms ranging from 
rule-based to MARL-based provided by third-party frameworks. In the experimental section, we utilize the designed tasks to test several state-of-the-art algorithms. Additionally, We also conduct a physical experiment to demonstrate UMAP's potential in sim-to-real applications, which is a significant advantage due to the high extensibility and authenticity of UMAP. We believe UMAP can play an important role in the MARL field by evaluating existing algorithms and helping them apply to real-world scenarios, thus advancing the field of MARL.

Decision: reject
Reviewer scores: 6, 8, 3, 6, 8, 3

Reviews and discussion:
<thread>
creation date:
1730393433293

thread_type:
official_review

<thread_object>
creator: reviewer_mxde

<content>
summary:
This paper introduces a platform for multi-agent benchmarks to be built on called UMAP. The authors argue that current MARL benchmarks are not grounded enough in real-world physics and propose leveraging Unreal Engine as a platform for building new physics-grounded tasks. They also present HMAP, a python library allowing researchers to quickly build new MARL algorithms and leverage existing ones.

strengths:
Benchmarking and evaluation is an incredibly important and often underappreciated part of the ML community. Multi-agent RL in particular could benefit significantly from new and improved benchmarks that could help guide its research towards more impactful areas and to drive more fundamental/empirical advances.

Releasing a set of benchmarks alongside the experimental pipeline to easily run experiments is a big positive of this paper.

weaknesses:
# Major

## References to other benchmarks
In the related work you list a lot of benchmarks but do not properly outline what is missing from some of these. For example what do you see i missing from GRF that your platform can provide?
No mention of PettingZoo, Pufferfish, JaxMARL, Robocup, or the Hanabi Challenge. 

Unity ML-Agents Toolkit is also not mentioned at all in the paper, which on the surface is extremely similar (RL environments created in a gaming engine). Why are you creating a new platform using unreal engine instead of extending this one?

## Motivations
What exactly are you proposing in this paper? A new benchmark, or a platform that researchers can build their own benchmarks on? A useful benchmark that the community can utilise should be pre-specified and essentially agreed on by as many areas/researchers as possible. Being able to customise things is very helpful, but then you lose the ability to easily compare performance across the community. This paper seems to stop at providing the platform only, and seems to be expecting other researchers to build a benchmark on top. Conversely, if you are proposing a standardised benchmark then a lot more care needs to be taken over the tasks being specified, what they are testing, what evaluation procedures look like, all of which is missing in this paper. The tasks you've provided feel quite scattershot, and after reading the results I am unsure what to take away. Which area needs the most work, what are you presenting as a challenge for the community, why would increased performance in these tasks be helpful?

I would suggest being very clear on what settings you're thinking about when designing this platform. What do you mean by real-world scenarios? How many agents, what kind of tasks, etc. Be specific and provide examples. By making these concrete with a few examples, you can then contrast with other benchmarks and show what is missing to provide motivations for your platform. 

Why are Points (3) and (4) advantages of UMAP? What exactly is helpful about controlling the simulation time flow, can't I record a video of any environment and watch it back slowly? Similarly, what does controllable-speed rendering mean? Are you trading off speed for visual fidelity/resource use?

## Details
Throughout the paper, there is very little mention of observation spaces. What are the inputs for the agents? Are they images or features? How do you intend or envision researchers will adjust this? Is there a particular configuration you want to see used? What about being able to access the internal state of the environment, are you allowing that? If you are proposing a benchmark, these needs to be specified in advance, and their motivations clearly outlined.

# Minor

UMAP is already a well-known method in ML. Why deliberately use it as a name for your platform?

Reference for POMGs is too new.

"...a city full of building..." - buildings?

"...a transnational airspace, or a planetary orbit..." - is the adjective transnational necessary? What do you mean by a planetary orbit? The path a planet could take? Then isn't the map much more than just the orbit?

What is different about metal clash compared to SMAC/SMACv2?

questions:
Some smaller questions, there are others throughout the weaknesses I would like to see addressed particularly on missing references and motivations. 

- Within a team, can different agents have different reward functions? Some more explanation on why a user would want to leverage the concept of team would be helpful. 

- Can you comment on your results in Figure 4 compared to "Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning". 

- What kind of hyperparameter sweep/tuning did you do for the algorithms you benchmarked?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730448041095

thread_type:
official_review

<thread_object>
creator: reviewer_tzcz

<content>
summary:
This paper proposes UMAP which is an extensible, physics-based 3D simulation environment implemented on the Unreal Engine. UMAP offers (1) diverse multi-agent tasks, (2) customizable task design, (3) controllable simulation speeds, and (4) rich, cross-platform rendering capabilities. The authors compare UMAP with existing environments, emphasizing its flexibility and extensibility.

strengths:
- The authors conduct physical experiment to demonstrate the potential of UMAP in bridging the sim-to-real gap. The integration of a motion capture system and communication between UMAP and real-world environments provides a framework validates the efficacy of the proposed system in replicating multi-agent policies from a virtual to a physical setup.
- Built on Unreal Engine, UMAP is highly extensible, providing a flexible foundation for further development and integration with HMAP.
- Based on UMAP, the authors design a series of foundation tasks including heterogeneity, large scale, sparse team rewards, and multi-team. These tasks impose higher requirements on cooperation and competition among multi agents compared to existing tasks.

weaknesses:
- This paper does not explain how the physics simulation in Unreal Engine contributes to the sim-to-real gap or why Unreal Engine was chosen as the simulation software.
- The extensibility comes at the cost of increased complexity, which could hinder adoption among non-expert users or practitioners without a strong technical background.

questions:
- What specific limitations are associated with UMAP when handling large-scale tasks (e.g., with agents over 100)?
- How would the framework perform in dynamic settings where data distributions change over time? Is UMAP adaptable to such scenarios?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730469315548

thread_type:
official_review

<thread_object>
creator: reviewer_1ovj

<content>
summary:
This paper introduces a new simulation environment, Unreal Multi-Agent Playground (UMAP), for multi-agent reinforcement learning, which is designed to overcome limitations in authenticity and complexity present in existing platforms. 
UMAP offers a suit of physics-based 3D environments that involve heterogeneous agents, large-scale simulations, and multiple teams. UMAP consists of a hierarchical five-layer architecture, while the interface layer is exposed to end users to easily customize tasks using Python. Authors also release the MARL experiment framework, HMAP, to facilitate the use of UMAP. Extensive experiments are done on different tasks in UMAP, and results reveal that there is still the need for more advanced algorithms to dominate those tasks. In addition, authors construct a digital-twin of a real-world task in UMAP to demonstrate its sim-to-real ability.

strengths:
1. The paper does a nice job of presenting its ideas clearly and making the technical aspects easy to follow.

2. UMAP tackles an important issue in the field: the struggle to make MARL simulations relevant for real-world applications. By creating a suit of realistic and flexible environments, UMAP makes substantial contribution to pushing MARL closer to real-world tasks.

3. The UMAP framework is solid, and diverse built-in scenarios gives researchers options for testing algorithms under different settings, making UMAP a valuable tool.

weaknesses:
1. The authors did not benchmark the efficiency of the UMAP simulation. Considering that many recent simulation frameworks (e.g., IsaacGym, IsaacLab, SAPIEN) support GPU parallelization, I am curious whether UMAP could potentially offer similar support. Such high-efficiency simulation has been shown to be highly beneficial for training RL policies in single-agent tasks.

2. The experiments on the physical environment in Section 6.4 do not support the authors claim on sim-to-real ability. The authors merely present rendered images from the sim and real environments without providing any results related to policy learning. For instance, they do not show how a policy trained in simulation performs in the real environment or whether there is a strong correlation between the performance of different policies in simulation and their performance in the real environment.

questions:
1. Considering the recent studies on offline MARL, do the authors have any plans to release offline datasets based on the UMAP environment and evaluate the performance of offline algorithms?

2. What would be the workload involved in designing a new environment? For example, if custom maps or objects need to be imported, and their properties set, what steps are required? Specifically, which parts must be modified through the Unreal Engine, and which can be directly configured through the Python interface?

If the authors address the above concerns, I am happy to raise the score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730554768363

thread_type:
official_review

<thread_object>
creator: reviewer_rub6

<content>
summary:
The authors develop UMAP, an unreal-engine based simulation framework for multi-agent reinforcement learning. They create customisable framework and create a number of tasks based on battle scenarios for evaluating MARL algorithms.

strengths:
The paper does have some strengths:
- The authors develop a new benchmark for multi-agent RL. This is an area with prominent evaluation problems [1] and hence more work on benchmarking is warranted and helpful. 
- The use of unreal engine in a benchmark is nice -- it provides better physics simulation than most commonly used multi-agent RL environments.

weaknesses:
However, I have a number of both technical and ethical considerations that I strongly believe should preclude this paper from publication at this conference. I describe my ethical concerns in a separate section and here focus on the paper's technical flaws. 

- The arguments that the authors make about MARL benchmarking in the introduction are poor. They claim that MARL benchmarks lack "complexity" and "authenticity" but they don't ever define these terms. Throughout they remain vague reasons to discredit prior work. For example, they claim that SMAC is "authentic" while MPE is not, but recent reimplementations such as SMAX [2] effectively reduce SMAC to a particle environment. Why is MPE not authentic while SMAC is? Is SMAX less authentic than SMAC? Similarly they criticise a lack of complex decision making, but at no point do they demonstrate that their benchmark requires, or even define what this means.
- The introduction also argues that SARL methods performing better than MARL methods implies that algorithms are specialised to environments and struggle to transfer to the real world. The second half of this may be true, but the first does not support it at all! A more realistic interpretation of the first part is that independence is a very strong baseline which multi-agent RL is yet to meaningfully overcome in these types of tasks. 
- In line 58 the authors claim that data generated from current benchmarks cannot capture the complexity of real multi-agent scenarios. This is extremely vague and the authors do not back this statement up with any further evidence.
- Their table comparing different MARL environments is also missing some related work, particularly more recent frameworks which have been implemented in JAX [2,3]. JAXMARL, for example, as a framework meets almost all the requirements listed in their table. 
- For the flag capture MAPPO environment, they do not seem to clearly explain how the MAPPO agents were trained, or whether they are co-training with the algorithm being evaluated.
- The results in Figure 4 do not follow best evaluation practice [1, 4], which makes the resulting analysis very difficult to believe. For example, they claim that different algorithms do best in different scenarios, but they do not evaluate the aggregate performance! Their likely is an overall best algorithm if you examine, e.g. inter-quartile means calculated with bootstrapped confidence intervals.

questions:
See weaknesses

[1] Gorsane, Rihab, et al. "Towards a standardised performance evaluation protocol for cooperative marl." Advances in Neural Information Processing Systems 35 (2022): 5510-5521.

[2] Rutherford, Alexander, et al. "Jaxmarl: Multi-agent rl environments in jax." arXiv preprint arXiv:2311.10090 (2023).

[3] de Kock, Ruan, et al. "Mava: a research library for distributed multi-agent reinforcement learning in JAX." arXiv preprint arXiv:2107.01460 (2021).

[4] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730647103343

thread_type:
official_review

<thread_object>
creator: reviewer_cwdv

<content>
summary:
Overall Summary:
- Proposes and provides open source implementation of
	- UMAP - new MARL environment that combines features of multiple other environments into one (Table 1)
	- HMAP - experimental framework that has implementations of popular MARL algorithms (to be paired with UMAP)
	- Experimental results on the application of 7 popular MARL algorithms on UMAP environments
	- Sim to real interaction between UMAP and real world hardware and scenarios. 
	

UMAP Summary:
- Unreal engine base, but users interact with higher layers (state, action, observation etc.)
- Tasks and scenarios:
	- Metal clash - cooperative/competitive - two teams battle in a team elimination type game
	- Tower challenge - full cooperative - agents work together to destroy a central objective
	- Flag capture - cooperative/competitive - multiple teams aim to capture a single flag
	- Landmark conquer - cooperative/competitive - attack and defend with two teams. 
- All scenarios have fully editable parameters including but no limited to: team size, number of teams, health and other attributes on each player within a team

HMAP Summary: 
- An entire experimental framework. 
- contains implementations of popular MARL algorithms
- Integrates UMAP as well as other popular environments (StarCraft, multi-particle, gym etc) with popular MARL algorithms by providing training scripts
	- Authors provide a description of their "glue module" system for training.
	- Looking at the appendix D, the glue module appears to make training extremely user friendly by allowing easy assignment of different algorithms to different tasks and teams. 

Experimental Results summary:
- Interpreted this section as a proof of concept for UMAP and HMAP
- Every task has its own best algorithms
- Biggest takeaway is that the UMAP scenarios are learnable by popular MARL algorithms
- Second biggest takeaway is the demonstration that HMAP experimental framework works well in training popular algorithms on UMAP

Sim to real summary:
- Proof of concept for bridging simulation training to real time execution. 
- Agents trained on UMAP and then controlled objects in real-world scenarios. 
- Created a system by which the physical hardware communicates with UMAP to update UMAP's internal environment, and then UMAP and HMAP communicate to form a decision. The decision is then passed back to the physical hardware to execute an action.

strengths:
Originality:
- Roots its contributions very well in previous works. Provides the reader with a sense of improvement and expansion of the field. 
- Takes positive benefits from contemporary environments and combines it into a single environment.
- Introduces time-flow speed that should aid in the speed up of training MARL algorithms, as well as the debugging process. 
	
Quality:
- Overall high quality writing.
- Figures and tables clearly support the claims made in the paper. 
- Committed to open-source work and development after review process
	
Clarity:
- Ideas conveyed clearly and concisely
- Hyperparameters provided in a clear manner
- Diagrams and figures detailing UMAP, results on UMAP, and glue module are intuitive
	
Significance:
- Provides both the UMAP environments and scenarios as well as a training framework for them, HMAP
- HMAP framework also works for previous environments as well. 
- Time in UMAP (Appendix C) and time flow control. Separates real time flow and simulation time flow and gives control to user for different purposes. In training, user will likely speed up simulation time to address a major issue in MARL, the need for a lot of data collection
- User can also slow down simulation time for debugging. Very useful feature overall

weaknesses:
Weaknesses:
- UMAP and HMAP did not appear to support agent to agent communication. This notion is available in previous environments such as MPE. As teams get larger, it becomes less feasible to use a central controller for all players on a team and thus communication between individual agents would become more important

- the diversity of tasks is quite limited in the current pool of UMAP scenarios. Would enjoy seeing a more diverse set of environments outside of largely combat based team tasks. Examples of this include but are not limited to warehouse management scenarios, hide and seek, and other non-combat scenarios primarily those missing physics simulation environments

- Training time will differ from hard ware to hardware, but it would be good to see how long K episodes of training in a UMAP scenario takes versus K episodes of training in a StarCraft environment (assumes episode lengths are equal)

	- Providing such a table looking at run times with the same hardware in an appendix section can give potential users a sense of the speed up or slow down UMAP provides. 

	- Along a similar vein, it would be nice to have a table or figure exploring how much real time is saved by utilizing the fast time dilation factor within UMAP scenarios. 

	- All in all, I recognize that the paper has made strides towards speeding up training appropriately for different hardware, but I have little idea on the expected training time compared to that of other environments from the paper itself.

questions:
My questions are directly related to certain points from the Weaknesses section:
- Given any fixed hardware, how slow or fast does it take to train an agent in UMAP as it does another physics-engine environment, such as SMAC?

- Are there plans to include more diverse types of scenarios in UMAP outside of health based combat scenarios?

- How much time does the time-dilation feature save in training a fixed agent, such as MAPPO?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730726446853

thread_type:
official_review

<thread_object>
creator: reviewer_1wxb

<content>
summary:
The paper introduce Unreal Multi-Agent Playground (UMAP), a 3D simulation environment for multi-agent reinforcement learning, built on Unreal Engine. UMAP addresses existing limitations by offering a flexible platform for complex tasks and research involving diverse agents and teams. Alongside UMAP, they present the Hybrid Multi-Agent Playground (HMAP), compatible with various algorithms. The paper showcases UMAP's capabilities through tasks, evaluations of MARL algorithms, and a sim-to-real experiment.

strengths:
1. Addressing a Significant Gap: The paper identifies and addresses a critical limitation in MARL research—the lack of simulation environments that are both realistic and capable of modeling complex, large-scale multi-agent interactions. By providing a physics-based environment, UMAP brings simulations closer to real-world scenarios, which is essential for the development and evaluation of practical MARL algorithms.
2. High Extensibility and Customization: UMAP's hierarchical, modular architecture allows users to easily customize tasks at various levels, from high-level configurations to low-level implementations. This flexibility enables researchers to create a diverse range of scenarios tailored to specific research questions or application domains.
3.Support for Diverse and Complex Tasks: The inclusion of scenarios featuring heterogeneous agents, large-scale populations, multiple teams, and sparse rewards demonstrates UMAP's ability to model a wide spectrum of complex MARL problems. This diversity is valuable for testing the robustness and generalization capabilities of MARL algorithms.
4.Open-Source Contribution: By releasing UMAP and HMAP as open-source projects, the authors contribute valuable tools to the MARL community. This openness encourages collaboration, reproducibility, and further development by other researchers.

weaknesses:
1. Depth of Experimental Analysis: The experimental results primarily focus on win rates and rewards. A deeper analysis of algorithm behaviors would provide more comprehensive insights into the challenges posed by the new environment.
2. Comparison with Existing Environments: Although the paper mentions limitations of current MARL environments, a more detailed comparison highlighting specific features and performance metrics would better contextualize UMAP's advantages. Including benchmarks or case studies demonstrating UMAP's superiority could strengthen the argument.
3. Accessibility and Cost of Use: While UMAP is described as user-friendly, potential barriers such as the need for familiarity with Unreal Engine and the complexity of setting up the environment could limit adoption. Providing comprehensive tutorials, documentation, and support could mitigate this issue. In addition, Physics-based environments, especially those built on game engines like Unreal Engine, can have significant computational overheads. Discussion about the performance implications, resource requirements, and optimizations would be beneficial for users with limited computing resources.

questions:
1. Though it is a general platform, what's the special strength of the new environment built on this platform for evaluating existing MARL algorithms?
2. How much effort should a new developer paid for introducing his own environment/problem into this platform?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733193147990

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

We would like to express our sincere gratitude to all reviewers for your dedicated time, constructive comments, and valuable feedback on our paper. Your suggestions and recommendations have helped us further clarify and strengthen the presentation of our work.

As we approach the end of the discussion phase, we kindly invite you to share any additional questions or concerns you may have. We are committed to addressing all remaining issues and would greatly appreciate your further input in the discussion phase.

We look forward to any additional feedback you may wish to provide.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733313115119

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Rebuttal

comment:
We thank all the reviewers for their valuable suggestions and engagement during the discussion phase. Your time and effort have clarified our work, and we sincerely appreciate the constructive feedback.

We are encouraged by the reviewers' appreciation that **UMAP addresses a significant gap in the MARL domain by incorporating Unreal Engine for improved physical simulation conditions** (1Wxb, rU86, 1ovJ), that **UMAP is highly scalable** (1Wxb, tZCz) and **supports a diverse range of task simulations** (1Wxb, 1ovJ, tZCz), that the collaboration between HMAP and UMAP, as well as **HMAP's compatibility with other frameworks, is very useful** (cWdv, MxdE), that **real-world experiments have shown UMAP's potential in sim-to-real applications** (tZCz), and that **our series of open-source plans will benefit the community** (1Wxb, cWdv). In terms of manuscript writing, we also thank the reviewers for praising the **high quality of our writing** (cWdv, 1ovJ), the clarity of our viewpoints, and providing **detailed experimental settings and clear charts** (cWdv).

To help address the concerns of the reviewers and facilitate further discussion, we have revised the PDF version of our manuscript. The revisions include: 
1. Incorporating additional related work on MARL simulation environments, refining the presentation of UMAP's unique advantages, and enriching the details in Table 1. 
2. Offering a detailed discussion on various related works in the appendix, clearly identifying features lacking in each work compared to UMAP. 
3. Expanding the appendix to include a detailed introduction to UMAP's built-in scenarios. 
4. Expanding the appendix to include a discussion on UMAP's training efficiency and resource consumption. 
5. Introducing a checklist for ethical evaluation of the paper in the appendix. 
6. Refined the expression of certain details within the paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents UMAP, a physics-based simulation environment built on Unreal Engine for multi-agent reinforcement learning. 
The paper suffers from fundamental positioning ambiguity (unclear benchmark vs platform focus), methodological shortcomings (insufficient comparative evaluation, weak validation of sim-to-real claims), and inadequately substantiated arguments against existing MARL frameworks. The empirical results fail to demonstrate compelling advantages over existing solutions, while concerns persist about accessibility and adoption barriers due to implementation complexity. These critical limitations, highlighted particularly by Reviewers rUB6 and MxdE, suggest the work requires substantial revision to meet acceptance standards.

Given these limitations, I recommend rejection with encouragement to address these issues in a future submission.