Neural MMO 2.0: A Massively Multi-task Addition to Massively Multi-agent Learning
Authors: Joseph Suarez
David Bloomin
Kyoung Whan Choe
Hao Xiang Li
Ryan Sullivan
Nishaanth Kanna Ravichandran
Daniel Scott
Rose S Shuman
Herbie Bradley
Louis Castricato
Phillip Isola
Kirsty You
Yuhao Jiang
Qimai Li
Jiaxin Chen
Xiaolong Zhu
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: environment; multi-agent; multi-task; reinforcement learning
TL;DR: Train agents to master diverse tasks in Neural MMO 2.0

Abstract: Neural MMO 2.0 is a massively multi-agent and multi-task environment for reinforcement learning research. This version features a novel task-system that broadens the range of training settings and poses a new challenge in generalization: evaluation on and against tasks, maps, and opponents never seen during training. Maps are procedurally generated with 128 agents in the standard setting and 1-1024 supported overall. Version 2.0 is a complete rewrite of its predecessor with three-fold improved performance, effectively addressing simulation bottlenecks in online training. Enhancements to compatibility enable training with standard reinforcement learning frameworks designed for much simpler environments. Neural MMO 2.0 is free and open-source with comprehensive documentation available at neuralmmo.github.io and an active community Discord. To spark initial research on this new platform, we are concurrently running a competition at NeurIPS 2023.

Neural MMO 2.0: A Massively Multi-task Addition to Massively Multi-agent Learning
Joseph Suárez
JSUAREZ @ MIU
Phillip Isola
PHILLIPI@MIT.ED
Massachusetts Institute of Technology
Kyoung Whan Choe
KYOUNK KYOEKGGGGGMAIL.COOM GMAIL.COM
David Bloomin
DAVEEY @ GMAIL.COM
Hao Xiang
HXL23 WXL.23CCAM.AC.UK
Nikhil Pinnaparaju
NICHILPPINMAPARRAJUUGGGMM CMMAILL.AAAAAC
Nishaanth Kanna
NISHAANTHK GMATI..COM
Daniel Scott
DSCOTT45 @ATTGGAAATHHHHI
Ryan Sullivan
RSULLI UMD_EDU
Rose S. Shuman
ROSE.SHUMAN @ BLUUMNIBBOWN.
Lucas de Alcântara
LUCASAGLLEITE CALLLIIIEEGMII.COMM
Herbie Bradley
HB574@CAM.AC.UU
Louis Castricato
LOUIS_CASTRICATO BROWN.EDU
CarperAl
Kirsty You
KIRSTYYOU @ CHAOCANNHUUAL
Yuhao Jiang
Qimai Li
QIMAILI@CHAOCANSHHUAN
Jiaxin Chen
JIAXINCHEN TIXINCHHENCCHACCAAN CAA.HEANNNHHAAAAAUAAAACA
Xiaolong Zhu
NIAOLONGZHU @
Parametrix.Al
Abstract
Neural MMO 2.0 is a massively multi-agent aalti- environment for reinforcement learning research. The key feature of this new version is flexible task system that allows users to define broad range of objectives and reward signals. We challenge researchers train agents capable of generalizing tasks, maps, and opponents never seen during training. Neural MMO features procedurally generated maps with 128 agents the standard setting and support for up Version 2.0 is a complete rewrite of its predecessor with three-fold improved performance and compatibility with CleanRL. We release the platform as free and open-source software with comprehensive documentation available at neuralmmo githuub.o and an active community Discord. To spark initial research on this new platform, we are concurrently running competition at NeurlPS 2023.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

DEFENSE
DAMAGE
Bottom
Shaving
TOP
Scrap
Halmet
Ehard
Gloves
Org
Crystal
+ FOOD
WATER
Rod
Fash
Haeb
USE RESOURCES TO MAKE ITEMS & WEAPONS
HD
USE TOOLS TO GATHER RESOURCES
Arcane
Ration
Pickaxe
Poultica
Chisal
FIGHT EDEMIES TO GAIN ARMOR & TODLS
USE WEAPONS TO FIGHT ENEMIES
DAMAGE
DAMAGE
Melee
DAMAGE
Wand
Renged
3 COMBAT STYLES
LEVEL UP SKILLS TO ACCESS STRONGER ITEMS
BOW
Magic
Sward
TRADE ITEMS ON THE MARKET
Figure 1: Overview Neural MMO 2.0. Users can define tasks to specify a broad range of agent objective. In general, these involve using tools to gather resources, using resources to make items and weapons, using weapons to fight enemies, and fighting enemies to gain armor and tools. Full documentation is available catrraamooggihuu.ii
Novelty and Impact
Neural MMO is reinforcement learning platform first released in 2019 [Suarez et al. 2019), with updates featured in short-form AAMAS 2020 [Suarez et al., 2020 and ICML 2020, and new version published in the 2021 NeurlPS Datasets & Benchmarks track (Sareez al. [021] J. Since then, the platform has gained traction through competitions at IJCAI 2022 and NeurIPS 2022, totaling 3500+ submission from 1200+ users, which significantly improved state-of-the-art on the platform. Alongside these developments, our community on Discord has grown to nearly 1000 members.
While previous versions of the environment defined fixed objectives through only the reward signal, Neural MMO 2.0 introduces flexible task system that allows users to define per-agent or per-team objectives and rewards, expanding the platform's 's applicability to broader range of problems. In particular, Neural MMO 2.0 enables research on generalization, open-endedness, and curriculum learning areas that were difficult to explore with prior versions and which require sophisticated, flexible simulators. There are few any other environments of comparable scope Neural MMO available for these problems.

---

Neural MMO Gameplay, Features, and Mechanics
Health
Water
Items Inventory
Weapons Inventory
Iters
Items
16
Agent
Warrio
Mage
Archer
Gathor Level Prospecting
Level
Melee Range Mage
Prospecting
Cariing
Foatures
Alchemy
Fushing
Herbaaaars
Hunes
Exchange Market
Ranged Aftack
Market
Food
Arrowai
Target
Figure 2: Neural MMO 2.0 features procedurally generated terrain, resources to collect, 3 combat styles, gathering and combat professions train and level up, scripted NPCs that roam the map, and 16 types of items in 10 quality levels including weapons, armor, consumables, tools, and ammunition. An environment-wee market allows agents to trade items with each other.
Practical engineering improvements the core of Neural MMO 2.0. These include:
1. A 3x faster engine. This was developed as part of a complete rewrite of our 5+ year old code base and is particularly important for reinforcement learning research, where simulation is often the bottleneck. For example, the upcoming competition would not be practical on the old engine.
2. Simple baselines with CleanRL, popular and user-friendly reinforcement learning library. CleanRL and most other reinforcement learning frameworks are not natively compatible with environments of this complexity, and previous versions required convoluted, environment- specific compatibility wrappers. Neural MMO 2.0 integrates PufferLib to solve this problem.
3. A web client available at neuralmmo, giithub.ioociiii generously open-sourced by Parametrix.Al. This client offers improved visualization capabilities and eliminates setup requirements.
Additionally, the platform's 's documentation has been professionally rewritten consultation with the development team. This, along with a more intuitive and accessible website layout, marks significant step towards improving user engagement. collection of papers detailing previous versions and competitions is available on carralmmo.githbbio
2 Neural MMO 2.0
Neural MMO (NMMO) is an open-source research platform that is computationally accessible. It enables populations of agents be simulated in procedurally generated virtual worlds. Each world features unique landscapes, non-playable characters (NPCs), and resources that change each round. The platform draws inspiration from Massively Multiplayer Online games (MMOs), which are online video games that facilitate interaction among a large number of players. NMMO is a platform for intelligent agent creation, typically parameterized by neural network. Agents in teams must forage for resources to stay alive, mine materials to increase their combat and task completion capabilities, level up their fighting styles and equipment, practice different professions, and engage in trade based on market demand.

---

In the canonical setting of NMMO that will support the upcoming competition, users control 8 out of total of 128 simulated agents. The ultimate goal is score more points by completing more tasks than the other 118 agents present in the same environment. Originally, we planned to introduce team-based tasks and objectives, but we decided to postpone the introduction of these given the practical limitations of learning libraries. After the conclusion of the competition, top submissions will be provided as baseline opponents. NMMO includes the following mechanisms to induce complexity into the environment:
Terrain: Navigate procedurally generated maps
Survival: Forage for food and water to maintain your health
NPC: Interact with Non-Playable Characters of varying friendliness
Combat: Fight other agents and NPCs with Melee, Range, and Magic
Profession: Use tools to practice Herbalism, Fishing, Prospecting, Carving, and Alchemy
Item: Acquire consumables and ammunition through professions
Equipment: Increase offensive and defensive capabilities with weapons and armor
Progression: Train combat and profession skills to access higher level items and equipment
Exchange: Trade items and equipment with other agents on a global market
A detailed wiki is available on the project's document site.
3 Background and Related Work
In the initial development phase of Neural MMO from 2017 to 2021, the reinforcement learning community witnessed the release of numerous influential environments and platforms. Particularly noteworthy among these are Griddly [Bamford et al. 2020). NetHack (Küttler et al., 2020). and MineRL [Guss et al. 2021]. A comprehensive comparison of these with the initial Neural MMO can be found in our previous publication [Suarez et al. 2021). The present work primarily focuses on recent advancements in the reinforcement learning environments sphere. Griddly has sustained ongoing enhancements, while MineRL has inspired several competitive initiatives. Since 2021, only few new environments have emerged, with the most pertinent ones being Melting Pot (Leibo et al., 2021]. and XLand [Team et al., 2021]. Melting Pot and its successor, Melting Pot 2.0 [Agapiou al. ) comprise many multiagent scenarios intended for evaluating specific facets of learning and intelligence. and and its sequel, XLand 2.0 [Team et al.] 2023), present large-scale projects focusing training across varied curriculum of tasks within procedurally generated environment, with a subsequent emphasis on generalization to novel tasks. Compared to Melting Pot, Neural MMO is larger environment with flexible task specifications, scenarios. XLand, while architecturally akin to Neural MMO, predominantly explores two-agent settings, whereas Neural MMO typically accommodates 128. A crucial distinction is that XLand is primarily a research contribution enabling the specific experiments presented in the publication. does not provide open-source access and not computationally practical for academic-scale research. Conversely, Neural MMO is an open-source platform designed for computational efficiency and fser-friendllinss.
opposed to set of individual
4
Task System
The task system of Neural MMO 2.0, a central component of the new version, comprises three interconnected modules: GameState, Predicates, and Tasks. This system leverages the new Neural MMO engine to provide full access to the game state in a structured and computationally efficient manner. This architectural enhancement surpasses the capabilities of Neural MMO I.x, allowing users to precisely specify the tasks for agents, paving the way for task-conditional learning and testing generalization unseen tasks during training.
4.1 GameState
The GameState module acts as high-peeformance data manager, hosting the entire game state in flattened tensor format instead of traditional object hicrarchies. This vectorization serves dual

---

purpose: first, it accelerates simulation speeds a crucial factor in generating data for reinforcement learning; and second. it offers researchers an efficient tool to cherry-pick the required bits of data for defining objectives. While this format was originally inspired by the data storage patterns used in MMOs, adaptations were needed to support the computation of observations and definition of tasks.
Alongside GameState, we also introduced auxiliary datastores to capture event data-unique unique in-game occurrences that would be not be captured otherwise. These datastores record things that happen, such as when an agent lands successful hit on an opponent or gathers a resource, rather than just the outcomes, i.e. damage inflicted a broader range of objectives in computationally efficient manner.
change in tile state. Events enable the task system encompass
To illustrate the flexibility provided by GameState access, let's walk through some representative query examples. The snippets in the GameState Appendix employ both the global and agent-specific GameState queries. Global access is useful for game dynamics such time and environmental constants. We also provide a convenience wrapper for accessing agent-specific data.
This query API gives researchers direct access the mechanics of the game environment, offering rich playground for studying complex multi-agent interactions, resource management strategies, and competitive and cooperative dynamics in reinforcement learning context.
4.2 Predicates
The Predicates module offers a robust syntax for defining completion conditions within the Neural MMO environment.
Predicates interface with the game state (the "subject") to provide convenient access to agent data and any additional arguments desired. Predicates return a float ranging from 0 to rather than boolean. This design choice supports partial completion of predicates- crucial for generating dense reward functions-whie still allowing tasks to be considered complete when the return value equals 1. As a starting point, Neural MMO offers 25 built-in predicates that can access every aspect of NMMO. The first example in the Predicates Appendix illustrates the creation of more complex objective, building on the game state and subject from the previous section.
The second example in the Predicates Appendix demonstrates how the Predicate system can be used to articulate complex, high-level objectives. The FullyArmed predicate demands that a specific number of agents in team be thoroughly equipped. An agent is considered fully equipped if it has an entire set of equipment (hat, top, bottom, weapon, ammo) of a given level. To acquire a complete equipment set, agents would need utilize various professions in different locations on the game map, which could take several minutes to accomplish. This task's 's complexity could be further amplified by setting condition that each team member be outfitted specifically with melee, ranged, or magical equipment, necessitating the coordinated use of all eight professions.
4.3 Tasks
The Task API allows users to formulate tasks by combining predicates and assigning per-agent rewards based on the outcomes of intermediary predicates. This approach not only maintains an account of tasks completed but also provides denser reward signal during training. We expect that most users will form tasks using the library of pre-built predicates. For advanced users, direct access to GameState enables mapping conditions on the game's 's internal variables to rewards, circumventing the need for intermediate predicates. The predicate can then be turned into task. See the Tasks Appendix for an example.
5 Performance and Baselines
Neural MMO 2.0's new engine runs at approximately 3,000 agent steps per CPU core per second, up from the approximately 800 to 1,000 in the previous version. design focuses on native compatibility with a vectorized datastore that represents game state. This allows us to keep the environment in Python while maintaining efficiency, providing easier access for researchers looking to modify or extend Neural MMO.
5

---

Simulation throughput is highly dependent upon agent actions within the game. We compute statistics by having agents take random actions, but to maintain a fair estimate, we eliminate mortality since dead agents do not require any computation time. Given that NMMO equates one action to 0.6 seconds of real time, a single modern CPU core can simulate at 5,000 times real-time per-agent, equivalent to 250M agent steps or roughly 2.5 terabytes of data per day at approximately 10 KB per observation.
We also release a baseline model with training code and pretrained checkpoints. Compared to the previous TorchBeast [Küttler al. 2019 baseline, our new model builds on top of CleanRL. This is simpler library that is much easier to work with, but it is not designed to work with complex environments like Neural MMO by default. To achieve interoperability, we integrate with PufferLib, a library designed to streamline the various complexities of working with sophisticated environments.
6 Limitations
Despite its enhancements, Neural MMO 2.0 does not incorporate any novel game mechanics absent in version 1.x. However, in the most recent competition, even the top approaches did not learn to comprehend and utilize all of the game systems, and there is substantial room for improvement. Moreover, agent specialization within a team remained limited. These circumstances are likely attributable to the overly broad survival objective that invariably promotes dominant strategies, posing a challenge to balance. However, with the introduction of a more flexible task system in Neural MMÒ 2.0, we redefine performance the capability to execute novel tasks, thereby enabling researchers to harness the existing game mechanics in a way not feasible in earlier versions.
7 Accessibility and Accountability
Neural MMO has been under active development with continuous support for the past 6 years. Each of the six major releases this period was accompanied by comprehensive documentation updates, guarantee of timely user support, and direct access to the development team via through the community Discord. The project will continue to support and maintenance. A fourth competition has been accepted to NeurlPS 2023 and is expected to improve the current bascline. The code for this project is hosted in perpetuity by the Neural MMO GitHub organization under the MIT license. We provide both pip package and a containerized setup including the baselines. Documentation is consistently available on neuralmmo, github. io with no major outages recorded to date. The entire project is available as free and open-source software under the MIT license.
Neural MMO implements the standard PettingZoo (Terry et al. 2021) ParallelEnv API, a direct generalization of the OpenAl Gym [Brockman et al. 2016) API for multi-agent environments. Our baselines utilize CleanRL's (Huang et al. 2021] Proximal Policy Optimization (PPO) (Schulman) et al. 2017] implementation, one of the simplest and most widely used reinforcement learning frameworks, with all algorithmic details encapsulated in a single file of approximately 400 lines. While CleanRL was originally designed for simpler environments like single-agent Atari (Bellemare ct al. [0122] games, Neural MMO extends its capabilities through PufferLib, which provides native compatibility through a multiagent vectorization backend. The details of this library are available at gufferai.github...
8 Ethics and Responsible Use
Neural MMO is an abstract game simulation featuring systems of combat and commerce. These elements are incorporated for visual interpretability and are not representative of any actual violence or commerce systems. We are confident that these systems are sufficiently removed from their real-world counterparts that Neural MMO would not be a useful training platform for developing such systems. The use of game- like elements in Neural MMO is a deliberate choice to align with human intuition and does not reflect any specific real-world scenario. Neural MMO's primary objective is to facilitate research on understanding and advancing the capabilities of learning agents. The project does not include any real-world human data other than the code and documentation voluntarily submitted by contributors and some 3D asset files commissioned at fair market rate.

---

Conclusion
Neural MMO 2.0 significant evolution of the platform. We invite researchers to tackle new challenge in generalization across unseen new tasks, maps, and adversaries. Furthermore, we have achieved sigmificant advancements in computational efficiency, yielding a performance improvement of over 300%, and have ensured compatibility with popular reinforcement learning frameworks like CL. This opens up the potential for broader utilization by researchers and makes the environment significantly more accessible, especially to those working with more modest computational resources. Neural MMO has five-year history of continuous support and development, and we commit to maintaining this support, making necessary adaptations, and facilitating lively and active community of users and contributors. With the concurrent NeurlPS 2023 competition, we look forward to sparking new research ideas, encouraging scientific exploration, and contributing to progress in multi-agent reinforcement learning.
Acknowledgements
Training compute for baselines provided by Stability AI, Carper AI, and Eleuther AI. Development for 2.0 was an open-source project under CarperAl led by Joseph Suarez and managed by Louis Castricato. Web client by Parametrix.Ar with artwork by Lucas de Alcântara. Technical documentation by Rose Shuman in collaboration with the development team. Engine work for 2.0 by David Bloomin. Special thanks Kyoung Whan Choe for major contributions development and ongoing environment support. Original project by Joseph Suarez.
This work was supported in part by ONR MURI grant N00014-22-1-2740.
References
John P. Agapiou. Alexander Sasha Vezhnevets, Edgar A. Duéñezz-Gzzite Jayd Matyas, Yiran Mao, Peter Sunehag, Raphael Köster, Udari Madhushani, Kavya Kopparapu, Ramona Comanescu, DJ Strouse, Michael B. Johanson, Sukhdeep Singh, Julia Haas, Igor Mordatch, Dean Mobbs, and Joel Z. Leibo. Melting pot 2.0, 2023.
Chris Bamford, Shengyi Huang, and Simon M. Lucas. Griddly: A platform for Al research in games. CoRR, abs/201 2020. URL htttt:tttaa/iiiiii org/ ht///bb//2200 2011 06363
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning en- vironment: An evaluation platform for general agents. CoRR, abs//207 2012. URL http://arxiv.. wwt.aaawwvv...aaaaaaaaaaee org/abs/ 1207 4708
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016. http://///rrxx org abs///b///11 1606. 01540 cite arxiv:100.011500.
William H. Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noboru Kuno, Stephanic Milani, Sharada Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin, Manuela Veloso, and Phillip Wang. The minerl 2019 competition sample efficient reinforcement learning using human priors, 2021.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, and Jeff Braga. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. CoRR, abs/2111.08815 2021.
08819
Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. Torchbeast: A pytorch platform for distributed RL. CoRR, abs/1910.03552, 2019, URL http://aara wwtp://w.wrr///////////// 1390000552110000
Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment, 2020.
Joel 7.. Leibo, Edgar A. Duzmez-Guzmán, Alexander Sasha Vezhnevets, John Agapiou, Peter Sunehag, Raphael Koster, Jayd Matyas, Charles Beattic, Igor Mordatch, and Thore Graepel.
7

---

Scalable evaluation of aglti reinforcement learning with melting pot. CoRR, abs/2107.06857. 2021. URL battps://aree http://arri...o..aaa////////////
wtwpg//aba//1000 / 06857
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347,, 2017. URL /atpp:////arx ortpsv.v..rr/////77777777
06347
Joseph Suarez, Yilun Du, Phillip Isola, and Igor Mordatch. Neural mmo: massively multiagent game environment for training and evaluating intelligent agents, 2019.
Joseph Suarez, Yilun Du, Igor Mordach, and Phillip Isola. Neural mmo v1.3: A massively multiagent game environment for training and evaluating neural networks. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '20, page 2020-2022, Richland, SC, 2020. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 7781450375188
Joseph Suarez, Yilun Du, Clare Zhu, Igor Mordatch, and Phillip Isola. The neural mmo plat- form for massively multiagent research. In J. Vanschoren and S. Yeung, editors, Proceed- ings of the Neural Information Processing Systems Track on Datasets and Benchmarks, vol- 1, 2021. URL
proceedingeeeeeooooooeeee.r..sse www.nnerrii.. meurips.cc/papen. Paper round1.
Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmice Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei Zhang. Kluman-timeseee adaptation in an open-ended task space, 2023.
Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, Nat McAleese, Nathalie Bradley-Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes-Fitt, Valentin Dalibard, and Wojciech Marian Czarnecki. Open-ended learning leads to generally capable agents, 2021.
J. K. Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis Santos, Rodrigo Perez, Caroline Horsch, Clemens Dieffendahl, Niall L. Williams, Yashas Lokesh, and Praveen Ravi. Pettingzoo: Gym for multi-agent reinforcement learning, 2021.
A Game State
True if any agent in subject any (tile_type index in t for t in subject.
tile of tile_ type
obs.. obs www.b..iil..aeeaaaaaaaeaaaaaaaae material_idt
True if all subjects are alive. np. count_ monzerossubjeeceeooo.o health health 0) (subjeeect)
Computes the mazimum i-inf distance between teammates current _iitttdditt-tte max (subject. row max () subjectt. row. min(), subject.col col. max subject col min())
Tracks hits scored with specific combat style hits = subject. event SCORE_ HIT. combat_style == combat_style SKILL ID eeeeecteeveet...
Computes the summed gold of all teammates subject gold. sum).
Evalutaes 1 if the current game tick the specified tick gs carrent_tick num_ticc
8

---

B Predicates
B.1 Example 1
Signature for predicates def predicate(gs: GameState, subject: Group, **kwargs) -> float:
def DistanceTraveledgga GameState, subject: Group, dist: int):
"""True if the summed l-inf distance between each agent current pos and spawn pos is greater than or equal to the specified _dist. not any (subjecct health 0):
return 0
r, с subject... row, subject. col
dists utils 1inf (list (1ist(zip(r,c)) [gs spasn_ [id_]
for id_ in subject. entity.ide id]) return max (min (dists sum() / dist, 1.0), 0.0)
B.2 Example 2
def (gs: GameState, subject: Group,
combat_style: unmo_skill CombatSkill, level: int, num_agent: int):
"""True if the number of fully equipped agents >= num agent
To determine fully equipped, compare the levels of the hat, top, bottom, weapon, ammo to .leve. "*"
WEAPON_IDS IDS =
nmmo_skinn Melee.Meeee: {'weapon' :5, 'ammo :133, # Spear, Whetstone
nmmo_skill Range: {'weapon's ! :6, ammo :14], # Bow, Arrows
nmmo_skinn Mage: { ('weapon' :7, 'ammo :15] Wand, Runes
item_ids { 1'at':2, 'top':3. :3, 'bottom's :4
item_ ids update (WEAPON_IDE) IDS [combat_styled
lvl_flt (subject item. level >= level) & (subject item. equipped 0)
type_flt np. ippisietsebeect..... itmeect.temm... wtw.iittteetmdd...t....... type_id, aiddliistittem____ ids. values -. peniimmen__nmberee = unique(
subject.iten somer_id[lvl.
type_flt], return_counts-True)
if num_agent == 0:
return 1.0
return max (min (wequieeent_uuuueeeeeeeeeeeeeeee _umubersubeesssee >= len( item_ids items ())) su / num agent, 1.0), 0.0)
C Tasks
def KillPredicateegs: GameState, subject: Group): " """The progress, the maæ of which is 1, should increase small for each p player kill increase big for the 1st and 3rd kills reach 1 with 10 kills
num_kills len (en event. Deont.LLAYED FLAYER_KILL)
progress = num kills * 0.06 num_kills 1:
progress += .1
if num_kills >= 3:
progress += .3
return (progress,s,, 1.0)
This predicate can be turned into task like this.

---

from nmmo task import predicate_api from nmmo. task. group import Group
Create a Predicate class with interfaces to GameState Group pred_cls predicate_ appi wtt.:appp..mmeeeeeeeee (Kk (Kil1)eddicc))))))
Create task for each agent task_list
for agent_id in agent_liss
task_list. (pred_ csujjeet Groop (agent_id)) create_task()
Create a task that evaluates and rewards a whole team together (agent_listt create_task()
team task pred_ cls (subject= (suu_ccsssuujettt
Make task that agent 1 gets rewarded for the agent 2 evaluation paatte_apimmak__peeeeeee maake_preiccae (Al1Dead) fask_for_agent.e cled_cls((
pls predicate_aph
subject=Group (agent_2) .etptt)e)eeee_______________ @reate_task_ae (ass
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We claim only a release of the platform with new features and improvements. These available for download now.
(b) Did you describe the limitations of your work? [Yes] See Discussion and Limitations: comparable game mechanics to 1.x
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Ethics and Responsible Use. This a simulated game with no personal information that is substantially abstracted from real world capabilities.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] No concerns for our work. The only portions of our work that are not currently open source are set of held-out tasks. These will be released following the competition.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments (e.g. for benchmarks)...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] The baselines repository available as part of our release
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] Included in training scripts. Tables be included with the competition summary instead of in this manuscript.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [N/A] To appear as part of the post-competition analysis of models
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Individual models are trainable in 8 A100 hours. Desktop-gradt GPUs are also fine.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Contributors are displayed prominently on our homepage. Our baseline is built CleanRL, cited in this paper, and PufferLib, currently unpublished
(b) Did you mention the license of the assets? [Yes] Everything is MIT
10

---

(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] No human data
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] No PII
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] This publication covers the environment release, not the associated competition
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]


Reviews:
[{"review_id": "m1afRpHqh1", "reviewer": "Reviewer_3uba", "summary": "", "questions": "", "limitations": "See below.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "This paper offers an in-depth examination of the Neural MMO 2.0 platform, elucidating its various components such as GameState, Predicates, and Tasks. Accompanied by detailed code snippets and comments, the paper effectively demonstrates the inner workings and structure of the platform, providing valuable insights for researchers and users alike.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "QJ9yyo62y1", "reviewer": "Reviewer_wNF1", "summary": "", "questions": "", "limitations": "The authors claim that the rules of their game are sufficiently far from reality that the platform cannot be used to train actual models of combat or commerce system. I think this is reasonable.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper is well-written, the platform clearly represents a significant amount of work and is already successful", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "JdCIbQ0urp", "reviewer": "Reviewer_3mZR", "summary": "", "questions": "", "limitations": "Ethical concerns have been adequately discussed in Section 4. Two paragraphs are dedicated to addressing the limitations of this environment, and there's the promise for the long-term support of this environment.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Version 2.0 added a user-friendly web UI allowing users to visualize the learning policies, which I think is a big improvement from the Unity-based visualization client from the older version. The web UI allows users to monitor the agents' behaviors frame-by-frame, e.g. market transactions, item consumptions, switching equipment, etc.\n- The introduction of the market exchange system in version 2.0 broadened the trading capabilities of agents, permitting both intra-team and inter-team transactions.\n- Version 2.0 further expanded on the Profession Progression mechanics, and intertwining it with the market exchange system. As agents start to specialize in diverse professions, their preferences for specific goods correspondingly rise, as do their acceptable price points for these goods. This intriguingly mirrors the economic law of supply and demand.\n- The code is highly modular and easy to modify to fit new research needs.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper describes Neural MMO 2.0, an improved environment for massive multiagent multitask RL experimentation.  In comparison to the previous version, this paper adds a flexible task system that enables research on generalization, open-endedness, and curriculum learning.  The technical advances are mostly engineering advances, but the framework has already been adopted by a large community and is enabling a concurrent competition at NeurIPS-2023.  The authors re-structured the paper based on the reviewers' comments.  This work is ready for publication.
Decision: Accept (Poster)