POGEMA: A Benchmark Platform for Cooperative Multi-Agent Pathfinding
Authors: Alexey Skrynnik
Anton Andreychuk
Anatolii Borzilov
Alexander Chernyavskiy
Konstantin Yakovlev
Aleksandr Panov
Venue: ICLR 2025 Poster (2025)
Keywords: MAPF
MARL
RL
Heuristic search
Primary area: datasets and benchmarks

Abstract: Multi-agent reinforcement learning (MARL) has recently excelled in solving challenging cooperative and competitive multi-agent problems in various environments, typically involving a small number of agents and full observability. Moreover, a range of crucial robotics-related tasks, such as multi-robot pathfinding, which have traditionally been approached with classical non-learnable methods (e.g., heuristic search), are now being suggested for solution using learning-based or hybrid methods. However, in this domain, it remains difficult, if not impossible, to conduct a fair comparison between classical, learning-based, and hybrid approaches due to the lack of a unified framework that supports both learning and evaluation. To address this, we introduce POGEMA, a comprehensive set of tools that includes a fast environment for learning, a problem instance generator, a collection of predefined problem instances, a visualization toolkit, and a benchmarking tool for automated evaluation. We also introduce and define an evaluation protocol that specifies a range of domain-related metrics, computed based on primary evaluation indicators (such as success rate and path length), enabling a fair multi-fold comparison. The results of this comparison, which involves a variety of state-of-the-art MARL, search-based, and hybrid methods, are presented.

Published as a conference paper at ICLR 2025
POGEMA: A BENCHMARK PLATFORM FOR COOPERATIVE MULTI-AGENT PATHFINDING
Alexey Skrynnik Anton Andreychuk s1 Anatolii Borzilov Alexander Chernyavskiy', Konstantin Yakoviev2., and Aleksandr Panoy
AIRI, Moscow, Russia; FRCC CSC RAS, Moscow, Russia; MIPP, Dolgoprudny, Russia
ABSTRACT
Multi-agent reinforcement learning (MARL) has recently excelled in solving chal- lenging cooperative and competitive multi-agent problems various environments, typically involving small number of agents and full observability. Moreover, range of crucial robotics-related tasks, such as multi-robot pathfinding, which have traditionally been approached with classical non-learnable methods (e.g., heuristic search), are now being suggested for solution using seaning-bbseed or hybrid meth- ods. However, in this domain, remains difficult, not impossible, to conduct a fair comparison between classical, learning based, and hybrid approaches due to the lack of unified framework that supports both learning and evaluation. To ad- dress this, we introduce POGEMA, a comprehensive set of tools that includes fast environment for learning, a problem instance generator, a collection of predefined problem instances, visualization toolkit, and benchmarking tool for automated evaluation. We also introduce and define an evaluation protocol that specifies a range of domain-relaan related metrics, computed based primary evaluation indicators (such as success rate and path length), enabling a fair multi-fold comparison. The results of this comparison, which involves variety of cate-of-thhe-ar MARI,, bearch-based, and hybrid methods, are presented.
Obstacles
Out-of
Caoddiation
40
0
Pathfirding
Cooperation
0
Target
Perfóraaanc
Seatanlitt
(a) task
observation
results
Figure 1: (a) Example of the multi-agent pathfinding problem considered in POGEMA: each agent must reach its goal, denoted by a flag of the same color. (b) Observation tensor of the red agent. (c) Evaluation results of MARL, hybrid, and search-based solvers on POGEMA benchmark.
INTRODUCTION
Multi-agent reinforcement learning (MARL.) has gained increasing attention recently, with significant progress achieved the field (Canese al., 2021; Nguyen al., 2020; Wong et al., 2023). MARL methods have demonstrated the ability to generate well-performing agents' policies in strategic games (Arulkumaran et al., 2019; Ye et al., 2020), sports simulators (Song et al., 2023; Zang et al., 2024), multi-component robot control (Wang et al., 2024), city traffic control (Kolat et al., 2023), and autonomous driving (Zhou et al., 2020). Several approaches currently exist for formulating
Corresponding authors: "skrynnikalexey@ gmmil.comm "andreychuk@airiine"

---

Published as a conference paper at ICLR 2025
and solving MARL problems, depending on the information available to agents and the type of communication allowed in the environment (Zhang et al., 2021). With the growing interest in robotic applications, decentralized cooperative learning that minimizes communication between agents has recently attracted particular attention (Singh et al., 2022; Zhang et al., 2020).
A prominent example of important and practically-inspiree problem that benefit from this type of learning is the so-called multi-agent pathfinding (MAPF) (Stern et al., 2019). In this problem a graph of locations and each time step an agent
group of (homogeneous) agents is confined can either wait at the current vertex or move to an adjacent one. Each agent is assigned a goal and the task is make the agents reach their goals as fast as possible (i.e. using fewer actions) in safe way, i.c. avoiding the inter-agent collisions as well as collisions with the static obstacles. Numerous practically important applications mimic the discrete nature of MAPF and actively exploit MAPF solutions in real world. A major example is automated warehouses (Dekhne et al., 2019) where robots move synchronously utilizing atomic moves. Furthermore, many works, e.g. (Hönig et al., 2016; Ma et al., 2019; Okumura et al., 2022), describe how MAPF solutions can be transferred to real robots that are subject to kinodynamic constraints, inaccurate execution, and other physically inspired complications. Thus, the MAPF problem serves highly useful abstraction that distills the core challenge of any multi- navigation problem: coordinating actions between agents to minimize the risk of collisions while optimizing a given cost objective.
Conventional MAPF solvers rely on such algorithmic techniques as systematic (heuristic) search: A A*+ID+OD (Standley, 2010), CBS (Sharon et al., 2015), M* (Wagner & Choset, 201 1), ICTS (Sharon et al., 2013); or reduce the MAPF problem to the other established computational problem like boolean satisfiability (Surynek et al., 2016) or integer linear programming; or leverage dedicated rule-based techniques (de Wilde et al., 2013). Meanwhile, learning- based MAPF solvers have recently been receiving increased attention, such as (Sartoretti al., 2019; Ma et al., 2021; Damani al., 2021; Skrynnik et al., 2024a), to name a few. A key advantage of such solvers is their decentralized nature, allowing each agent to act independently, which can significantly reduce costs compared to classical MAPF solvers requiring centralized control.
From learning perspective, MAPF is a highly specific problem that differs significantly from well-known MARL problems such as the StarCraft Multi-agent Challenge (SMAC) (Ellis et al., 2024) or Google Research Football (Kurach et al., 2020). In MAPE, generalization to different types of maps and numbers of agents is essential, real-worl world MAPF solvers must handle varying map topologies and agent populations. Furthermore, the number of agents in MAPF is often very large not just dozens (as in SMAC) but hundreds or even thousands of agents are moving simultaneously in the environment. Unsurprisingly, the vast majority of state-of-the-ar learnable MAPF solvers, such as (Skrynnik et al., 2023; 2024a:b; Ma al., 2021; Wang et al., 2023; Sartoretti et al., 2019; Wang et al., 2020a; Liu et al., 2020; Damani et al., 2021), are the hybrid solvers that rely on both traditional search-based techniques and learnable components.
These solvers are developed using different frameworks, environments and datasets and are evaluated accordingly, i.e. in the absence of the unifying evaluation framework, consisting of the evaluation tool, protocol (that defines common performance indicators) and the dataset of the problem instances. Moreover, currently most of the pure MARL methods, i.e. the ones that not involve search-based modules, such as QMIX (Rashid et al., 2020), MAMBA (Egorov & Shpilman, 2022), MAPPO (Yu et al., 2022) etc., are mostly not included in comparison. This exclusion is likely due to the lack of a unified benchmark that is compatible with or integrated with existing MARL learning frameworks.
To close the mentioned gaps we introduce POGEMA, a comprehensive set of tools that includes:
a fast and flexible environment supporting different multi-robot pathfinding problems, coupled with generator of diverse problem instances to facilitate multi-task learning and generalization testing,
a visualization toolkit enabling the creation of high-quality vector- plots and animations for enhanced analysis and presentation,
a benchmarking tool for automated, parallel evaluation of learning-based planning-based, and hybrid approaches, streamlining comparison across methodologies,
a standardized evaluation protocol offering domain--seciife metrics derived from primary performance indicators, ensuring robust and fair comparisons between methods.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
Currently, huge variety of MARL environments exists that inspired by various practical ap- plications and encompass a broad spectrum of nuances in problem formulations. Notably, they include diverse array of computer games (Samvelyan al., 2019b; Ellis et al., 2024; Rutherford et al., 2023; Carroll et al., 2019; Suarez et al., 2024; Johnson et al., 2016; Bonnet et al., 2023; Baker al.. 2020; Kurach et al., 2020). Additionally, they address complex social dilemmas (Agapiou et al., 2022) including public goods games, resource allocation problems (Papoudakis et al., 2021), and multi-agent coordination challenges. Some are practically inspired, showcasing tasks such as competitive object tracking (Pan et al., 2022), infrastructure management and planning (I (Leroy et al., 2024), and automated scheduling of trains (Mohanty et al., 2020). Beyond these, the environments simulate intricate, interactive systems such traffic management and autonomous vehicle coordina- tion (Vinitsky et al., 2022), multi-agent control tasks (Rutherford et al., 2023; Peng et al., 2021), and warehouse management (Gupta al., 2017). Each scenario is designed challenge and analyze the collaborative and competitive dynamics that emerge among agents in varied and complex contexts. We summarize the most wide-spread MARL environments in Table 1. detailed description of each column is presented below.
Hurdware-sgnosie setup
Performance 1OK Stepa/s
Scalabilitt >- 1000 Agents
Requires genemilzzatio
Procedural generation
Partilly observable
Evaluation protocols
Induced hehaviior
Repository
Novigation
PyPi Listed
Python
Tosts
Environment
Flatland (Mohanty et ol., 2020)
link
Conp
Gigustep (Lechmer el al.,,.22200 2024)
link
Mixed
GoBigger (Zhang al., 2023)
link
Mixes//Coop
Google Research Football (Kurach
link
Mixed
et al.. 2020)
Griddly (Bamford, 2021)
link
Mixed
Hide-and-Seck (Baker et al., 2020)
link
X
Comp
IMP-MARL (Leroy et al.
Coop
Jumanji (XLA) (Bonnet et ol., 2023)
link
Mixed
LBF (Papoudakis et 2021)
2021)
link
Coop
MAMuloCo (Peng al., 2021)
link
/
Coop
Magent (Zheng et al., 2018)
link
X
X
/
Mixed
(Pan al., 2022)
Coop
MeltingPot (Agapiou al., 2022)
link
/
Mixes//Copp
MALMO (Johnson al., 2016)
link
X
Mixed
(Lowe al., 2017)
Mixed
MPE (XLA) (Rutherford et 2023)
link
Mixed
Multi-agent Brax (XLA) (Rutherford
link
Coup
al.. 2023)
Multi-Car Racing (Sclwarting al.,
link
X
X
Comp
2021)
Neural MMO (Suarer al., 2024)
link
Comp
Nocturne (Vinitsky et al., 2022)
link
x
Mixed
Overcooked (Camoll et 2019)
Conp
Overcooked (XLA) (Rutherford et al.,
link
Coop
2023)
RWARE (Papondakis
2021)
link
Coop
SISL (Gupla al., 2017)
link
/
X
Coup
SMAC (Samvelyan et al., 2019b)
link
x
x
Mixed/Coop
SMAC (Eilis
2024)
link
x
Mixed/CCop
SMAX (XLA) (Kutheeford et al., 2023)
link
/
Mixed/Coop
POGEMA (ours)
/
/
Mixed
Table I: Comparison of different multi-agent reinforcement learning environments.
Navigation Navigation tasks arise in almost all multi-agent environments (e.g. unit navigation in SMAC or robotic warchouse management RWARE), however only a handful of environments specifically focus on challenging navigation problems: Flatland, Nocturne, RWARE, and POGEMA.
3

---

Published as a conference paper at ICLR 2025
Partially observable Partial observability is intrinsic feature of generic multi-agent problem, meaning that an individual agent does not have access to the full state of the environment but rather is able to observe it only locally (e.g. agent is able to determine the locations of the other agents and/or static obstacles only in its vicinity). Most of the considered environments are partially observable, with the exception of Overcooked.
Environment Setup Python based means the environment is implemented in Python, unlike many other multi-agent environments that include bindings for other languages or external dependencies, which can complicate usage. Pure Python implementations ensure ease of modification and cus- tomization, allowing researchers to easily adapt and extend the environments. Hardware-agnoss setup means the environment doesn't require any specific type of hardware for training or inference, offering flexibility across different systems.
Training and evaluating multi-agent reinforcement learning agents often requires making billions of steps (transitions) in the environment. Thus, is crucial that each transition is computed efficiently. In general, performing more than 10K steps per second is good indicator of the environment's efficiency. While XLA versions can provide high performance by vectorizing the environment on GPU, they require modern hardware setups, which can be barrier for some researchers. In contrast, fast environments like POGEMA or RWARE can achieve high performance without such stringent hardware requirements, making them more accessible and easier to integrate into a variety of research projects.
Performance >10K Steps/s
Procedural generation To improve the ability of RL agents to solve problem instances that are not the same that were used for training (the so-called ability to generalize) procedural generation of the problem instances is commonly used. Le. the environment does not rely on a predefined set of training instances but rather procedurally generates them prevent overfitting. As highlighted in the Procgen paper (Cobbe et al., 2020), this approach ensures that agents develop robust strategies capable of adapting to novel and diverse situations. Moreover, in multi-agent settings, agents must be able to handle and adapt to variety of unforeseen agent behaviors and strategies, ensuring robustness and flexibility in dynamic environments (Agapiou al., 2022).
Evaluation protocols means that the environment features a comprehensive evaluation API, in- cluding computation of distinct performance indicators and visualization tools. These capabilities allow precise performance measurement and deeper insights into RL agents' behavior, going beyond just reward curves, which can often hide agents exploiting the reward system rather than genuinely solving the tasks.
Tests and CI means the environment is set up for development with continuous integration and is covered with tests, which are essential for collaborative open-source development. PyPI listed indicates that the environment library is listed on PyPl', making it easy to install and integrate into projects with a simple pip install command. Scalability to >1000 Agents refers to the environment's 's ability to handle over 1000 simultaneously acting agents, ensuring robust performance and flexibility for large-scale multi-agent simulations.
Induced behavior Multi-agent behavior can be influenced by modifying the reward func- tion (Shoham & Leyton-Brown 2008). Competitive (Comp) behavior arises when a joint strategy benefits one player but disadvantages others. In a two-player game, this corresponds to a Pareto- efficient outcome. Minimax games, where agents' rewards sum to zero, are classic examples of 2008)
competitive games. Cooperative (Coop) behavior (Du et al., 2023; Shoham & occurs when agents share unified reward function or pursue the same goal, rewarded only by its completion. Social dilemmas are key example of cooperation. Mixed behavior (Littman, 1994) doesn't 't limit the agents' objectives or interactions, blending cooperative and competitive behaviors. A well-known example is the iterated prisoner's 's dilemma.
As we aim to create a lightweight and easy-to-configure multi-agent environment for reinforcement learning and pathfinding tasks, we consider the following factors essential. First and foremost, our en- vironment is fully compatible with the native Python API: we target pure Python builds independent of
https::/pypi.oo
4

---

Published as a conference paper at ICLR 2025
hardware- speecic software with minimal number of external dependencies. Moreover, we underline the importance of constant extension and flexibility of the environment. Thus, we prioritize testing and continuous integration as comnertonns of the environment, well trouble-free modification of the transition dynamics. Secondly, we highlight that our environment targets generalization and may utilize procedural generation. Last but not least, target high computational throughput (i.e., the number of environment steps per second) and robustness to an extremely large number of agents (i.e., the environment remains efficient under high loads). The detailed overview of the mentioned related papers is provided in Appendix F.
Despite the diversity of available environments, most research papers tend utilize only selected few. Among these, the most popular are the StarCraft Multi-agent Challenge (SMAC), Multi-agent MuloCo (MAMuJoCo)) and Google Research Football (GRF), with SMAC being the most prevalent in top conference papers. The popularity of these environments is likely due to their effective contextualization algorithms. For instance, to demonstrate the advantages of a method, it is crucial to test it within well-known environment.
The evaluation protocols in these environments typically feature learning curves that highlight the performance of each algorithm under specific scenarios. For SMAC, these scenarios involve games against predefined bots with specific units on both sides. In MAMuJoCo, the standard tasks involve agents controlling different sets of joints, while in GRF, the scenarios are games against predefined policies from Football Academy scenarios. Proper evaluation of MARL approaches is a serious concern. For SMAC, it's highlighted in the paper (Gorsane al., 2022), which proposes unified evaluation protocol for this benchmark. This protocol includes default evaluation parameters, performance metrics, uncertainty quantification, and results reporting scheme.
The variability of results across different studies underscores the importance of a well-defined evaluation protocol, which should be developed alongside the presentation the environment. In our study, we provide not only the environment but also the evaluation protocol, popular MARL baselines, and modern learnable MAPF approaches to better position our benchmark within the context.
POGEMA
POGEMA, which comes from Partially-Observabel Grid Environment for Multiple Agents, is an umbrella name for a collection of versatile and flexible tools aimed at developing, debugging and evaluating different methods and policies tailored to solve several types of multi-agent pathfinding tasks. The source code is available at: POGEMA Benchmark?, POGEMA Toolbox and POGEMA Environment
3.1 POGEMA ENVIRONMENT
POGEMA environment is core of POGEMA suite. It implements the basic mechanics of agents' interaction with the world. The environemnt open-soureed under MIT license. POGEMA provides integration with existing RL frameworks: PettingZoo (Terry et al., 2021), PyMARL (Samvelyan et al., 2019a), and Gymnasium (Towers et al., 2023).
Basic mechanics The workspace where the agents navigate is represented as grid composed of blocked and free cells. Only the free cells are available for navigation. At each timestep each agent individually and independently (in accordance with policy) picks an action and then these actions are performed simultaneously. POGEMA implements collision shielding mechanism, i.e. if an agent obstacle (or out-of-the toff-themme (hap) than it stays put, the same applies for two or more agents that wish to occupy the same cell. POGEMA also has an option when one of the agents deciding move the common cell does it, while the others stay where they were. The episode ends when the predefined timestep, episode length, is reached. The episode can also end before this timestep if certain conditions are met, i.c. all agents reach their goal locations if MAPF problem (see below) is considered.
picks action that leads
5

---

Published as a conference paper at ICLR 2025
Problem settings POGEMA supports two generic types of multi-agent navigation problems. In the first variant, dubbed MAPF (from Multi-agent Pathfinding), each agent is provided with the unique goal location and has to reach it avoiding collisions with the other agents and static obstacles. For MAPF problem setting POGEMA supports both stay-al-targe behavior (when the episode successfully ends only if all the agents are at their targets) and disappear-at-taree removed from the environment after it first reaches its goal). The second variant is a lifelong version of multi-agent pathfinding and is dubbed accordingly LMAPF. Here each agent upon reaching goal is immediately assigned another one (not known to the agent beforehand). Thus, the agents are constantly moving through in the environment until episode ends.
(when the agent is
Observation At each timestep each agent in POGEMA receives an individual ego-centric observa- tion represented as a tensor - see Figure 1. The latter is composed of the following (2R+1) (2R+11 binary matrices, where R is the observation radius set by the user:
1. Static Obstacles means the free cell, I static obstacle
2. Other Agents means no agent in the cell. the other agent occupies the cell
3. Targets projection of the (current) goal location of the agent to the boundary of its field-of- view
The suggested observation, which is. indeed, minimalist and simplistic, can be modified by the user using wrapper mechanisms. For example, it is not uncommon in the MAPF literature to augment the observation with additional matrices encoding the agent's 's path-to-goal (constructed by some global pathfinding routine) (Skrynnik et al., 2024a) or other variants of global guidance (Ma et al., 2021).
Reward POGEMA features the most intuitive and basic reward structure for learning. I.e. an agent is rewarded with +1 if it reaches the goal and receives 0 otherwise. For MARL policies that leverage centralized training shared reward supported, i.e. rt = goals/agents agents where goals is the number of goals reached by the agents at timestep t and agents is the number of agents. Indeed, the user can specify its own reward using wrappers.
Performance indicators The following performance indicators considered basic and tracked in each episode. For MAPF they are: Sum-of-costs (SoC) and makespan. The former is the sum of time steps (across all agents) consumed by the agents to reach their respective goals, the latter the maximum over those times. The lower those indicators are the more effectively the agents are solving MAPF tasks, For LMAPF the primary tracked indicator the throughpur which is the ratio of the number of the accomplished goals (by all agents) to the episode length. The higher- the better.
3.2 POGEMA TOOLBOX
The POGEMA Toolbox is comprehensive framework designed to facilitate the testing of learning- based approaches within the POGEMA environment. This toolbox offers a unified interface that enables the seamless execution of any learnable MAPF algorithm in POGEMA. Firstly, the toolbox provides robust management tools for custom maps, allowing users to register and utilize these maps effectively within POGEMA. Secondly, enables the concurrent execution of multiple testing instances across various algorithms in a distributed manner, leveraging Dask for scalable processing. The results from these instances are then aggregated for analysis. Lastly, the toolbox includes visualization capabilities, offering convenient method to graphically represent the aggregated results through detailed plots. This functionality enhances the interpretability of outcomes, facilitating a deeper understanding of algorithm performance.
POGEMA Toolbox offers a dedicated tool for map generation, allowing the creation of three dis- tinct types of maps: random, mazes and warehouse maps. All generators facilitates map creation using adjustable parameters such as width, height, and obstacle density. The maze generator was implemented based on the generator provided in (Damani al., 2021).
"https://githiee dask/aakkkka /kakkk

---

Published as a conference paper at ICLR 2025
3.3 BASELINES
POGEMA integrates a variety of MARL, hybrid and planning-based algorithms with the environment. These algorithms, recently presented. demonstrate state-of-the-ee performance in their respective fields. Table 2 highlights the differences between these approaches. Some, such as LaCAM and RHCR, are centralized search-based planners. Other approaches, such as SCRIMP and DCC, while decentralized, still require communication between agents to resolve potential collisions. Learnable modern approaches for LifeLong MAPF that do not utilize communication include Follower (Skrynnik et al., 2024a), MATS-LP (Skrynnik al., 2024b), and Switchers (Skrynnik al., 2023). All these approaches utilize independent PPO (Schulman et al., 2017) as the training method.
Decentralized
Partial Observability
Fully Integrated into POGEMA
Supports MAPF
Supports Lifel Lifel.ong MAPP
No Global Obstacles Map
No Communicatinl
Decentralized
Model-Bused Leaming
No Imitation Learning
Parameter Sharing
Algorithm
MAMBA (Eggoro & Shpilman, 2022)
QPLEX (Wang
2020b)
IQL (Tan, 1993)
X <
VDN (Sunchag al., 2018)
QMI (Rashid et al., 2020)
Follower (Skrynnik et al., 2024a)
X
MATS-LP (Skrynik
al., 2024b)
Switcher (Skrynnik et al., 2023)
SCRIMP (Wang et al., 2023)
DCC (Ma et al., 2021)
LaCAM (Okumura. 2024; 2023)
RHCR (Li al., 20216)
X
Table 2:
This table provides an overview of various baseline approaches supported by POGEMA and their features in the context of decentralized multi-agent pathfinding.
The following modern MARL algorithms are included as baselines: MAMBA (Egorov & Shpil- man, 2022), QPLEX (Wang al., 2020b), IQL (Tan, 1993), VDN (Sunehag al., 2018), and QMIX (Rashid et al., 2020). For environment preprocessing, we used the scheme provided in the Follower approach, enhancing with the anonymous targets of other agents' local observations for MAPF scenarios. We utilized the official implementation of MAMBA, as provided by its authors?, and employed PyMARL2 framework for establishing MARL baselines. We used the default param- eters for MAMBA, since Dreamer (which serves the foundation for MAMBA) is known to work effectively across domains with nearly the same hyperparameters. For the other MARL approaches, we performed hyperparameter sweep over the learning rate, batch size, replay buffer size, and GRU hidden state size, using the best parameters based on the performance scores from the training scenarios on the Random and Mazes maps.
EVALUATION PROTOCOL
4.1 DATASET
We include the maps of the following types our evaluation dataset (with the intuition that different maps topologies are necessary for proper assessment):
Mazes maps that encounter prolonged corridors with 1-cell width that require high level of cooperation between the agents. These maps are procedurally generated.
Random one of the most commonly used type of maps, as they are easy to generate and allow to avoid overfitting to some special structure of the map. POGEMA contains an integrated random maps generator, that allows control the density of the obstacles.
wwr-acoo/jjrriiaaaa jbbsai /amba pettps://github.com thub. http:/ijjjzzzzz
'https://
7

---

Published as a conference paper at ICLR 2025
Warehouse this type of maps usually used in the papers related to LifeLong MAPF. While there no narrow passages, high density of the agents might significantly reduce the overall throughput, especially when agents are badly distributed along the map.
Cities a set of city maps from MovingAI the existing benchmark widely used in heuristic-search community Sturtevant (2012). The contained maps have varying structure and 256 x 256 size. It can be used to show how the approach deals with single-agent pathfinding and also deals with the maps that have outto--distribution structure.
a modified Cit ies set of maps. Due to the large size of the original maps, it's hard to get high density of the agents them. To get more crowded maps, slice the original maps on 16 pieces with 64 x 64 size.
Cities-tiles
Puzzles set of small hand-crafted maps that contains some difficult patterns that mandate the cooperation between that agents.
Mazes
(b) Random
(c) Warehouse
Figure 2: Examples of map generators presented in POGEMA.
Start and goal locations are generated via random generators. They generated with fixed seeds, thus can be reproduced. It is guaranteed that each agent has its own goal location and the path to it from its start location exists. Examples of the maps presented in Figure 2.
4.2 METRICS
The existing works related to solving MAPF problems evaluates the performance by two major criteria success rate and the primary performance indicators mentioned above: sum-of-costs, makespan, throughput. These
directly obtainable from POGEMA. While these metrics allow to evaluate the algorithms at some particular instance, might be difficult to get a high-level conclusion about the performance. Thus, want introduce high-level metries that cover multiple different aspects:
Performance how well the algorithm works compared to other approaches. To compute this metric run the approaches set of maps similar to the ones, used during training. and compare the obtained results with the best ones.
/SCresst SoC
0 if not solved
PerformanceLMMPF throughput, throughputeesst ttrronghpt//hrrogggpuu....
Out wwt--o--Dissiibutiinnn how well the algorithm works on out-of-distribution maps. This metric computed in the same way Performance with the only difference that the approaches evaluated on set of maps, that were not used during training phase and have different structure of obstacles. For this purpose utilize maps from Cities-til iles set of maps.
SoChrest/ S SoC (0 if not solved
Out of DistributionmaPP
(3)
Out of DistrubuutionnLMPP throughput/
wtwoougghuut/hrroughuuutt.. /throughputwest
Cooperation iiaationn how well the algorithm is able to resolve complex situations. To evaluate this metric the algorithm Puzzl of maps.
8

---

Published as a conference paper at ICLR 2025
SoCbest S
CooperationMAPF 0 if not solved
(5)
Cooperation = ttrroughput/ throughputees throuughput/hrrogghhptts.
(6)
Scalability - how well the algorithm scales to large number of agents. To evaluate how well the algorithm scales large number of agents, run large warchouse map with increasing number of agents and compute the ratio between runtimes with various number of agents.
Scalability prwntiinoggentts///mmmmeeegoooie
(agentsi// [agendai///geendas]] [ggnlss)
Coordination how well the algorithm avoids collisions between agents and obstacles. Majority of learning-based based approaches are decentralized, so there may be cases where multiple agents attempt occupy the same cell traverse the same edge simultaneously. There also might be the case when an agent tries to move to the blocked cell. The fewer collisions that occur during the episode, the better. To compute this metric, we use the results obtained on the Mazes and Random sets of maps.
accured_collisionn.
Coordination
8
lagents) X apisodeelengll)
Pathf lothfinndngg how well the algorithm works in case of presence of a single agent on a large map. This metric is tailored determine the ability of the approach effectively lead agents to their goal locations. For this purpose we run the approaches on large maps from Cit Cities benchmark sets. The closer the costs of the found paths to the optimal ones the higher the score.
path_coste taathhhossspaa cost/path http:ppahhhhossppttteeee costoptimal
Path finding =
0 if path not found
First three metrics, i.e. Performance, Out-of-De witiiiiiiii stssstttttoon ion, and Cooperat on, have the same formula but differ in the of maps used to compute them, while the remaining three metrics, ic. Scalability, Coordination ion, and Pathfindii tailored to specific aspects.
4.3 EXPERIMENTAL RESULTS
We have evaluated all the supported baselines (12 in total) on both MAPF and LMAPF setups on all datasets. The results of this evaluation are presented in Figure The details about number of maps, number of agents, seeds, etc. are given in the supplementary material (as well as details on how our results can reproduced). In both setups, i.e., MAPF and LMAPF, the best results in terms of cooperation, out-of-distribution and performance metrics were obtained by centralized planners, ie.. LaCAM and RHCR respectively.
LaCAM
DCC
VON
OPLEX
RHCR
MATS-LP
VON
QPLEX
Aswitcher
Follower
Switcher
Distiibiinn Coordination
Of Distribbutinnn Coordination
100
100
80
80
60
60
40
Pathfinding
Cooperation
Pathfinding
Cooperation
Performannc
callaillty
Performance
Scalability
Figure 3: Evaluation of baselines available in POGEMA MAPF (b) LMAPF instances.

---

Published as a conference paper at ICLR 2025
For MAPF tasks, LaCAM notably outperforms all other approaches. Specialized learnable MAPF approaches, such as DCC and SCRIMP, perform better than any of the MARL approaches. When comparing these two specialized approaches, SCRIMP outperforms DCC across such metries Per- formance, Cooperation, and Out-of-Distribution. Surprisingly, SCRIMP underperforms on pathfind- ing tasks, revealing weakness in single-agent scenarios that don't 't require communication, which may represent an cutto--distribution case for this algorithm. It is also worth noting that SCRIMP incorporates an integrated tie-breaking mechanism that ensures collision-free actions. MARL algo- rithms such as QPLEX, VDN, and QMIX perform significantly worse than other approaches, showing a substantial gap in the results. This can be attributed to the absence of additional techniques used in hybrid approaches. This may suggest that the MARL community lacks large-scale approaches and benchmarks for these tasks. Among MARL approaches, MAMBA achieves the best results in performance, cooperation, and pathfinding metrics, which can be attributed to its communication mechanism. However, its performance remains much worse than that of specialized methods, and it fails to solve any instances in the oat-of-distribution dataset.
For LMAPF tasks, the centralized approach, RHCR, is superior in all cases except for the scalability metric. Among the non-centralized approaches, the best results, depending the metric, shown by either MATS-LE or Follower. The only metric in which these approaches differ significantly is the cooperation metric, where MATS-LP performs better than Follower. However, MATS-LP requires considerably more test-time computation, as it runs MCTS for each agent at every step (see Appendix I for more details). Additionally, there are two hybrid approaches ASwitcher and LSwitcher - that differ in how they alternate between planning-based and learning-bast based components. One reason for their mediocre performance is the lack of global information, i.e., they assume that agents have no prior knowledge of the map, requiring each agent to reconstruct it based on the local observations. Differently from the MAPF scenarios, MARL approaches can compete with hybrid methods LMAPF instances. This behavior is partly explained by the use of Follower's 's observation model, which specifically designed to solve LMAPF. Among MARL approaches, QPLEX delivers the best results, in contrast to MAPF tasks it even outperforming MAMBA.
5 CONCLUSION, LIMITATIONS AND FUTURE WORK
This paper presents POGEMA powerful suite of tools tailored for creating, assessing, and comparing methods and policies for multi-agent pathfinding. POGEMA encompasses a fast learning environment and comprehensive evaluation toolbox suitable for pure MARL, hybrid, and search- based solvers. It includes a wide array of methods as baselines. The evaluation protocol described, along with a rich set of metrics, assists in assessing the generalization and scalability of all approaches. Visualization tools enable qualitative examination of algorithm performance. Integration with the well-known MARL API and map sets facilitates the benchmark's expansion. Existing limitations are two-fold. First, a conceptual limitation is that communication between agents is not currently disentangled in POGEMA environment. Second, technical limitations include the lack of JAX support and integration with the other GPU parallelization tools.
Future work could explore large-scale training setups for MARL methods, capable of handling large agent populations and scenarios, particularly within the CTDE (Centralized Training. Decentralized Execution) paradigm. Second, advancing communication learning suited large-scale settings, where local interactions are crucial, is another promising direction. Third, addressing memory limitations by developing efficient approaches for long-horizon tasks without relying on global guidance is also vital. Furthermore, leveraging POGEMA's procedural map generation and expert data from centralized solvers could enhance imitation learning and aid in training decentralized foundation models for MAPF. Finally, studying heterogeneous policy coordination may enable effective collaboration among diverse agents and advance concurrent training of different algorithms in shared environments.
ACKNOWLEDGMENTS
We thank the researchers using POGEMA in their studies, as well as the repository users who contributed issues, questions, and feedback to improve the platform. Our thanks also go to the anonymous reviewers for their valuable comments and suggestions, which enhanced the quality of this work.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
John P Agapiou, Alexander Sasha Vezhnevets, Edgar A Dueitez-Guzmán, Jayd Matyas, Yiran Mao, Peter Sunehag, Raphael Köster, Udari Madhushani, Kavya Kopparapu, Ramona Comanescu, et al. Melting pot 2.0. arXiv preprint arXiv:221 1..2211.13445 2022.
Kai Arulkumaran, Antoine Cully, and Julian Togelius. Alphastar: An evolutionary computation perspective. In Proceedings of the genetic and evolutionary computation conference companion, pp. 314-315, 2019.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In International Conference on Learn- ing Representations, 2020. URL https: openreviev. net/ uurmm?? ===
Christopher Bamford. Griddly: A platform for ai research in games. Software Impacts, 8: 80066 2021.
Clément Bonnet, Daniel Luo, Donal John Byrne, Shikha Surana, Paul Duckworth, Vincent Coyette, Laurence Illing Midgley, Sasha Abramowitz, Elshadai Tegegn, Tristan Kalloniatis, al. Jumanji: diverse suite of scalable reinforcement learning environments in jax. In The Twelfth International Conference on Learning Representations, 2023.
Lorenzo Canese, Gian Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, and Sergio Spanò. Multi-agent reinforcement learning: review of challenges and applications. Applied Sciences, 11(11):49444 2021.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
Filippos Christianos, Lukas Schäfer, and Stefano Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. Advances in neural information processing systems, 33: 10707-10717, 2020.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference machine learning, pp-
2048-2056. PMLR, 2020.
Mchul Damani, Zhiyao Luo, Emerson Wenzel, and Guillaume Sartoretti. Primal 2: Pathfinding via reinforcement and imitation multi-agent learning lifelong. IEEE Robotics and Automation Letters, 6(2):2666-2677 2021.
Boris de Wilde, Adriaan W ter Mors, and Cees Witteveen. Push and rotate: cooperative multi-agent path planning. In Proceedings of the I 12th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2013), pp. 87-94, 2013.
Ashutosh Dekhne, Greg Hastings, John Murnane, and Florian Neuhaus. Automation in logistics: Big opportunity, bigger uncertainty. McKinsey Q, 24, 2019.
Yali Du, Joel Z Leibo, Usman Islam, Richard Willis, and Peter Sunehag. A review of cooperation in multi-agent learning. arXiv preprint arXiv:2312.0516. 2023.
Vladimir Egorov and Alexei Shpilman. Scalable multi-agent model-based reinforcement learning. In Proceedings of the 21 International Conference on Autonomous Agents and Multiageni Systems, pp. 381-390, 2022.
Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Foerster, and Shimon Whiteson. Smaev2: An improved benchmark for cooperative multi- agent reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Rihab Gorsane, Omayma Mahjoub, Ruan John de Kock, Roland Dubb, Siddarth Singh, and Arnu Pretorius. Towards a standardised performance evaluation protocol for cooperative marl. Advances in Neural Information Processing Systems, 35:5510-5522 2022.

---

Published as a conference paper at ICLR 2025
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. Autonomous Agents and Multiagent Systems. pp. 66-6668888 83, 2017.
Wolfgang Hönig, TK Satish Kumar, Liron Cohen, Hang Ma, Hong Xu, Nora Ayanian, and Sven Koenig. Multi-agent path finding with kinematic constraints. In Proceedings of The 26th In- ternational Conference on Automated Planning and Scheduling (ICAPS 2016), pp. 477-485, 2016.
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pp. 4246-4247, 2016.
Máté Kolat, Bálint Kövári, Tamás Bécsi, and Szilárd Aradi. Multi-agent reinforcement learning for traffic signal control: A cooperative approach. Sustainability, 15(4):3479, 2023.
Karol Kurach, Anton Raichuk, Piotr Stańczyk, Michat Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: novel reinforcement learning environment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 4501-4510, 2020.
Florian Laurent, Manuel Schneider, Christian Scheller, Jeremy Watson, Jiaoyang Li, Zhe Chen, Yi Zheng, Shao-Hura Hung Chan, Konstantin Makhnev, Oleg Svidchenko, et al. Flatland competition 2020: Mapf and marl for efficient train coordination on grid world. In NeurlPS 2020 Competition and Demonstration Track, pp. 275-301. PMLR, 2021.
Mathias Lechner, Tim Seyde, Tsun-Hsuar Hsuan Johnson Wang, Wei Xiao, Ramin Hasani, Joshua Rountree, Daniela Rus, et al. Gigastep-one billion steps per second multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Pascal Leroy, Pablo G Morato, Jonathan Pisane, Athanasios Kolios, and Damien Ernst. Imp-marl: a suite of environments for large-scale infrastructure management planning via marl. Advances in Neural Information Processing Systems, 36, 2024.
Jiaoyang Li, Zhe Chen, Yi Zheng, Shao-Hung Chan, Daniel Harabor, Peter J Stuckey, Hang Ma. and Sven Koenig. Scalable rail planning and replanning: Winning the 2020 flatland challenge. In Proceedings of the international conference on automated planning and scheduling, volume 31, pp. 477-485, 2021a.
Jiaoyang Li, Andrew Tinka, Scott Kiesel, Joseph W. Durham, T. K. Satish Kumar, and Sven Koenig. Lifelong multi-agen agent path finding in large-scale warchouses. Proceedings the AAAI Conference on Artificial Intelligence,
May 2021b. doi: 10. 10.1609/aai..v35133.. https: /ojs.....///// jas.......... aaai org/ index,xx phpp /AAT/AAAAAAAA AwA.//atticle//eee//////aaattte/
URL
Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
Zuxin Liu, Baiming Chen, Hongyi Zhou, Guru Koushik, Martial Hebert, and Ding Zhao. Mapper: Multi-agent path planning with evolutionary reinforcement learning in mixed dynamic environ- ments. In Proceedings of the 2020 IEEE/RS/ International Conference Intelligent Robots and Systems (IROS 2020), pp. 11748-11754, 11754, 2020.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettie environments. Advances in neural information processing systems, 30, 2017.
H. Ma, W. Hönig, T. K. S. Kumar. N. Ayanian, and S. Koenig. Lifelong path planning with kinematic constraints for multi-agent pickup and delivery. Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI 2019), pp. 7651-7658, 2019.
Ziyuan Ma, Yudong Luo, and Jia Pan. Learning selective communication for multi-agent path finding. IEEE Robotics and Automation Letters, 145 2021. 2021.
12

---

Published as a conference paper at ICLR 2025
Sharada Mohanty, Erik Nygren, Florian Laurent, Manuel Schneider, Christian Scheller, Nilabha Bhattacharya, Jeremy Watson, Adrian Egli, Christian Eichenberger, Christian Baumberger, et al. Flatland-rl: Multi-agent reinforcement learning trains. arXiv preprint arXfv:2012.05893, 2020.
Thanh Thi Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE transactions on cybernetics, 50(9):382633839, 2020.
Keisuke Okumura. Lacam: Search-based algorithm for quick multi-agent pathfinding, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11655-11662 2023.
Keisuke Okumura. Engincering lacam*: Towards real-time. large-scale, and near-optimal multi-agent pathfinding. Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 1501-1509, 2024.
Keisuke Okumura, Manao Machida, Xavier Défago, and Yasumasa Tamura. Priority inheritance with backtracking for iterative multi-agent path finding. Artificial Intelligence, 310: 103752, 2022.
Xuehai Pan, Mickel Liu, Fangwei Zhong, Yaodong Yang, Song-Chun Zhu, and Yizhou Wang. Mate: Benchmarking multi-agent reinforcement learning in distributed target coverage control. Advances in Neural Information Processing Systems, 35:27862-217 228797,, 2022.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https: wpwnrevieeeeeeeeeeeeeeeeee net/ / forum? cieiii=cIiPPPPP Sn5n.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Alexa--Ae Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Faemac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 14:12208-1122) 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178) 2020.
Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardar Ing- varsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, et al. Jaxmarl: Multi-agent rl environments in jax. arXiv preprint 1rXiv:2311.10000 2023.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1 002.04043, 2019a.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung. Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MulriAgent Systems, pp. 2186-2188, 2019b.
Guillaume Sartoretti, Justin Kerr, Yunfei Shi, Glenn Wagner, TK Satish Kumar, Sven Koenig, and Howie Choset. Primal: Pathfinding via reinforcement and imitation multi-agent learning. IEEE Robotics and Automation Letters, 4(3):2378-2333 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Wilko Schwarting, Tim Seyde, Igor Gilitschenski, Lucas Liebenwein, Ryan Sander, Sertac Karaman, and Daniela Rus. Deep latent competition: Learning to race using visual control policies in latent space. In Conference Robot Learning, pp. 1855-1870. PMLR, 2021.
G. Sharon, R. Stern, A. Felner, and N. R. Sturtevant. Conflict-based search for optimal multiagent path finding. Artificial Intelligence Journal, 218:40-66 2015.
Guni Sharon. Roni Stern, Meir Goldenberg, and Ariel Felner. The increasing cost tree search for optimal multi-agent pathfinding. Artificial intelligence, 195:470-495, 2013.
13

---

Published as a conference paper at ICLR 2025
Yoav Shoham and Kevin Sesyton-Browny Multiagent systems: Algorithmic, game-theoreti, and logical foundations. Cambridge University Press, 2008.
Bharat Singh, Rajesh Kumar, and Vinay Pratap Singh. Reinforcement learning in robotic applications:
comprehensive survey. Artificial Intelligence Review, 55(2):945-999) 2022.
Alexey Skrynnik, Anton Andreychuk, Konstantin Yakovlev, and Alcksandr I Panov. When to switch: planning and learning for partially observable multi-agent pathfinding. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Alexey Skrynnik, Anton Andreychuk, Maria Nesterova, Konstantin Yakovlev, and Alcksandr Panov. Learn to follow: Decentralized lifelong multi-agent pathfinding via planning and learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 17541-17549, 2024a.
Alexey Skrynnik, Anton Andreychuk, Konstantin Yakovlev, and Aleksandr Panov. Decentralized monte carlo tree search for partially observable multi-agent pathfinding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17531-17540, 2024b.
Yan Song, He Jiang, Haifeng Zhang, Zheng Tian, Weinan Zhang, and Jun Wang. Boosting studies of multi-agent reinforcement learning on google research football environment: the past, present, and future. arXiv preprint arXiv:2309. /2951, 2023.
T. S. Standley. Finding optimal solutions to cooperative pathfinding problems. In Proceedings of The 24th AAAI Conference on Artificial Intelligence (AAAI 2010), pp. 173-178, 2010.
Roni Stern, Nathan R Sturtevant, Ariel Felner, Sven Koenig, Hang Ma, Thayne T Walker, Jiaoyang Li, Dor Atzmon, Liron Cohen, TK Satish Kumar, et al. Multi-agent pathfinding: Definitions, variants, and benchmarks. In Proceedings of the 12th Annual Symposium on Combinatorial Search (SoCS 2019), pp. 151-158, 2019.
Nathan R Sturtevant. Benchmarks for grid-based pathfinding. IEEE Transactions on Computational Intelligence and Al in Games, (2)::4441453 2012.
Joseph Suarez, David Bloomin, Kyoung Whan Choe, Hao Xiang Li, Ryan Sullivan, Nishaanth Kanna, Daniel Scott, Rose Shuman, Herbie Bradley, Louis Castricato, et al. Neural mmo 2.0: A massively multi-task addition to massively multi-agent learning. Advances in Neural Information Processing Systems, 36, 2024.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czamecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanetot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls. et al. Value-decommpoiition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MulliAgen gent Systems, pp. 2085-2087, 2018.
Pavel Surynek, Ariel Felner, Roni Stern, and Eli Boyarski. Efficient sat approach multi-agent path finding under the sum of costs objective. In Proceedings of the 22nd European Conference Artificial Intelligence (ECAl 2016), pp. 810-818. IOS Press, 2016
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 330-337, 1993.
Jordan Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez-Vicente et al. Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:15032-15043, 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033.
IEEE, 2012. doi: 10.1
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL https: : /zenodo. org/ ret.rd/cood////// 81005.......
14

---

Published as a conference paper at ICLR 2025
Eugene Vinitsky, Nathan Lichtlé, Xiaomeng Yang, Brandon Amos, and Jakob Foerster. Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world. Advances in Neural Information Processing Systems, 35:3962-3974 2022.
Glenn Wagner and Howie Choset. M A complete multirobot path planning algorithm with performance bounds. In Proceedings of The 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2011). pp. 3260-3267, 2011.
Binyu Wang, Zhe Liu, Qingbiao Li, and Amanda Prorok. Mobile robot path planning in dynamic environments through globally guided reinforcement learning. IEEE Robotics and Automation Letters, (44):6932-993)) 2020a.
Haoyu Wang, Xiaoyu Tan, Xihe Qiu, and Chao Qu. Subequivariant reinforcement learning framework for coordinated motion control. arXiv preprint arXiv:2403. 15100, 2024.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. In International Conference on Learning Representations, 2020b.
Yutong Wang. Bairan Xiang, Shinan Huang, and Guillaume Sartoretti. Scrimp: Scalable commu- multi-agent pathfinding. In IFEE/RS)
nication for reinforcement-and mnitation-learnnn--aar International Conference Intelligent Robots and Systems (IROS), pp. 9301-9308. IEEE, 2023.
Annie Wong, Thomas Bäck, Anna V Kononova, and Aske Plaat. Deep multiagent reinforcement learning: Challenges and directions. Artificial Intelligence Review, 56()):6023--55556 2023.
Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control moba games with deep reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6672-6679, 2020.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611 1-24224, 2022.
Yifan Zang, Jinmin He, Kai Li, Haobo Fu, Qiang Fu, Junliang Xing, and Jian Cheng. Automatic grouping for efficient cooperative multi-agent reinforcement learning. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Kaiqing Zhang, Zhuoran Yang. and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcemenl learning and control, pp. 321-384, 2021.
Lin Zhang, Yufeng Sun, Andrew Barth, and Ou Ma. Decentralized control of multi-robot system in cooperative object transportation using deep reinforcement learning. IEEE Access, 8: 8:184109- 184119, 2020.
Ming Zhang. Shenghan Zhang, Zhenjie Yang, Lekai Chen, Jinliang Zheng, Chao Yang, Chuming Li, Hang Zhou, Yazhe Niu, and Yu Liu. Gobigger: A scalable platform for cooperative-compettiv multi-agent interactive simulation. The Eleventh International Conference on Learning Repre- sentations, 2023. URL https: https:://pppeeeeeeeeeeee
Lianmin Zheng, Jiacheng Yang, Han Cai, Ming Zhou, Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many-agen reinforcement learning platform for artificial collective intelligence. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018
Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, Aurora Chongxi Huang, Ying Wen, Kimia Hassanzadeh, Daniel Graves, Dong Chen, Zhengbang Zhu, Nhat Nguyen, Mohamed Elsayed, Kun Shao, Sanjeevan Ahilan, Baokuan Zhang, Jiannan Wu, Zhengang Fu, Kasra Rezaee, Peyman Yadmellat, Mohsen Rohani, Nicolas Perez Nieves, Yihan Ni, Seyedershad Banijamali, Alexan- der Cowen Rivers, Zheng Tian, Daniel Palenicek, Haitham bou Ammar, Hongbo Zhang, Wulong Liu, Jianye Hao, and Jun Wang. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving, 2020.
15

---

Published as a conference paper at ICLR 2025
APPENDIX CONTENTS
Appendix Sections
Contents
Appendix
Evaluation Setup Details
Appendix Results for MAPF Benchmark
Appendix Results for LifeLong MAPF Benchmark
Appendix Code examples for POGEMA
Appendix POGEMA Toolbex
Appendix Extended Related Work
Appendix
Examples of Used Maps
Appendix
MARL Training Setup
Appendix
Resources and Statistics
Appendix
Community Engagement and Framework Enhancements
Appendix
Ingestion of MovingAI Maps
Appendix
POGEMA Speed Performance Evaluation
A EVALUATION SETUP DETAILS
POGEMA benchmark contains different sets of maps and all baseline approaches were evaluated on them either on MAPF on LMAPF instances. Regardless the type of instances, number of maps, seeds and agents were the same. Table contains all information about these numbers. In total, this corresponds to 3,376 episodes for each scenario type. Note that there is no MaxSteps (LMAPF) value for Cities set of maps. This set maps was used only for pathfinding meta-metric, thus all approaches were evaluated only on MAPF instances with single agent.
We used implementation LaCAM-v3', RHCR 10, For learning-basee based approaches beyond MARL, we used their official implementations and the provided weights for Follower!! MATS-LP Switcher!3 SCRIMP/4 and DCC!s
Agents
Maps
MapSize
Seeds
MaxSteps (MAPF)
MaxSteps (LMAPF)
Random
[8. 16, 24, 32, 48, 64]
128
17x17 x2
128
256
Mazes
[8. 16, 24, 32, 48, 64]
128
17x17-21xx0 x21
128
256
Warehouse
[32, 64, 96, 128, 160, 192]
33x46
128
256
Puzzles
[2, 3, 4|
5x5
128
256
Cities
[1]
256x256
2048
Cities-tiles
[64, 128, 192, 256]
128
64x64
256
256
Table 3: Details about the instances on different sets of maps.
B RESULTS FOR MAPF BENCHMARK
In this section, present the extended results of the MAPF benchmark analysis, highlighting the performance, outtof-distti handling, scalability, cooperation, coordination, and pathfinding
"utts//gittbb.co////i//laaaaa
12 Phup:://gttbbboooommmmoiiill/ees
13 http://ggthh..comm..eettatttttts
16

---

Published as a conference paper at ICLR 2025
capabilitiies of various approaches. The experiments were conducted on different map types and sizes, employing metrics such as SoC, CSR, and makespan to evaluate effectiveness. Detailed visual and tabular data illustrate how centralized and learnable approaches perform under various conditions.
Please note that the previous arXiv version contains different SCRIMP results due to technical error, which we discovered while refactoring the code for the camera-ready version.
B. PERFORMANCE
The performance metrics were calculated using Mazes and Random maps of size close to 20 x 20. The primary metrics here are SoC and CSR. The results of all the MAPF approaches over different numbers of agents are presented in Figure 4. The superior performance is shown by the centralized approach, LaCAM. Next hest-performing approach SCRIMP that is substantially outperforms another specialized solver DCC. Among the MARL methods, better results shown by MAMBA for both metrics. However, it narrowly lags behind the specialized approaches, DCC and SCRIMP.
Random / Mazes
Random / Mazes
1.0
8000
DCC
-X
IQL
0.8
DCC
LaCAM
IOL
6000
*++
MAMBA
0.6
LaCAM
QMIX
SeC
QMIX
4000
OPLEX
0.4
OPLEX
VON
SCRIMP
0.2
2000
0.0
64
16
24 32
Number Agents
Number of Agents
Figure 4: Performance of MAPF approaches on Random and Mazes maps, based on CSR (higher is better) and SoC (lower is better) metrics. The shaded area indicates 95% confidence intervals.
B.2 OUT-OF-DISTRIBUTION
nut-of-Distriiution metric was calculated on Cities-tiles dataset, that consists of pieces of cities maps with 64 x 64 size. Due much larger size compared to Mazes and Random maps, the amount of agents was also significantly increased. The results are presented in Figure 5. Here again centralized search-based planner, i.e. LaCAM, demonstrates the best results both in terms of CSR and SoC. Among hybrid methods, SCRIMP demonstrates better performance in terms of CSR, however, average SoC of its solutions almost the same solutions found by DCC. MARL approaches unable to solve any instance even with 64 agents.
Warehouse
B.3 SCALABILITY
128
DCC
The results of how well the algorithm scales with a large number of agents shown in Figure 6. The experi- ments were conducted on a warehouse map. The plot log-scaled. The best scalability achieved with the cen- tralized LaCAM approach, which is a nigh-performant approach. DCC demonstrates the worst results terms of runtime; however, regarding the Scalability metric, DCC's results are identical to SCRIMP's. Despite an initially high runtime, the scalability of MAMBA is better than other approaches; however, this could be attributed to the high cost of GPU computation, which is due the large number of parameters in the neural network and is the limiting factor of this approach.
Runtime (seconds)
32
-X
LaCAM
IQL
MAMBA
QMIX
QPLEX
SCRIMP
2
VON
32
64
96
128
160
192
Number of Agents
Figure 6: Runtime in seconds for each algorithm. The plot is log-scaled.
17

---

Published as a conference paper at ICLR 2025
Out Outto--Distribution
Out Dfst---Dttti
1.0
60000
DCC
DCC
-X-
IQL
0.8
IQL
50000
---- LaCAM
0.6
LaCAM
MAMBA
40000
QMIX
MAMBA
QMIX
30000
OPLEX
QPLEX
SCRIMP
SCRIMP
20000
0.2
VDN
10000
128
256
Number of Agents
Number Agents
Figure 5: Performance MAPF approaches on Cities-tiles maps. These results were utilized to compute Out-of- Disttiibutittii metric. The shaded area indicates 95% confidence intervals.
Algorithm
CSR
SoC
B.4 COOPERATION
DCC
0.74 0.04
93.50:10.477
How well the algorithm is able to resolve complex situations on the Puzzles set reflected in the results presented in Table 4. The only approach out of the evaluated ones that is able to solve all the instances in this set is LaCAM. Out of the rest ap- proaches, as well as on other sets of instances, the best results is demonstrated by SCRIMP, which outperforms DCC both in terms of CSR and SoC. Among MARL approaches, better cooperation demonstrated by QMIX, outperforming QPLEX, VDN, IQL, and even MAMBA.
IQL
0.47
250.03+19.09
LaCAM
1.00
20.84 1.65
MAMBA
0.00
173.60=14.04
QMIX
0.35
253.23 753.28-+16044
QPLEX
234.79-119.15
SCRIMP
0.85 + 0.03
77.86-F11.09
VDN
0.45 0.05
242. 242.422+17.92
Table 4: Comparison of algorithms' cooperation Puzzles + shows confidence inter- vals 95%. Here, tan boxes highlight the best approach, and teal boxes highlight the best ap- proach with a learnable part.
B.5 PATUFFNNDIG
To compute Pathfinding metric we the approaches the instances with a single agent. For this purpose we utilized large Cities maps with 256 x 256 size, the results are presented in Table 5. While this task seems easy, most of the hybrid and MARL approaches are not able effectively solve them. Only LaCAM is able to find optimal paths in all the cases, as utilizes precomputed costs to the goal location heuristic. Most of the evaluated hybrid and MARL approaches are also contain a sort global guidance in the channels of their observations. However, large maps with out-of-disstibution structure, the absence com- able
Algorithm
Makespan
DCC
189.56 + 26.29
IQL
1096.86 196.97
LaCAM
179.82 = 20.97
MAMBA
416.45 139.34
QMIX
1055.75 193.03
QPLEX
795.09 187.72
SCRIMP
755.98 + 183.31
VDN
1114.21 + 211.93
munication and other agents in local observations to lead to inconsistent behavior of the models that are not able to effectively choose the actions that lead the agent its goal. Please note, SoC and makespan metrics in this case equal, as there only one agent every instance.
Table 5: Comparison of makespan (the lower is better) used for pathfinding metric. tan boxes highlight the best approach, and teal boxes highlight the best approach with learnable part.
B.6 COORDINATION
The coordination metric is based on the number of collisions that occur during an episode. These collisions can occur either between agents, when two or more agents try to occupy the same cell traverse the same edge simultancously, or with static obstacles, when agent tries occupy blocked cell. All such collisions are prevented by POGEMA during the action execution process, all colliding actions replaced with waiting actions instead. Figure 7 shows the average total
18

---

Published as a conference paper at ICLR 2025
Mazes
Random
5000
-0-
DCC
DCC
4000
--- IQL
4000
-0G-
IQL
MAMBA
MAMBA
QMIX
QMIX
3000
QPLEX
Colltsions
3000
QPLEX
VDN
VON
2000
2000
1000
0
0
8
16
24
32
48
64
16
24
32
48
64
Number of Agents
Number of Agents
Figure 7: Total number of collisions occurred during solving MAPF instances with corresponding number of agents of Mazes and Random sets maps.
number collisions that occurred while solving instances with the corresponding number of agents on the Ma zes and Random map sets.
The highest number of collisions occurred with the MAMBA approach, while the fewest were with DCC. The low number of collisions made by DCC compared to other approaches can be attributed to the presence of a communication mechanism that helps avoid collisions. The results of two evaluated approaches, LaCAM and SCRIMP, were omitted and not presented in Figure LaCAM is a centralized planner, so its solutions are collision-fres by design. SCRIMP has an integrated environment with communication and tie-breaking mechanisms that resolve all collisions.
C RESULTS FOR LIFELONG MAPF BENCHMARK
In this section, we present the extended results of the LifeLong MAPF benchmark analysis, highlight- ing performance, out-of-distribution handling, scalability, cooperation, coordination, and pathfinding.
C.I PERFORMANCE
Performance metric in LMAPF case is based on the ratio of throughput compared to the best obtained one. In contrast to SoC, throughput should be as high possible. There is also no CSR metric, as there is no need for agents to be their goal locations simultaneously. As well as MAPF case, the best results are obtained by centralized search-based approach RHCR. The best results among decentralized methods demonstrate Follower and MATS-LP following them, comparable results are shown by QPLEX, QMIX, ASwitcher which significantly outperforms MAMBA both Mazes and Ra ndom maps. The results are presented in Figure
Mazes
Random
ASwitcher
ASwitcher
Follower
Throughput
Average Throughput
2.0
LSwitcher
MAMBA
MAMBA
1.5
MATS-LE LP
MATS-LP
Average
QMIX
QMIX
1.0
OPLEX
OPLEX
RHCR
RHCR
0.5
VON
VDN
16
24
32
48
64
16
24
32
48
64
Number of Agents
Number Agents
Figure 8: Performance results for LifeLong scenarios on the Mazes and Random maps.
19

---

Published as a conference paper at ICLR 2025
C.2 OUT-OF-DISTRIBUTIO
The evaluation on out-of-distribution set of maps confirms the results obtained on Random and Mazes maps. The best results demonstrates RHCR. Next best results are obtained by Follower and MATS-LP, which are much closer to RHCR in this experiment. While MATS-LP outperforms Follower on the instances with 64, 128 and 192 agents, Follower still better on the instances with 256 agents. Such relation is probably explained by the presence of dynamic edge-costs in Follower that allows better distribute agents along the map and reduce coordination between them.
Algorithm
Agents
128 Agents
192 Agents
256 Agents
ASwitcher
1.26 + 0.08
2.30 0.13
3.14 0.17
3.80++.200
Follower
1.00
0.12
3.15
4.21
IQL
0.06
0.14
0.12
0.17
LSwitcher
1.07
0.12
3.06
0.27
MAMBA
0.05
1.02
0.12
2.46
MATS-LP
0.12
0.28
4.04
0.39
QMIX
0.00
0.12
3.46+0.16
4.20
QPLEX
0.08
0.12
3.61
0.22
RHCR
1.57
0.00
4.22
0.33
VDN
1.12
0.10
2.81
2.85
Table 6: Evaluation Dut-of-Distributiion maps. tan boxes highlight the best approach according the average throughput metric, and teal boxes highlight the best approach with a learnable component.
Warehouse
C.3 SCALABILITY
4096
ASwitcher
Figure 9 contains log-scaled plot of average time spent by each of the algorithms to process instance on Warehouse map with the corre- sponding amount of agents. Most of the ap- proaches scales almost linearly, except RHCR. This centralized search-based method lacks of exponential grow, as needs to find a collision- free solution for at least next few steps, rather than just to make a decision about next action for each of the agents. The worst runtime demon- strate MATS-LP, as it runs MCTS and simulates the behavior of the other observable agents. It's still scales better than RHCR as it builds trees for each of the agents independently.
1024
Follower
Runtime (seconds)
IQL
256
LSwitcher
MAMBA
MATS-LP
QMIX
QPLEX
RHCR
1
VDN
32
64
96
128
160
192
Number of Agents
Figure 9: Runtime in seconds for each algorithm. Note that the plot is log-scaled.
C.4 COOPERATION
Algorithm
Average Throughput
As well as for MAPF setting, cooperation met- ric is computed based on the results obtained on Puzzles dataset. Table 7 contains average throughput obtained by each of the evaluated ap- proaches. Here again the best results are obtained by RHCR algorithm. In contrast to Random, Mazes and Wa Warehouse sets of maps, where MATS-LPP and Follower demonstrate close results, the ability to simulate the behavior of other agents, provided by MCTS in MATS-LP, allows to significantly out- perform Follower on small Puzzles maps. The rest approaches demonstrate much worse results, es- pecially IQL, MAMBA, VDN that have almost times worse average throughput than RHCR.
ASwitcher
0.164 0.015
Follower
0.319 0.020
IQL
0.125=0.011
LSwitcher
0.206 0.013
MAMBA
0.133 0.014
MATS-LP
0.394 0.021
QMIX
0.228 0.018
QPLEX
0.217 0.019
RHCR
0.538 0.021
VDN
0.144 0.015
Table 7: Average throughput on Puzzles maps that were used to compute Cooperation metric.
20

---

Published as a conference paper at ICLR 2025
C.5 PATHFINDING
Algorithm
Makespan
Pathfinding metric is tailored to indicate how well the algo- rithm is able to guide an agent to its goal location. As a result, there is actually no need to run the algorithms on Lifel.ong instances. Instead, they were run on the same of instances that were utilized for MAPF approaches.
ASwitcher
340.56 + 79.41
Follower
181.00 20.95
IQL
900.73 188.60
LSwitcher
472.64 11.64+1199 119.23
MAMBA
416.45 136.01
The results this evaluation are presented Table Again, the best results were obtained by search-based approach RHCR. Its implementation was slightly modified to work on MAPF instances, when there is no goal after reaching the current one. Fither optimal or close to optimal paths able find MATS-LP, Follower and QPLEX. Followers misses optimal paths due to the integrated technique that changes the edge-costs. MATS-LP adds noise the root of the search tree that might result in choosing of wrong actions. For approaches in the Switcher family, it is nearly impossible find optimal paths, as they lack information about the global map and rely solely on local observations. Surprisingly, ASwitcher outperforms MAMBA, QMIX, IQL, and VDN, which are provided with a global map.
MATS-LP
179.93 22.45
QMIX
$51.90=147.11
QPLEX
181.10 20.95
RHCR
179.82 + 20.21
VDN
1609.50 172.46
Table 8: Pathfinding results.
C.6 COORDINATION
Figure 10 illustrates the average total number of collisions that occurred during the solving of LMAPF instances. The absolute values are higher than those obtained during the solving of MAPF instances. This behavior is explained by the extended episode length in LMAPF instances, which is twice as long. Moreover, in MAPF scenarios, the episode can end when all agents reach their goal locations, whereas in LMAPF scenarios, all agents continue act until the episode length limit is reached.
All MARL approaches show poor results, with MAMBA being the worst among them. The fewest collisions are made by the Switcher approaches, i.e., ASwitcher and LSwitcher. A comparable number of collisions is demonstrated by MATS-LP and Follower on the Random set of maps. The difference in the behavior of these two approaches due to the more complex structure of obstacles on the Mazes maps, where their heuristic guidance more often leads to collisions. It's also worth noting that MATS-LP has collisions with static obstacles, as employs a masking mechanism that prevents selecting an action that leads an agent into blocked cell. Such a mechanism could be implemented in other approaches to prevent this type of collision and potentially improve their performance. The results of the RHCR approach are omitted, as is centralized planner and its solutions are guaranteed be collision-fres.
the Random and Mazes map sets is likely
Mazes
Random
5000
ASwitcher
ASwitcher
---
Follower
-X-
Follower
. IQL
... IQL
LSwitcher
8000
LSwitcher
MAMBA
MAMBA
Collisions
3000
MATS-LP
Collisions
MATS-LP
QMIX
QMIX
2000
QPLEX
OPLEX
VDN
VDN
1000
2000
8
16
24
32
48
64
16
24
32
48
64
Number of Agents
Number of Agents
Figure 10: Total number of collisions occurred during solving LMAPF instances with corresponding number of agents of Mazes and Random sets of maps.
21

---

Published as a conference paper at ICLR 2025
D CODE EXAMPLES FOR POGEMA
POGEMA is an environment that provides a simple scheme for creating MAPF scenarios, specifying the parameters GridCon gig. The main parameters are: on on_target (the behavior an agent on the target, e.g., restart for LifeLong MAPF and nothing for classical MAPF), seed to preserve the same generation of the map; agent; and their targets for scenario, s size used for cases without custom maps to specify the size of the map, density comnityya the density of obstacles, num_agents the number of agents, iios_rdduuu conflicts are handled in the environment (we used soft collision system for all of our experiments). The example of creation such instance is presented in Figure 11.
observation radius, collis sassssioonssooseeeeeeeeeeeesssse
controls how
from pogema import pogema_ vo, GridConfig, Animati iomomitoo
grid
# Define new configuration with 6 randomly placed agents grid_conffl = GridConfig (map-grid, num_agents=$0
Create custom Pogema environment. with AnimationMont env AnimationMoni tor (pogema_vv v0 (grld_ revere ()
# Saving SVG animation nev.save_animathe (offrriiionnooooaeeeeeeeoooa "four-rooms.svg"
Figure 11: Setting up a POGEMA instance with custom map and generating an animation.
Visualization of the agents is crucial tool for debugging algorithms, visually comparing them, and presenting the results. Many existing MARL environments lack such tools, or have limited visualization functionality, e.g., requiring running the simulator provide replays, or offering visualizations only in one format (such as videos). In the POGEMA environment, there are three types of visualization formats. The first one is console rendering. which can used with the default render methods of the environment; this approach is useful for local or server-side debugging. The preferred second option SVG animations. An example of generating such a visualization is presented in the listing above. This approach allows displaying the results using any modern web browser. It provides the ability to highlight high-quality static images (e.g.. g.. as the images provided in the paper) to display results GitHub). This format ensures high-quality vector graphics. The third option is to render the results to video format, which is useful for presentations and videos.
website (c. g., animations of the POGEMA repository
22

---

Published as a conference paper at ICLR 2025
E POGEMA TOOLBOX
The POGEMA Toolbox provides three types of functionality.
The first one is registries to handle custom maps and algorithms. To create custom map, the user first needs to define using ASCII symbols or by uploading from a file, and then register it using the toolbox (see Figure 13). The same approach is used to register and create algorithms (see Figure 12). In that listing, the registration of simple algorithm is presented, which must include two methods: act and reset_states, This approach also accommodate set of hyperparameters which the Toolbox handles.
from pogema import BatchAstarAgent
I Registring algorithm
wtw.a//ggsstyy..iisttttaaaaaaaa ('Ax', BatchAStarAgentt
# Creating a algerithm
algo roobooeegiistyy.rrrtee....zzoood ("aarr
Figure 12: Example of registering the A algorithm as an approach in the POGEMA Toolbox.
from Soggem__toobboe www.ooow.o.....o..ooo.ooooo stry import ToolboxRegistzy
# Creating custom map custom map m
#
Registring custom map
ToolboxRegistry. register_maps
(t" (f"custon_map" custom_mapy
Figure 13: Example of registering a custom map to the POGEMA Toolbox.
Second, it provides unified way of conducting distributed testing using Dask and defined configurations. An example of such a configuration is provided in Figure 14. The configuration is split into three main sections; the first one details the parameters of the POGEMA environment used for testing. It also includes iteration over the number of agents, seeds, and names of the map (which were registered beforehand). The unified existing parameter of the environment. The second part of the configurations is a list of algorithms to be tested. Each algorithm has its alias (which will be shown in the results) and name, which specifies the family of methods. It also includes list of hyperparameters common to different approaches, e.g., the number processes, parallel backend, etc., and the specific parameters the algorithm.
search tag allows for the examination
any
The third functionality and third part of the configuration concern views. This is form of presenting the results of the algorithms. Working with complex testing often requires custom tools for creating visual materials such plots and tables. The POGEMA toolbox provides such functionality for MAPF tasks out-of-the-box. The listing provides two examples of such data visualization: a plot and table, which, based the configuration, provide aggregations of results and present information in a high-quality form, including confidence intervals. The plots and tables in the paper are prepared using this functionality.
https:: // hww..ccom/o//..cc /dakk
23

---

Published as a conference paper at ICLR 2025
environment: Configuring Test Envi ronments
name: Environment
on on_target: rresaar'
max max_episode steps: 128
abservation_type 'POMAPF'
collision_system. 'soft'
seed:
grid_ search: [ 0, 1, 2, 3, 4, 5, 6, 7, 8,
num agents:
grid_search: [ 8, 16, 24, 32, 48, ]
map_ name:
grid search: [
validation-mazzs-aaee validation-mazes-
validation-mazes 002, validattion-azzs-00000 003,
validation-maze--00 valication-maze-O000
algorithms: # fyini a algorithms and it's hyperparameters
RHCR_ 1
name: RHCR
:aaleel 'balanced_dask'
num process: 32
simulation window: :indoo:
behaning_windowe 10 time :ime_iimit:
low jww_level_planerr planner: 'STPP'
solver: 'PBS'
results views:
results_views: & Defining results visualization
MazesPlot:
type: plot
num_agents
avg throughout
width: 4.0
height: 3.1
line width: line_width:
use see_log_scaae scale x: True
legend font_size:
font_size: 8
name: Mazes
ticks: [8, 16, 24, 32, 48, 64]
Tabularfhroughutt:
type: tabular
print_results: True
Example of the POGEMA Toolbox configuration for parallel testing of the RHCR approach and visualization of its results.
Figure 14:
F EXTENDED RELATED WORK
StarCraft Multi-Agent Challenge - The StarCraft Multi-Agent Challenge (SMAC) is highly used benchmark in the MARL community. Most MARL papers that propose new algorithms provide evaluations in this environment. The environment offers large set of possible tasks where group of units tries to defeat another group units controlled by bot (a predefined programmed policy). Such tasks partially observable and often require simple navigation. However, the benchmark has
24

---

Published as a conference paper at ICLR 2025
several drawbacks, such as the need to use the slow simulator of the StarCraft II engine, deterministic tasks, and the lack of an evaluation protocol.
Nevertheless, some of these drawbacks have already been addressed. SMAX Rutherford et al. (2023) provides a hardwarr--ccellertted JAX version of the environment, but it cannot guarantee full compatibility since the StarCraft II engine is proprietary software. SMAC v2 Ellis et al. (2024) solves the problem of determinism, highlighting this issue in the original SMAC environments. Moreover, an evaluation protocol for the SMAC environment is proposed in Gorsane et al. (2022). Despite these efforts, it's hard to say that these tasks require the generalization ability of the agent, since the training and evaluation are conducted on the same scenario.
Multi-agent MuJoCo- - In MAMuJoCo, the standard tasks involve agents controlling different sets of joints (or a single joint) within simulated robot. This set of environments is a natural adaptation of the environment presented in the well- MuloCo physics engine Todorov et al. (2012). These tasks don't require high generalization abilities or navigation. In the newer version, MuJoCo provides a serddwrr--aceleratee version, forming the basis for Multi-agent BRAX Rutherford et al. (2023), which enhances performance and efficiency.
Google Research Football Google Research Football Kurach et al. (2020) is a multi-agent football simulator that provides a framework for cooperative or competitive multi-agent tasks. Despite the large number of possible scenarios in the football academy and the requirement for simple navigation, the tasks are highly specific to the studied domain. Additionally, the number of possible agents is limited. Moreover, the framework offers low scalability, requiring heavy engine.
Multi-robot warehouse - The multi-robot warehouse environment RWARE Papoudakis et al. (2021) simulates a warehouse with robots delivering requested goods. The environment is highly specific to delivery tasks; however, it doesn't 't support procedurally generated scenarios, thus not requiring generalization abilities or evaluation protocol. The best-performing solution Christianos et al. (2020) in this environment is trained on only 4 agents. The benchmark is highly related to multi-agent pathfinding tasks; however, doesn't 't provide centralized solution integration, which could serve both as an upper bound for learnable decentralized methods and as a source of expert demonstrations.
Level- Based Foraging Multi-agent environment LBF Papoudakis et al. (2021) simulates food collection by several autonomously navigating agents in grid world. Each agent is assigned level. Food is also randomly scattered, each having level on its own. The collection of food is successful only if the sum of the levels of the agents involved in loading is equal to or higher than the level of the food. The agents are getting rewarded by level of food they collected normalized by their level and overall food level of the episode. The game requires cooperation but also the agents can emerge competitive behavior. The environment very efficiently designed and very simple to set up: however, it doesn't support procedurally generated scenarios, thus not requiring generalization abilities evaluation protocol.
Flatland - The Flatland environment Mohanty et al. (2020) is designed to address the specific problem of fast, conflict-free train scheduling on a fixed railway map. This environment was created for the Flatland Competition Laurent et al. (2021). The overall task is centralized with full observation; however, there is an adaptation partial observability for RL agents. Unfortunately, during several competitions, despite the presence of stochastic events, centralized solutions Li et al. (202 (0) from operations research field have outperformed RL solutions by a large margin in both quality and speed. The environment is procedurally generated, which requires high generalization abilities, and the benchmark provides an evaluation protocol. significant drawback is the extremely slow speed of the environment, which highly restricts large-scale learning.
The Overcooked a benchmark environment for fully cooperative human-Al task performance, based on the widely popular video game (Carroll et al., 2019). In the game, agents control chefs tasked to cook some dishes. Due to possible complexity of the cooking process, involving multistep decision-making it requires emergence of cooperative behavior between the agents.
Overcooked
Griddly This is a grid-based game engine (Bamford, 2021), allowing make various and diverse grid-world scenarios. The environment is very performance efficient, being able to make thousands step per second. Moreover, there is test coverage and continuous integration support, allowing
25

---

Published as a conference paper at ICLR 2025
open-source development. The engine provides support for different observation setups and maintains state history, making it useful for search based methods.
Multi-player game simulators - Despite the popularity of multi-player games, it's challenging problem to develop simulators of the games that could be used for research purposes. One of the most popular adaptations are MineCraft MALMO (Johnson et al., 2016) that allows to utilize MineCraft as a configurable research platform for multi-agent research and model various agents' interactions. In spite of the game's 's flexible functionality, it depends on external runtime, so might be very hard to set up or extremely slow to iterate with. That's why there are several alternatives that prioritize fast iteration over the environment complexity. like Neural MMO Suarez et al. (2024) that models a simple MMO RPG with agents with a shared resource pool. On top of that, there are even faster implementation, targeting coordination or cooperation, like Hide-and-Seed Baker et al. (2020), which models competition, or GoBigger Zhang et al. (2023), focusing on competition between cooperating populations.
Magent- Is well-recognized environment within the community, though less widely used than SMAC. However, lacks procedural generation capabilities, essential for testing agent generalization. The benchmark consists of six static scenarios, with the largest agent population observed in the Gather scenario, supporting up to 495 agents. The scenarios include Adversarial Pursuit, where red agents navigate obstacles to tag blue agents without causing damage; Battle, large-scale team battle rewarding individual agent performance: Bautlefiel, similar to Battle but with fewer agents, but with predefined obstacles the field; Combined Arms, team battle featuring ranged and melee units with differing attack ranges, speeds, and health points; Gather, where agents compete for finite food resources requiring multiple "attacks" consume; and Tiger-Deer, in which tigers team up hunt deer, earning rewards through successful cooperation.
Gigastep A GPU-accelerated multi-agent benchmark that supports both collaborative and ad- versarial tasks, continuous and discrete action spaces, and provides RGB image and feature vector observations. It features diverse set of scenarios with heterogeneous agents, asymmetric teams, and varying objectives (e.g., tagging, waypoint following. snde-e.d-seek).. However, it lacks procedural generation, which is crucial for testing generalization. Its evaluation protocol minimal, relying on comparisons between PPO and hand-designed bots, offering limited insights into algorithmic diversity.
Multi-agent Driving Simulators - Autonomous driving is one of the important practical applica- tions of MARL, and Nocturne Vinitsky et al. (2022) is a 2D simulator, written in C++, that focuses on different scenarios of interactions c.g. intersections, roundabouts etc. The simulator is based on trajectories collected in real life, allowing it to model practical scenarios. This environment has evaluation protocols and supports open-source development with continuous integration and covered by tests. There are also environments, focusing on particular details of driving, for example, Multi-car Racing Schwarting et al. (2021) that represents racing from bird's eye view.
Suits of multi-agent environments - These multi-agent environments are designed to be very simple benchmarks for specific tasks. Jumanji Bonnet et al. (2023) is set of environments for different multi-agent scenarios connected to combinatorial optimization and control, for example, routing or packing problems. With the purpose for each environment be focused on the particular task, the overall suit doesn't 't test generalization or enable procedural generation. Multi Particle Environments (MPE) Lowe et al. (2017) is communication oriented set of partially observable environments where particle agents are able to interact with fixed landmarks and each other, communicating with each other. SISL Gupta et al. (2017) is a set of three dense reward environments was developed to have a simple benchmark for various cooperative scenarios. For environment suits, testing generalization, MeltingPot Agapiou et al. (2022) comes into place. This set of the environments contains a diverse set of cooperative and general-sum partially observable games and maintains two populations of agents: focal (learning) and visiting (unknown to the environment) to benchmark generalization abilities of MARL algorithms. The set in based on the own game engine and might be extended quite casily.
Real-world Engineering in Practice Real-world engineering tasks can often be addressed by MARL solutions. IMP-MARL Leroy et al. (2024) provides a platform for evaluating the scalability of cooperative MARL methods responsible for planning inspections and repairs for specific system components, with the goal of minimizing maintenance costs. At the same time, agents must cooperate
26

---

com jww.ijkzzzziccc pymar12 Published as conference paper at ICLR 2025
to minimize the overall risk of system failure. MATE Pan et al. (2022) addresses target coverage control challenges in real-world scenarios. It presents an asymmetric cooperative-oompetie game featuring two groups of learning agents, cameras and targets, each with opposing goals.
G EXAMPLES OF USED MAPS
The examples of used maps are presented in Figure 13, showing a diverse list maps. The map types used in the POGEMA Benchmark include: Mazes, with prolonged 1-cell width corridors requiring high-level cooperation; Random, easily generated maps to avoid overfitting with con- trollable obstacle density; Cities-tiles, smaller modified slices of Cities maps; Puzzles, small hand-crafted maps with challenging patterns necessitating agent cooperation; Warehouse, widely used in LifeLong MAPF research, featuring high agent density and throughput challenges; and Cities, large maps with varying structures for single-agent pathfinding.
600
Mazes
(b) Random
Citiess-tiees
Puzzles
(e) Warehouse
( Cities
Figure 15: Examples of maps presented in the POGEMA Benchmark. The city map (on which the pathfinding metric was tested) is shown without grid lines and agents for clarity.
H MARL TRAINING SETUP
Hyperparameter
IQI.
QPLEX
QMIX
VDN
For training MARL approaches such as QMIX, QPLEX, IQL,, and VDN, we started based on the default hyperparameters pro- vided in the corresponding repositories, employing the PyMARL2 framework!? These hyperparameters are mostly tuned for the SMAC environment, so we tuned main ones for our use case. For this hyperparameter sweep, we used grid search over parameters such learning rate, batch size, replay buffer size, and neural network parameters like the size of RNN blocks. We used the default functionality of the Wandb framework 18 for this sweep, with the optimization target being the CSR of the agent the training maps. The best found hyperparameters
Batch Size
64
64
32
32
Learning Rate
0.002
0.002
0.001
0.001
RNN Size
128
128
256
256
Table 9: Best hyperparameters found by hyperparameter sweep, which different from defaults ones.
"https: / thu.. wtt. wwto///wwn///////// wttp////ddbbw/dbbbbbbeeee wandb
27

---

Published as a conference paper at ICLR 2025
which is different from default ones presented in Table 9. We used the default hyperparameters for MAMBA, provided corresponding repository"".
As imput, apply preprocessing from the Follower approach, which is the current state-of- of-thh--rt-tttt for decentralized LifeLong MAPF. We attempted add a ResNet encoder, as used in the Follower approach; however, this addition worsened the results, thus opted for vectorized observation and default MLP architectures. For centralized methods that work with the state the environment (e.g., QMIX or QPLEX), we utilized the MARL integration of POGEMA, which provides agent positions, targets, and obstacle positions in a format similar to the SMAC environment (providing their coordinates).
Our initial experiments on training this approach with large number of agents, similar to the Follower model, showed very low results. We adjusted the training maps be approximately 16 16, which proved to be more effective and populated them with 8 agents. All the MARL approaches were trained using the Ma zes map generator. This setup produced better results. We continued training the approaches until they reached a plateau, which for most algorithms is under I million steps.
I RESOURCES AND STATISTICS
To evaluate all the presented approaches integrated with POGEMA we have used two workstations with equal configuration, that includes NVidia Titan GPU, AMD Ryzen Threadripper 3970X CPU and 256 GB RAM. The required computation time is heavily depends the approach by itself.
Random
Warehouse
Cities-tille
Puzzles
Cities
DCC
2.11
2.46
11.07
22.70
0.09
0.02
IQL
0.05
0.04
0.13
0.13
LaCAM
0.20
0.29
0.24
0.23
0.37
0.01
MAMBA
6,62
6.47
8.36
12.27
2.59
3,40
QMIX
0.04
0.04
0.14
0.13
0.01
0.01
QPLEX
0.05
0.04
0.13
0.13
0.01
0.01
SCRIMP
1.66
2.20
16.54
21.63
0.08
0.21
VDN
0.05
0.04
0.13
0.13
0.01
0.01
Table 10: Total time (in hours) required by each of the algorithms to run all MAPF instances on the corresponding datasets.
Random
Mazes
Warehcuse
Cities-tiles
Puzzles
Cities
ASwitcher
1.03
0.47
2.95
1.76
0.31
0.04
Follower
0.48
0.23
0.69
0.77
0.26
0.89
IQL
0.08
0.04
0.26
0.02
0.01
LSwitcher
6.18
2.61
17.30
10.70
0.81
0.21
MAMBA
13.82
6.69
15.81
11.07
7.83
3,40
MATS-LP
77.31
35.34
163.68
129.78
3.80
0.14
QMIX
0.08
0.04
0.26
0.25
0.02
0.01
QPLEX
0.08
0.04
0.26
0.25
0.02
0.01
RHCR 0.57
0.25
17.04
6.28
0.01
0.01
VDN
0.08
0.04
0.25
0.25
0.02
0.01
Table 11: Total time (in hours) required by each algorithms to run all LMAPF instances on corresponding datasets.
The statistics regarding the spent time on solving MAPF and LMAPF instances are presented in Table 10 and Table 11 respectively. Please note, that all these approaches were run parallel multiple threads utilizing Dask, that significantly reduces the factual spent time.
We used pretrained models for all the hybrid methods, such as Follower, Switcher, MATS-LP, SCRIMP, and DCC, thus, no resources were spent on their training. RHCR and LaCAM are pure
Phttps://gu thub. com jor-aarrrr 1abs /mabaa
28

---

Published as a conference paper at ICLR 2025
search-basek based planners and do not require any training. MARL methods, such MAMBA, QPLEX, QMIX, IQL, and VDN, were trained by us. MAMBA was trained for 20 hours on the MAPF instances, resulting in 200K environment steps, and for days LifeLong MAPF instances, resulting in 50K environment steps, which corresponds to the same amount of GPU hours. For MARL. approaches, trained them for million environment steps, which corresponds average of 5 GPU hours for each algorithm.
J COMMUNITY ENGAGEMENT AND FRAMEWORK ENHANCEMENTS
Our team committed maintaining an open and accountable POGEMA framework. We ensure transparency in operations and encourage the broader AI community to participate. Our frame- work includes fast learning environment, problem instance generator, visualization toolkit, and automated benchmarking tools, all guided by clear evaluation protocol. We have also implemented and evaluated multiple strong baselines that simplify further comparison. We practice rigorous software testing and conduct regular code reviews.
INGESTION OF MOVINGAI MAPS K
We incorporated an ingestion script to convert MovingAl maps to be compatible with POGEMA. The ingestion script is straightforward. and an example of its usage presented Figure 16. This script downloads the full archive of, in this case, the street-map series, which will be saved to a YAMI. file compatible with POGEMA.
from pogema_toclbox, wewprbtorggeeratrr...ooo. generator_utiln import maps_dict_to_yaal yaml)
from pogema_toclbbx download_movingg_ii_nno
www.bboo..evingggaa....a inggtiiinnnnionnn import (
url httpp:///oooiigga..c/////eeadees maps wwwnloaddmoving..oomm maps (url)
mipp
maps diet_ sriic__t_yymmll ('maps, yaml', ' maps)
Figure 16: Ingestion script convert MovingAl maps into POGEMA-compatibe format
Instead of converting and distributing these maps ourselves, we provide this script to allow users to convert the maps their own, ensuring compliance with licensing terms. Additionally, by using this script, users can always work with the most up-to-date versions of the maps in the MovingAI dataset, addressing any changes updates made to the dataset over time.
L POGEMA SPEED PERFORMANCE EVALUATION
The speed of an environment is a critical aspect in reinforcement learning (RL), significantly influ- encing training performance and usability. Table provides an overview of the speed performance of various multi-agen environments, demonstrating that POGEMA can process more than 10K steps per second.
Here we provide more detailed information about POGEMA's speed performance. POGEMA's continuous integration (CI) includes speed measurement procedure that runs alongside the tests. Here, present the results from recent CI run. We compared POGEMA's performance three CPU setups: the AMD Ryzen Threadripper 3970X 32-Core Processor, a modern high-performance CPU: an older server-side Intel(R) Xeon(R) CPU @ 2.20GHz, commonly used the Google Colab platform, representing an average-perte prrrggepeerfommacc setup: and the Apple Silicon M1, a laptop setup that widely regarded for its power efficiency and solid performance in computational tasks.
For this test, we used default observation parameters commonly employed in learnable multi-agent pathfinding (MAPF) approaches (e.g., Follower, SCRIMP, DCC). Specifically, the observation
29

---

Published as a conference paper at ICLR 2025
Ryzen Threadripper
Intel(R) 2.20GHz
Apple Silicon MI
Agents
Size
MAPF
IMAPF
MAPF
LMAPF
MAPF
IMAPF
1
32
20 684
21 810
9391
13358
29 699
44224
64
10390
9602
6981
6452
19244
999
32
32
96918
90 631
61 132
61 232
204637
191 069
32
64
89 984
847
61 297
38326
144
175 962
64
32
111 976
105 558
69 121
39
189 624
217 482
64
64
102 104
96 709
68 126
49 902
196 808
194
Table 12: POGEMA performance in observations per second (OPS) across different CPU types using a single CPU core. Note that the reported OPS represents the total frames (observations) received by all agents, the environment steps.
radius to 5, corresponding to an 11 X 11 field. The results for both MAPF and LMAPF scenarios, using a random policy, are reported Table 12. In scenarios with more than 32 agents, POGEMA achieves 2 80K steps per second (OPS) on fast CPUs like the Ryzen Threadripper and Apple Silicon MI, and 2 38K OPS on the Intel Xeon setup, which is notably fast for a sinleeenvironment, single-thread configuration. For comparison, EnvPool reports 50K frames per second (FPS) for Atari on 12-CPU setup.
We further investigate POGEMA's speed performance using SampleFactory" as the sampler for parallel asynchronous execution of the environment. We used an almost default configuration with random policy (instead of PPO) and employed two environments per worker (for double buffering Samplefactory). The number of sampling workers varied alongside the number of agents in on workstation equipped with single AMD Ryzen
the environment, and the tests Threadripper 3970X 32-Core Processor (64 threads). We conducted a sampling procedure for minutes for each setup using Random maps with size of 32 x 32, The results both MAPF and Lifelong MAPF scenarios are presented in Figure 17.
Observations per second for MAPF
b) Observations per second for Lifelong MAPF le6
3.0
Number of agents
2.5
2.0
2.0
1000 Somtin de
142k
282K
2.0M
1.5
120k
1.7M
1.0
1.0
11k
21k
42k
82k
154k
179k
0.5
8k
37k
33k
64k
120k
145k
32
Number workers
Number workers
Figure 17: Observations per second performance of POGEMA for (a) MAPF and (b) Lifelong MAPF across different number of workers and agents in the environment, using AMD Ryzen Threadripper 3970X 32-Core Processor (64 threads). We tested Random maps with size 32 x 32.
Looking at the single- worker setup. it is notable that the performance from the previous experiment with a single CPU setup is noticeably higher. This slowdown can be attributed to the overhead produced by the parallel asynchronous execution of the framework. However, it allows to significantly improve performance; the results for the best configuration with 64 agents and workers achieved 3. OPS for MAPF and 2.7M OPS for Lifelong MAPF. For both setups, only 16 workers needed to exceed the significant threshold of IM OPS. The best performance is observed the MAPF scenario, where 916,464.000 samples were generated in 5 minutes. This demonstrates the efficiency of utilizing a high number of CPU cores for large-scale sampling tasks, which scaled almost linearly up to 32 workers. This also indicates that performance can be further improved with additional CPU resources.
"https://githbb.cr http://......oaaaag////////eaa/p
httpps: http://ggtttbbcccce thub. / hts//aleexeppee -eex-pttrenko// samp' factory
30

---

Published as a conference paper at ICLR 2025
To compare, refer to the JaxMARL paper, which provides insights into the speed performance of XLA-accelerated environments. The repository includes several environments, and we will focus on the SPS of the STORM environment, which offers grid-base bad-based tasks. Based on the paper, the speed of the environment is 2,48k with a single environment, 175k for 100 environments, and 14.6M SPS for 10,000 environments. These results were obtained using a single NVIDIA A100 GPU.
While, as expected, POGEMA is slower in very large vectorized setups compared to XLA vectorized environments that use GPU or TPU hardware acceleration, this trade-off has its advantages. It is challenging to devise an approach that can effectively utilize such a large amount of data. Additionally, by not relying on GPUs or TPUs for environment simulation, these resources remain fully available for training neural networks, which often represent the primary bottleneck in large-scale RL experiments.
We further evaluate the scalability POGEMA with large population of agents. For this purpose. we use random map scenario of size 3072 x 3072 and test agent counts starting from 1,000 agents. The results, presented in Table 13, demonstrate that POGEMA efficiently supports up to million agents within single environment.
Agents
OPS
SPS
Reset (seconds)
1,000,000
68700.6
0.0687
173.8
100,000
67104.9
0.6710
139.6
10,000
45894.7
4.589
132.8
1,000
56477.7
56.477
120.6
Table 13: Speed performance of POGEMA with large agent populations using a random policy.
In previous experiments, we relied on a random policy to evaluate the environment's 's speed perfor- mance, as it is a common choice for such tests. However, the speed performance of the environment may vary when a more advanced approach is used, as it explores a larger portion of the state space compared to the random policy. To better test the speed performance of POGEMA under real inference, we used the Follower approach and LifeLong Mazes scenarios, with a size of 128 x 128 and up to 2048 agents, they required creating new goals for agents upon reaching them. The results are presented in Figure 18. Here, one can see that with an increasing number of agents, the SPS of POGEMA decreases almost linearly. Additionally, the OPS throughput grows with the number of agents. This setup also highlights the ability of POGEMA to handle a large population of agents operating in the same environment. The experiment was conducted on setup with an AMD Ryzen Threadripper 3970X 32-core processor, using single CPU core and single environment (no parallelization.
Maps
Maps
POGEMN Deservatios per Second
256
Follower
40860
Follower
per
128
32768
24576
64
128
256
512
1024
2048
64 128 256 512 1024 2048
Agents
Agents
Figure 18: Steps per second and observations per second throughput of POGEMA with a large agent population using the Follower approach (without parallelization). Both are log scale.
31


Reviews:
[{"review_id": "ad0132tohh", "reviewer": "Reviewer_mDvH", "summary": "This paper introduces POGEMA, a benchmark for multi-agent pathfinding (MAPF) and its lifelong version (LMAPF) on grids. The contributions include a fast CPU-based environment, problem generators, visualization, and benchmarking tool for learnable, hybrid, and purely learned approaches.", "questions": "1. One problem I have about the paper is more on the conceptual level, i.e. why the “PO” in POGEMA. “PO” stands for partially observable, and this makes sense in unknown environments that need exploration, such as the mentioned Nocturne benchmark. However, in the case of the proposed benchmark, I do not see why one should limit the observability as partial. Indeed, in my experience, the (L)MAPF problem arises in industrial settings in which maps are known a priori, which motivates the use of global heuristic controllers such as Lacam and RHCR. Why is partial observability used in such discrete settings, while SOTA heuristic approaches don’t use such a notion? Could leveraging full maps improve the performance of underperforming neural approaches?\n    \n2. What is the impact of sparse vs dense rewards in MARL for MAPF? Is it better to give +1 only at the end of the episode or a dense reward at each step?\n    \n3. In terms of conclusions for such a benchmark, it would be more interesting to identify the shortcomings of current models and help identify future research directions. What do you think these could be?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The writing of this paper is crisp, and the visualizations are of excellent quality. The authors provide several examples and extensive code. While this is not the first paper to focus on MARL navigation, it is the first to fully focus on MAPF variants in a single repository. The proposed metrics are a much-needed tool to assess the performance of (L)MAPF approaches, not just based on the classical SoC/throughput but also on other metrics that can identify possible research directions.", "weaknesses": "1. I believe the title and abstract are misleading about the scope of the paper: while POGEMA appeals to a broad audience of MARL-based navigation in title and abstract, in fact it is about two variants of MAPF on discrete grids with simplified settings. For instance, in terms of MAPF, continuous variants like [1] does not appear to be considered. Moreover, there does not seem to be any mention about any-angle versions that would make the problem more realistic and interesting like [2].\n    \n2. One main concern about this paper is that while authors introduce a new MARL environment, they mostly perform inference with it and several results are not reproduced by retraining models. For instance, the DCC and SCRIMP codes do not contain any training script or guidance on how to reproduce results. I believe comparing training performance or speed with the proposed environment against the original implementation would strengthen the paper.\n    \n3. Authors claim scalability of >1000 agents. I cannot find any result to substantiate this claim.\n    \n4. The LaCAM version the authors mention refers to the first publication. However, new variants were released open source, i.e. LaCAM3 [3]. Moreover, versions as [4] can solve scales up to 10k agents in seconds.\n    \n5. Authors acknowledge the limitation of lack of JAX support and GPU-based parallelization tools. In this aspect, I believe the proposed (L)MAPF environments could be created by quite simple modifications of existing environments in Jumanji as the RobotWarehouse ([https://instadeepai.github.io/jumanji/environments/robot_warehouse/](https://instadeepai.github.io/jumanji/environments/robot_warehouse/)).\n    \n6. While this is a relatively minor point, I believe there are quite a few problems in terms of writing for the related work. Firstly: I don’t see why the paper should mention all multi-agent benchmarks, when the considered problems are only two very specialized ones. In this sense, Table 1 should contain information about “topic” or “area”, which is missing. GPU parallelization is not considered in the table. The main issue here is that it appears that POGEMA solves all the issues of previous MARL benchmarks such as Nocturne, while they solve arguably much simpler problems in restricted settings. Finally: I don’t see why there is a need to explain every single component as a separate paragraph for a total of almost 3 pages, such as “Python-based” and “PyPI listed\" .\n\n7. Finally, as a benchmark, it would be nice to include insights, i.e., what are possible future directions of research. Looking at the graph, one might conclude that there is no point in conducting research in MARL+MAPF, given that heuristic approaches perform well in all metrics, except scalability, only for the LMAPF case.\n    \n\n[1] Andreychuk, Anton, et al. \"Multi-agent pathfinding with continuous time.\" Artificial Intelligence 305 (2022): 103662.\n\n[1] Yakovlev, Konstantin, Anton Andreychuk, and Roni Stern. \"Optimal and Bounded Suboptimal Any-Angle Multi-agent Pathfinding.\" Proceedings of the International Symposium on Combinatorial Search. Vol. 17. 2024.\n\n[3] Okumura, Keisuke. \"Engineering LaCAM\\*: Towards Real-time, Large-scale, and Near-optimal Multi-agent Pathfinding.\" Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems. 2024.\n\n[4] Okumura, Keisuke. \"Improving lacam for scalable eventually optimal multi-agent pathfinding.\" arXiv preprint arXiv:2305.03632 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B8CWSYoePT", "reviewer": "Reviewer_iVSi", "summary": "This paper presents a benchmark for multi-agent navigation tasks. The overall work includes a fast and scalable environment for multi-agent navigation tasks, a series of built-in tasks, a set of visualization tools, and an evaluation framework.", "questions": "In addition to my primary concerns, there are also the following questions: \n1. Please carefully check the writing of the paper. For example, on the first line of the third page, \"both\" needs to be removed.\n2. Is it \"NP-Hard\" or \"HP-Hard\" in line 88 of the paper? \n3. The definitions of competitive and cooperative behaviors in lines 221-225 might not be rigorous enough. For instance, in competitive behaviors, the rewards of agents might not sum up to a fixed value but could have a relationship akin to being inversely proportional; in cooperative behaviors, agents do not necessarily share rewards, but they are working towards the same task. Do the authors have a more rigorous perspective on this?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The overall platform proposed in this paper is capable of integrating search-based, learning-based, and hybrid approaches together, using the same metric for comparison.\n2. As an environment that supports large-scale multi-agent reinforcement learning algorithm training, performing more than 10K steps per second is quite remarkable.", "weaknesses": "(Major) The primary issue with this paper is that its contributions are not sufficiently prominent. As a benchmark in the multi-agent systems domain, POGEMA's positioning is unclear and lacks irreplaceable advantages. For instance, for a researcher in the MARL domain, it may mainly offer features related to partial observability, rapid training, and scalability to >1000 agents. However, benchmarks already exist in the MARL domain that possess these characteristics, such as [1][2], making this work not irreplaceable. For a researcher interested in the domain of multi-agent cooperative navigation, the overall scenario elements of this work are overly simplified and idealized, presenting a significant gap from real-world scenarios. It might be better to focus more on issues in real-world applications, such as unexpected situations in open environments and the transferability of algorithms from simulation to reality.\n\n\n(Minor) There are concerns and suggestions regarding the writing of the introduction part. For example, the content about decentralized cooperative learning discussed in lines 53-58, is it closely related to the necessity of this work? Similarly, the transition in lines 59-77 related to partial observability and large-scale agents is too lengthy and seems not to be the focus of the problems the paper intends to solve. A well-written introduction should quickly highlight contradictions and then state the necessity of the work. Writing too much content with no clear relevance can impact readability.\n\n[1] Lechner M, Seyde T, Wang T H J, et al. Gigastep-one billion steps per second multi-agent reinforcement learning[J]. Advances in Neural Information Processing Systems, 2024, 36.\n\n[2] Zheng L, Yang J, Cai H, et al. Magent: A many-agent reinforcement learning platform for artificial collective intelligence[C]//Proceedings of the AAAI conference on artificial intelligence. 2018, 32(1).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4P45u4Coyg", "reviewer": "Reviewer_Jpm1", "summary": "The paper proposes a new benchmark for multi-agent path-planning and reinforcement learning. The main contribution of the new benchmark appears to be its ability to be computationally efficient, the ability to procedurally generate new problems, and to allow a diverse set of map styles. The tasks are generally navigation tasks, in which an agent has to avoid collisions and reach a goal. The paper presents sufficient details on the new benchmark and compares a large number of different MARL and other multi-agent path planning algorithms.", "questions": "Feel free to respond to my concerns raised in the weaknesses. Overall, I think the implementation work seems good, the evaluation of algorithms extensive, but I fail to see the truly novel aspect where this will raise new open questions for multi-agent planning, new avenues for research or challenge the existing algorithms. \n\nHence, a rating of 6 may be the highest I am willing to give the paper in its current state.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- I like the extensive comparison and differentiation to previous multi-agent environments\n- I appreciate the large number of evaluated algorithms on the new benchmark\n- The analysis and experimental studies in this paper are strong\n- The new environment seems to be in particularly interesting when evaluating a large number of agents in large environments\n- I like that the authors promise to release the code under an open-access license.", "weaknesses": "- While I appreciate the computational efficiency and the ability to procedurally generate new environments, I am wondering after reading the paper if the benchmark can further the field by sparking new ideas or raising open problems. The benchmark seems more like an engineering achievement allowing the study of larger agent populations, but I am wondering if this is really _the_ open problem we need to consider in MARL.\n- Similarly, the observation space seems relatively simple. I may have missed it, but there is also no uncertainty or noise in the sensor measurements. Again, this reinforces the impression that the introduced task-sets are relatively simple, I am failing a bit to see that this will truly push forward the field.\n- This is due to the fact that the considered environments are, in principle, simple, as they are _just_ navigation tasks requiring a minimalistic representation. Again, I worry this is simply a too-simple set of tasks to further the research frontier...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)