SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning
Authors: Benjamin Ellis
Jonathan Cook
Skander Moalla
Mikayel Samvelyan
Mingfei Sun
Anuj Mahajan
Jakob Nicolaus Foerster
Shimon Whiteson
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Multi-Agent Reinforcement Learning
TL;DR: Demonstrate that the popular benchmark the Starcraft Multi-Agent Challenge lacks stochasticity and propose a new version, SMACv2 to address this shortcoming.

Abstract: The availability of challenging benchmarks has played a key role in the recent progress of machine learning. In cooperative multi-agent reinforcement learning, the StarCraft Multi-Agent Challenge (SMAC) 
has become a popular testbed for centralised training with decentralised execution. However, after years of sustained improvement on SMAC, algorithms now achieve near-perfect performance. In this work, we conduct new analysis demonstrating that SMAC lacks the stochasticity and partial observability to require complex *closed-loop* policies. In particular, we show that an *open-loop* policy conditioned only on the timestep can achieve non-trivial win rates for many SMAC scenarios. To address this limitation, we introduce SMACv2, a new version of the benchmark where scenarios are procedurally generated and require agents to generalise to previously unseen settings (from the same distribution) during evaluation. We also introduce the extended partial observability challenge (EPO), which augments SMACv2 to ensure meaningful partial observability. We show that these changes ensure the benchmark
requires the use of *closed-loop* policies. We evaluate state-of-the-art algorithms on SMACv2 and show that it presents significant challenges not present in the original benchmark.  Our analysis illustrates that SMACv2 addresses the discovered deficiencies of SMAC and can help benchmark the next generation of MARL methods. Videos of training are available on our [website](https://sites.google.com/view/smacv2).

SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning
Benjamin Ellis! Jonathan Cook' Skander Moalla 1,5 Mikayel Samvelyan 2.3 Mingfei Sun Anuj Mahajan' Jakob N. Foerster' Shimon Whiteson'
Unniversity of Oxford Univeesityy College London Meta AI *University of Manchester SEPFI
Abstract
The availability of challenging benchmarks has played key role in the recent progress of machine learning. In cooperative multi-agent reinforcement learning, the StarCraft Multi-Agent Challenge (SMAC) has become popular testbed for the centralised training with decentralised execution paradigm. However, after years of sustained improvement on SMAC, algorithms now achieve near- rear-perfect performance. In this work, we conduct new analysis demonstrating that SMAC lacks the stochasticity and partial observability to require complex closed-loop policies (i.e., those that condition the observation). In particular, we show that an open-loop policy conditioned only on the timestep can achieve non-trivial win rates for many SMAC scenarios. To address this limitation, we introduce SMACv2, a new benchmark where scenarios are procedurally generated and require agents to generalise to previously unseen settings during evaluation We show that these changes ensure the benchmark requires the use of closed-loop policies. We also introduce the extended partial observability challenge (EPO), which augments S.MAc22 to ensure meaningful partial observability. We evaluate state-of-the algorithms SMACv2 and show that it presents significant challenges not present in the original benchmark. Our analysis illustrates that SMACv2 addresses the discovered deficiencies of SMAC and can help benchmark the next generation of MARL methods. Videos of training are available on our website
Introduction
many real-world cases, control policies for cooperative multi-agent reinforcement learning (MARL) can be learned in setting where the algorithm has access to all agents' observations, e.g., in must be deployed where such centralisation not possible. Centralised training with decentralised execution (CTDE) 10 25 is paradigm for tackling these settings and has been the focus of much recent research [32 44 12 26 (2). CTDE, the learning algorithm sees all observations during training, as well as possibly the Markov state, but must learn policies that can be executed without such privileged information.
simulator or laboratory,
As in other areas of machine learning, benchmarks have played an important role in driving progress CTDE. Examples of such benchmarks include the Hanabi Learning Environment (2]. Google football 20). PettingZoo [46), and Multi-Agent Mujoco 31. Perhaps the most popular one is the StarCraft Multi-Agent Challenge (SMAC) (37, which focuses on decentralised micromanagement challenges in the game of StarCraft II. Rather than tackling the full game with centralised control (50). SMAC tasks a group of learning agents, each controlling a single army unit, to defeat the units of the
Coorespondence bencllis@robots.ox.ce
"Code available at https::://// //gi thub. cot./o/ohrrr1///maaac
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

(a) Terran
Zerg
(c) Protoss
Figure I: Screenshots from SMACv2 showing agents battling the built-in AL
enemy army controlled by the built-in heuristic AI. SMAC requires learning several complex joint action sequences such focus fire Pl and kiting enemy units For these reasons, many prominent
MARL papers [51 26 52 33
9 55
53
rely on SMAC to benchmark their performance!
However, there is significant evidence that SMAC is outliving its usefulness [55] 13 II]. Recent work reports near- near-perfect win rates on most scenarios (55] [3] and strong performance without using the centralised state via independent learning [55 g This suggests that, due to ceiling effects, SMAC
may no longer reward further algorithmic improvements and that new benchmarks are needed.
In this paper, we present new, more fundamental problems with SMAC and hence reason for new benchmarks. In particular, we show that an open-loop policy, which conditions only on the timestep and ignores all other observations, performs well on many SMAC scenarios. We also extend this analysis by examining the estimates of the joint Q-function learnt by QMIX. We demonstrate that, even with all features masked, regression to the joint Q-function is possible with less than 10% error in all but two scenarios from just timestep information.
Together these results suggest that SMAC is not stochastic enough to necessitate complex closed-loop (i.e. conditioned on the observation) control policies on many scenarios. Therefore, although SMAC scenarios may require difficult to discover action sequences such as focus fire, agents do not need to adapt to diverse range of situations, but can largely repeat fixed action sequence with little deviation. Additionally, meaningful partial observability (see Section ④ is minimal in SMAC due to large fields-of-view. For partial observability to be meaningful, one agent must observe information that is relevant to the current or future action selection of another agent, unknown to that other agent, and uninferrable from the other agent's observation. Meaningful partial observability is crucial to decentralisation and the NEXP MEXP-completeness of Dec-POMDPs [3].
To address these shortcomings, we propose SMACv2, a new benchmark that uses procedural content generation (PCG) (36) to address SMAC's lack of stochasticity. In SMACv2, for each episode we randomly generate team compositions and agent start positions. Consequently, it is no longer sufficient for agents to repeat a fixed action sequence, but they must learn to coordinate across diverse range of scenarios. To remedy the lack of meaningful partial observability, we introduce the extended partial observability challenge (EPO), where we only allow the first agent to spot an enemy to see for certain when it in range. This requires agents to implicitly communicate enemy information to prioritise targets. Additionally, we update the sight and attack ranges to increase the diversity of the agents and complexity of scenarios.
Our implementation is extensible, allowing researchers to implement new distributions easily and hence further expand on the available SMACv2 scenarios. We also analyse SMACv2 and demonstrate that the added stochasticity prevents an open-loop policy from learning successfully, but instead requires policies to condition on ally and enemy features.
We evaluate several state-of of-the-art algorithms on SMACv2 and show that they struggle with many scenarios, confirming that SMACv2 poses substantial new challenges. Finally, we perform an ablation on the new observation features to demonstrate how each contributes to the difficulty, highlighting the specific areas on which future MARL research should focus.
IIn focused fire, allied units jointly attack and kill enemy units one after another.
4 'Kiting draws enemy units to give chase while maintaining distance so that little damage is incurred. 'SMAC has been cited over 600 times on Google Scholar.

---

2
Related Work
The MARL community has made significant use of games for benchmarking cooperative MARL algorithms. The Hanabi Learning Environment [2] tasks agents with learning to cooperate in the card-game Hanabi. Here, each agent observes the cards of its teammates but not its own, which must be implicitly communicated via gameplay. Hanabi is partially observable and stochastic, but only features teams of to 5 players, which is fewer than all but the smallest SMACv2 scenarios. Kurach et al. 20 propose Google Football as complex and stochastic benchmark, and also have multi-agent setting. However, it assumes fully observable states, which simplifies coordination.
Peng et al. (31 propose a multi-agent adaptation of the MuJoCo environment featuring complex continuous action space. Multi-Particle Environments (MPE) [25] feature simple communication- oriented challenges where particle agents can move and interact with each other using continuous actions. In contrast to both multi-agent MuJoCo and MPE, SMACv2 has a discrete action space, but challenges agents handle wide range of scenarios using procedural content generation. A plethora 24 where agents move and perform a small number of discrete actions on 2D grid, but these tasks have much lower dimensional observations than SMACv2. OpenSpiel 22 and PottingZoo [46] provide collections of cooperative, competitive, and mixed sum games, such as grid-world and board games. However, the cooperative testbeds in either of these suites feature only simple environments with deterministic dynamics or a small number of agents. Neural MMO 41] provides massively multi-agent game environment with open-ended tasks. However, it focuses on emergent behaviour within a large population of agents, rather than the fine- greinee coordination of fully cooperative agents. Furthermore, none of these environments combines meaningful partial observability, complex dynamics, and high-dimensional observation spaces, whilst also featuring more than a few agents that need coordinate to solve a common goal.
of work in MARL uses grid-worlds [25 23 54 48
35
StarCraft has been frequently used as a testbed for RL algorithms. Most work focuses on the full game whereby a centralised controller serves as a puppeteer issuing commands for the two elements of the game: macromanagement, i.e., the high-level strategies for resource management and economy, and micromanagementt i.c., the fine-grained control of army units. TorchCraft [45) and TorchCraftAI M provide interfaces for training agents on StarCraft: BroodWar. The StarCraft II Learning Environment (SC2LE) [4 provides Python interface for communicating with the game of StarCraft II and has been used to train AlphaStar (50), a grandmaster-level but fully centralised agent that is able to beat professional human players. SMAC and SMACv2 are built on top of SC2LE and concentrate only on decentralised unit micromanagement for the CTDE setting.
One limitation of SMAC is the constant starting positions and types of units, allowing methods to memorise action sequences for solving individual scenarios (as we show in Section 5.1). while also lacking the ability to generalise new settings test time, which is crucial for real- hoalwword applicatin of MARL [28]. To address these issues, SMACv2 relies on procedural content generation [PCG; whereby infinite game levels are generated algorithmically and differ across episodes. PCG environments have recently gained popularity in single-agent domains 7 [7 21
36 [8
38) for improving generalisation in RL 19] and we believe the next generation of MARL benchmarks should follow
suit. Iqbal et al. [5 and Mahajan et al. (28) consider updated versions of SMAC by randomising the number and types of the units, respectively, to assess the generalisation in MARL. However, these works do not include the random start positions explored in SMACv2, analyse the properties of SMAC to motivate these changes, or address SMAC's lack of meaningful partial observability. They also do not change the agents' held-of-view and attack ranges or provide a convenient interface to generate new distributions over these features.
3 Background
3.1 Dec-POMDPs
partially observable, cooperative multi-agent reinforcement learning task can be described by a decentralised partially observable Markov decision process (Dec-POMDP) 30). This is a tuple (n, S, A, T, 0, O, R, 7) where n is the number of agents, is the set of states, is the set of individual actions, T: S x A + ((S) is the transition probability function, O is the set of joint observations, S x A > A(O) the observation function, R: : S A -> A(R) is the reward function, and Y is the discount factor. We use A(X) to denote the set of probability distributions over set X. At
3

---

each timestep t, each agent E fl..... n chooses an action a E A. The global state then transitions from state s E S s' S and yields reward r R after the joint action a, according to the distribution T(*, a) with probability P(s"|s, ss a) and R(s, a) with probability P(r'|s,, a), respectively. Each agent then receives an observation according to the observation function so that the joint observation E O is sampled according to (s,, a) with probability P(o|s, a). This generates a trajectory T. E (O x A")". The goal is to learn n policies n, (O, x A)" + A(A) that maximise the expected cumulative reward E 7'Te+a].
3.2 StarCraft Multi-Agent Challenge (SMAC)
Rather than tackling the full game, the StarCraft Multi-Agent Challenge 37 SMACJ benchmark focuses on micromanagement challenges where each military unit is controlled by a single learning agent. Units have a limited field-of-view and no explicit communication mechanism when at test time. By featuring a of challenging and diverse scenarios, SMAC has been used extensively by the MARL community for benchmarking algorithms. It consists of 14 micromanagement scenarios, which can be broadly divided into three different categories: symmetric, asymmetric, and micro-trick. The symmetric scenarios feature the same number and type of allied units as enemies. The asymmetric scenarios have one or more extra units for the enemy and are often more difficult than the symmetric scenarios. The micro-trick scenarios feature setups that require specific strategies to counter. For vs example, 3s_vs__ 52 requires the three allied stalkers to kite the five zealots, and corridor requires six zealots to block off narrow corridor to defeat twenty-foor zerglings without being swarmed from all sides. Two state-of-the-an algorithms on SMAC are MAPPO [55] and QMIX [32]. We provide background to these methods in Section 2 of the appendix.
4 Partial Observability and Stochasticity in CTDE
If the initial state and transition dynamics of a Dec-POMDP are deterministic, then open-loop policies suffice, i.e., there exists an optimal joint policy that ignores all information but the timestep and agent ID. Such a policy amounts to playing back fixed sequence of actions. One way of determining whether SMAC has sufficient stochasticity therefore is to measure how strongly the learned policies condition a policy may only be required to condition on a small subset of the observation space for a few timesteps, whereas observation features. We use this concept in section 5.2 to evaluate SMAC's s stochasticity.
their observations. For example, in mostly deterministic environment,
stochastic environment might require conditioning on large number of
Since open-loop policies do not rely on observations, the observation function and any partial observability it induces only become relevant when the environment stochastic enough to require closed-loop policies. However, even in stochastic environments not all partial observability is meaningful. If the hidden information is not relevant to the task, or can be inferred from the observation, it is resolvable by single agent alone, and if it is unknown and undiscoverable by any of the agents, then it cannot contribute to solving the task. Without meaningful partial observability, task cannot be considered truly decentralised because policy can either ignore infer any unknown information without the intervention of other agents. Meaningful partial observability is essential to the difficulty (and NEXP- (EXP-completeness) of Dec-POMDPs B. While SMAC is technically partially observable, there is limited hidden information because large fields-of ofdds--f-view render most relevant observations common among agents.
Limitations of SMAC
In this section, we analyse stochasticity in SMAC by examining the performance of open-loop policies and the predictability of Q-values given minimal observations. We show that SMAC is insufficiently stochastic to require complex closed-loop policies.
5.1 SMAC Stochasticity
We use the observation that if the initial state and transition functions are deterministic, there always exists an optimal deterministie policy that conditions solely on the timestep information to investigate stochasticity in SMAC. In particular, range of SMAC scenarios to compare

---

24.9 540g
2056 xs 2356
bens vs bans
Em va 9m
comsdor
0.0
10%
1014
1200
10R
1301
open-loop
Figure 2: QMIX (left) and MAPPO (right) open-loop and closed-loop results.
policies conditioned only on the timestep (which we refer to as open-loop) to policies conditioned on the full observation history (i.e., all the information typically available to each agent in SMAC), which we refer to as closed-loop. The open-loop policies observe no ally or enemy information, and can only learn a distribution of actions at each timestep. If stochasticity in the underlying state is a serious challenge when training policies on SMAC scenarios, then policies without access to environment observations should fail. However, if the open-loop policies can succeed the tasks, this demonstrates that SMAC is not representing the challenges of Dec-POMDPs well and so would benefit from added stochasticity.
We use MAPPO and QMIX as our base algorithms because of their widespread adoption and very strong performance and train open- and closed-loop versions of each. We train the open-loop policies on SMAC, but only allow the policies to observe the agent ID) and timestep, whereas the closed-loop policies are given the usual SMAC observation as input with the timestep appended. The open- loop policies still have access to the central state as usual during training since this is only used in CTDE to aid in training and not during execution. The QMIX open-loop policy was trained only on the maps which the MAPPO open-loop policy was not able to completely solve. For QMIX, the hyperparameters used are the same in [3] and for MAPPO the hyperparameters are listed in Appendix
Figure 2 shows the mean win rates and standard deviation of both policies on selected scenarios. The full results are available in Figure in the appendix. Overall, the open-loop policies perform significantly worse than their closed-loop counterparts. However, this performance difference varies widely across scenarios and algorithms. For MAPPO, some scenarios, such as bane_vs_ Sane_ve_bane, 3s5z, 1c3s5z and 2s3z achieve open-loop performance indistinguishable from that of closed-loop. In other scenarios, such as 8m_vs tt_vv_9m 9m and 27m_vs 30m. there are large performance differences, but the open-loop policy still achieves high win rates. There are also scenarios, such as corridor and 2c_vs_ 2c_vs_64zg, where the MAPPO open-loop policy does not learn at all, whereas the closed- loop policies can learn successfully. QMIX also achieves good performance on se_vs_642g, MMM2, 27m_vs_ _m__v_300m and 8m_ amm_v_9___ vs_9m, although for the latter maps the closed-loop policy strongly outperforms the open-loop QMIX. Altogether, there are only four SMAC maps where the open-loop approach cannot learn good policy at all: 3s5z_ vs ____v___sss v___62, corridor, 6h_vs fa_vs__8 _z and 5m_ vs 6m. open-loop policies perform well on range of scenarios in SMAC. The easy scenarios show no difference between the closed-loop and the open-loop policies, and some hard and very hard scenarios show either little difference or non-trivial win rates for the open-loop policy. These successes are striking given the restrictive nature of the open-loop policy. This suggests that stochasticity is not significant challenge for a wide range of scenarios in SMAC.
vmmm. Overall,
These results highlight an important deficiency in SMAC. Stochasticity is either not evaluated or not part of the challenge for the vast majority of SMAC maps. Not testing stochasticity a major flaw for a general MARL benchmark because without stochasticity the policy only needs to take optimal actions along a single trajectory. Additionally, without stochasticity, there can be no meaningful partial observability. Given widespread use of SMAC as benchmark to evaluate algorithms for Dec-POMDPs, this suggests that a new benchmark is required to evaluate MARL algorithms.
5.2 SMAC Feature Inferrability Relevance
In this section, we look at stochasticity from different perspective. We mask (i.e. "zero-out") all features of the state and observations on which QMIX's joint Q-function conditions. the Q-values can still be casily inferred via regression, then SMAC does not require learning complex closed-loop
5

---

5m de slly
5m vs enemy
Sm 6t episodes running dine
3650 1ode onl
3632 XX Tef2 enemy
3652 us. 16fi ephodes nurninng
Ealende Duretee
2000
1000
1200
50
25
50
50
100 Step
100
100
Step
Step
Stp
Step
Step
everything distnce
exceptt actions
team
health
nothing
bsttaction only
Figure 3: Comparison of the loss for different feature masks when regressing to a trained QMIX policy. The x-axis plots steps within episode. "Episodes Running' is the number of episodes that have not terminated by the given timestep. everything masks the entire observation and nothing masks no attributes. team_all masks all attributes of the team. fast_action_onl masks the last actions of all the allies in the state. The all _el__eeee_____ _actions masks all the allied information except the last actions the state. The mean plotted for each mask and standard deviation across 3 seeds used as error bars. The low error rates for the everything mask imply that Q-values can be effectively inferred given only the timestep.
policies as Rec-POMDPs in general do, but is solvable by ignoring key aspects of StarCraft II gameplay. If the central Q-value could be inferred only from timestep information, this would also imply the task not decentralised because all agents implicitly know the timestep and so could operate with the same central policy.
We also extend this experiment to masking subsets of the observation and state features. By only masking subset of features, measure the impact of different features the learned policy. The more observation features the Q-function conditions on, the more confident we can be that SMAC requires understanding decentralised agent micromanagement rather than learning action sequences. We measure this conditioning by regressing to Q-values of trained policy when masking different sets of features. The observation features hidden by each mask in Table in the appendix. Since we evaluate the regression performance of the joint Q-function, we always mask features in the state well the observation. More details of the regression procedure are in Appendix C
Some interesting trends can observed in Figure 3 which shows Q-value regression loss for each mask and scenario against steps in the RL episode. First, the root-mean-squared error of masking everything is low, reaching about 15% of the mean Q-value at peak, and 5-10% for most of the episode. Mean root-mean-squaree error as proportion of mean Q-value, as well as other metrics, is given in the appendix in Table For all scenarios except 5m_vs 6m this value is below 0.12. These not much higher than the baseline nothing masks, suggesting the observation features do not really inform the Q-value. These results highlight the lack of stochasticity and meaningful partial observability in SMAC and the necessity of novel benchmarks to address these shortcomings.
6 SMACv2
As shown in the previous section, SMAC has some significant drawbacks. To address these, we propose three major changes: random team compositions, random start positions. and increasing diversity among unit types by using the true unit attack and sight ranges. These changes increase stochasticity and so address the deficiency discovered the previous section.
Teams in SMACv2 are procedurally generated. Since StarCraft does not allow units from different races be on the same team, scenarios are divided depending on whether they feature Protoss, Terran or Zerg units. In scenarios with the same number of enemy and ally units, the unit types on each team are the same. Otherwise the teams are the same except for the extra enemy units, which are drawn identically to the other units. We use three unit types for each race, chosen to be the same unit types that were present in the original SMAC. We generate units teams algorithmically, with each unit type having a fixed
Figure 4: Examples of the two different types of start positions, reflect and sur- round. Allied units are shown in blue and enemy units in dark red.

---

probability. The probabilities of generating each unit type are given in Table 9 in the appendix. These probabilities are the same test time and train time.
Random start positions in SMACv2 come two different flavours. In reflect scenarios, the allied units spawned uniformly at random on side of the scenario. The enemy positions are the allied positions reflected in the vertical midpoint of the scenario. This is similar to how units spawned in the original SMAC benchmark but without clustering them together. In surround scenarios, allied units are spawned at the centre and surrounded by enemies stationed along the four diagonals. Figure 4 illustrates both flavours of start position.
The final change is to the sight range and attack range of the units. In the original SMAC, all unit types had fixed sight and attack range, allowing short- and long-ranged units to choose attack at the same distance. This did not affect the real attack range of units within the game a unit ordered to attack enemy while outside its striking range would approach the enemy before attacking. We change this to use the values from SC2, rather than the fixed values. However, we impose a minimum attack range of 2 because using the true attack ranges for melee units makes attacking too difficult.
Taken together, these changes mean that before an episode starts, agents no longer know what their unit type or initial position will be. Consequently, by diversifying the range of different scenarios the agents might encounter, they are required to understand the observation and state spaces more clearly, which should render learning successful open-loop policy not possible. We also define convenient interface for defining distributions over team unit types and start positions. By only implementing a small Distribution class one can easily change how these are generated.
Akin to the original version of SMAC 37 we split the scenarios into symmetric and asymmetric groups. In the symmetric scenarios, allies and enemies have the same number and type of units. In the asymmetric scenarios, the enemies have some extra units chosen from the same distribution as the allies. There are 5 unit, 10 unit and 20 unit symmetric scenarios, and 10 vs 11 and 20 vs 23 unit aymmetric scenarios for each the three races.
6.1 Extended Partial Observability Challenge
We now introduce the Extended Partial Observability (EPO) challenge, where make additional modifications to SMACv2 to ensure an extension SMAC with meaningful partial observability. While SMAC is clearly partially observable in the sense that agents do not observe the global state, this partial observability is not particularly meaningful. SMAC certainly occludes information that would be relevant to each agent through the of fixed sight ranges. However, the ranges of attack actions are always less than sight ranges and so ally teams can do without communica- tion by moving within firing range and attacking any observed enemies. Finding enemies is itself a significant challenge, as ally and enemy units spawn closely together. Therefore, much of the information that is relevant to the action selection of a given agent is within its observation.
To create benchmark setting with meaningful partial ob- servability, we introduce two further changes to SMACv2. First, enemy observations each agent. Once an enemy has been observed for the first time within episode, the agent that was first guaranteed to observe it as normal. If two or more agents observe an enemy that had not yet been observed the same timestep, there is a random tie break. As soon as this initial sighting is made, random binary draw occurs for all other agents, which determines whether an agent will be able to observe that enemy for the remainder of the episode if it enters the agent's 's sight range. Agents for which the draw is unsuccessful have their observations masked such that they cannot observe any information about that enemy for the rest of the episode, even if en- ters their sight range. The draw has a tunable probability p of success. We suggest p = the challenge. Figure illustrates this setup.
stochastically masked for
it is
Figure 5: Example EPO enemy sight- ing. Allied units that do not observe the enemy are shown in blue, those that do shown in green and the enemy unit in dark red. Initially, an ally spots an enemy. Later (right), when the enemy is within all allied sight ranges, only the first ally observe the enemy and the ally for which the draw was successful can see
7

---

55
2erg 10 20
zerg zerg,20 20 20
20 vs
zern vs
10M
10M
16M
10N
100
Step
Stap
Siep
Step
Step
10M
1ON
terran 3.
teman 15
termn 20 vs
terrnn, 10 v5
terran vs
Stop
Stop
Step
Stop
Stop
QMIX
NAPPO
IPPO
OPLEX
Figure 6: Comparison of the mean test win rate of QMIX, MAPPO, IPPO, QPLEX and an open-loop policy on SMACv2. Plots show the mean and standard deviation across 3 seeds.
In SMAC, the environment provides an available action mask that guarantees agents can only choose attack actions if the corresponding enemy is within shooting range. In this setting only, we remove this available actions mask. This makes deciding on a target more difficult because agents cannot rely on the available actions to infer which enemies they can attack. Hence they must either infer or communicate this information. If an invalid action is chosen, it becomes a no-op. We also found that without removing the available actions, there was little separation between the p 1 (i.e. all allies can target all enemies) and p = 0 (i.e. only the first ally to see an enemy can target it) cases, shown in Figure [3 in the appendix. We believe that this is because the available actions mask encourages heavy weight to be placed on attack actions, knowing that they will not chosen unless available. In EPÓ, because of the increased difficulty, allied agents battle against 5 enemy units. We found that this increased the gap between the = and p = 0 settings. We believe this is because in preoccupied with other
the more difficult 5_ v_v setting, agents target prioritisation to be a large issue. Results for the 5_vs_5 _ setting are shown in Figure [4]in the appendix and for the 6_vs_5 setting in Figure 7 In EPO, there 3 maps, one for each race, with allies and 5 enemies per team. All other settings such as unit types and start positions are identical to SMACv2. We recommend setting of
basic challenges for
7 SMACv2 Experiments
In this section, we present results from a number of experiments on S.ACv22. We first train state-of- the-art SMAC baselines SMACv2 assess its difficulty. Currently. both value-based and policy optimisation approaches achieve strong performance on all SMAC scenarios [55] 13]. As basclines
QMIX [32], strong value-base method and MAPPO (55), strong policy optimisation method. We also investigate the effect of the new SMACv2 environment features with ablations. These results are available in Figure 12 in the appendix.
7.1 SMACv2 Baseline Comparisons
First we compare the performance of MAPPO, QMIX, [PLEX and IPPO 8 and an open-loop policy, based on MAPPO, on SMACv2. We run each algorithm for 10M training steps and environment seeds. For MAPPO, use the implementation from Sun et al. 43 and for QMIX the one from Hu et al. [3] The results of the runs of the best hyperparameters are shown in Figure
These results shown in Figure 6 reveal few trends. First, QMIX generally performs better than MAPPO across most scenarios. QMIX strongly outperforms MAPPO in two Protoss scenarios (protoss_ aarotoss_0__vt vs_20 20 and protoss 10_vs_ vs 11), but otherwise final performance is similar, although QMIX is more sample efficient. However, QMIX memory-intensive because of large replay buffer and so requires
compute to run than MAPPO. Additionally, MAPPO appears to still be

---

Mean Test Win Rate
zerg vs_5
protoss vs
terran 6 5
zerg vs
protoss 6 vs_5
terran
1.00
0.75
0.50
0.25
0.00
10M
10M
10M
10M 0
10M 0
10M
Step
Step
Step
Step
Step
Step
0.5
Figure 7: Comparison of mean test win rate when p 0.5, with 6 agents against 5 enemy units, for QMIX (left) and MAPPO (right). Here, the available actions mask has been removed.
increasing its performance towards the end of training in several scenarios, such as the 10 vs. and 20_v8_20 20 maps. possible that MAPPO could attain better performance than QMIX if larger compute budget were used. MAPPO and IPPO perform nearly identically across the maps, suggesting MAPPO gains little benefit from the centralised state. QPLEX performs worse than QMIX across a number of maps, particularly the 20_vs_20 20 maps.
Roughly even win rates attained the symmetric maps. By contrast, the asymmetric maps, win rates are very low, particularly on the 20_vs__2__ _r__3 scenarios. Additionally, there is no real change in difficulty the number of agents scales. However, the maps do not uniformly difficult, with all algorithms struggling with the Zerg scenarios more than other maps. Reassuringly the open-loop method cannot learn good policy on any maps even those where QMIX and MAPPO both achieve high win rates. This is strong evidence that SMAC's lack of stochasticity has been addressed.
7.2 EPO Baselines
We now compare QMIX and MAPPO on EPO. A comparison between the p 1, p and p 0.5 O struggles to attain even modest performance, of room for
settings is shown in Figure 7 In this setting, whereas p = comes close to solving the task in a number of instances. This leaves the development of algorithms that learn to use implicit communication as means of overcoming meaningfully restrictive partial observability and reduce the performance gap between the p = 1 and p 0 settings. The performance in the 0.5 case is similar to the 0 case. This suggests that any communication protocol would have to be reasonably robust to have an impact on performance. This makes sense additional 50% chance to observe enemy would still make it difficult to perform group target prioritisation. In general QMIX has a smaller gap between the p = and p 1 cases, suggesting it is perhaps better able to use the global state to resolve partial observability differences than MAPPO.
7.3 SMACv2 Feature Inferrability & Relevance
In this section we repeat the experiments from Section 5.2 on the 5 unit SMACv2 scenarios. We use three previously trained QMIX policies to generate 3 different datasets for training, and compute error bars in the regression across these three policies. The results are given in Table and Figure 11 in the appendix.
When comparing results between SMAC and SMACv2, we focus the difference between the everything (all features not visible) and nothing (no features hidden) masks. This is to account for some scenarios having much higher errors in the nothing mask than others. For example, 5m_vs_ 6m has an error as a proportion of the mean Q-value in the nothing mask of 0. 14, compared to 0.018 for 2c_vs_64zg The SMACv2 scenarios have higher errors than any of the SMAC scenarios when subtracting Sugs for the everything and nothing masks. For example, this value is 0.12 for 5 5_gen, __gen__roooss. which has the lowest value for any tested SMACv2 map. This is significantly higher than the 0.07 for 5m_vs_6m, which has the highest value for any SMAC map. Tables 2 and 3 in the appendix list the full values for SMAC and SMACv2. These results suggest that SMACv2 is significantly more stochastic than SMAC. Second, examining stepwise results in Figure [ in the appendix shows that SMACv2 maps have similar patterns of feature importance to SMAC maps. Enemy health is the most important individual feature and important throughout an episode for all scenarios tested. Importantly, both the ally and enemy all masks reach significant thresholds of the

---

average Q-value, suggesting that it is important in SMACv2 for methods to attend to environment
features.
8 Conclusion
Using a set of novel experimental evaluations, we showed that the popular SMAC benchmark for cooperative MARL suffers from a lack stochasticity and meaningful partial observability. We then proposed SMACv2 and repeated these evaluations, demonstrating significant alleviation of this problem. We also demonstrated that baselines that achieve strong performance on SMAC struggle to achieve high win rates on the new scenarios in SMACv2. We performed ablations and discovered that the stochasticity of the unit types and random start positions combine to explain the tasks' difficulty. We also introduced and evaluated current methods the extended partial observability (EPO) challenge and demonstrated that meaningful partial observability is a significant contributor to the challenge this environment.
Gorsane al. 1T found that it was most common for MARL papers accepted at top conferences to only evaluate on one benchmark, relying on the multiple scenarios within that benchmark to provide sufficient variety. However, in addition to issues shown that paper with cherry-picking of scenarios within benchmark, this evaluation protocol is vulnerable to issues in environment dynamics, such as SMAC's lack of stochasticity, and community-wide overfitting. Given the difficulty in designing both benchmarks and sufficiently diverse scenarios within them, recommend that evaluating on multiple benchmarks should become the norm where possible to help avoid issues.
We hope that SMACv2 can contribute to MARL research as significantly challenging domain capturing practical challenges.
Acknowledgements
Benjamin Ellis and Jonathan Cook are supported by the EPSRC centre for Doctoral Training in Autonomous and Intelligent Machines and Systems EP/S024050/1. The experiments were made possible by generous equipment grant from NVIDIA.
References
[1] Torchcraftai: A bot platform for machine learning research on starcraft: Brood war. https http:///......ocoaaaaee/rrraaaaa 2018. GitHub repository.
[2] Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: new frontier for ai research. Artificial Intelligence, 280: 103216, 2020.
[3] Daniel Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4): 819-840, 2002.
[4] Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. dAlché-Bue, E. Fox, and R. Gar-
In H. Wallach, H. Larochelle, A. Beygelzimer, netr, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL Papere pdf
[5] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas. and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 2021.
[6] Maxime Chevalier-Boisse Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environ- ment for OpenAl Gym. https: http:///gggttaaoooooa/aaaeee///a
2018.
10

---

[7] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation benchmark reinforcement learning. Proceedings of the 37th International Conference on Machine Learning, pages 2048-2056, 2020.
[8] Christian Schroeder de Witt. Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.00 0.0533,,....,.......11 2020.
[9] Yali Du, Lei Han, Meng Fang. Ji Liu, Tianhong Dai, and Dacheng Tao. Liir: Learning individual intrinsic reward in multi-agent reinforcement learning. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d' Alch's e Buc, E. Fox, and R. Garnett, editors, Advances Neural Information Pro- cessing Systems, volume 32. Curran Associates, Inc., 2019. URL
Papera pdf
[10] Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett, editors, Advunces in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
Paper pdf
[11] Rihab Gorsane, Omayma Mahjoub, Ruan de Kock, Roland Dubb, Siddarth Singh, and Arnu Pretorius. Towards standardised performance evaluation protocol for cooperative marl. arXiv preprint 2rXiv:2209.10488 2022.
[12] Tarun Gupta, Anuj Mahajan, Bei Peng, Wendelin Böhmer, and Shimon Whiteson. Uneven: Universal value exploration for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 3930-3944 PMLR, 2021.
[13] Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and SW Liao. Rethinking the implementation tricks and monotonicity constraint in cooperative multi- reinforcement learning. arXiv preprint arXiv:2102.034770 2021.
[14] Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: Universal multi-agent rl via policy decoupling with transformers. In International Conference on Learning Representations, 2020.
[15] Shariq lqbal, Christian Schroeder De Witt, Bei Peng. Wendelin Boehmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 4596- 4606. PMLR, 18-24 Jul 2021. URL htt.tttt:://ooceeaiiiiiii html
mlr www.spresssveee (ttess/v13//ibbal1e1
[16] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34, 2021.
[17) Arthur Juliani, Ahmed Khalifa, Vincent-Pierre Berges, Jonathan Harper, Ervin Teng, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning. In IJCAI, 2019.
[18] Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Procedural level generation improves generality of deep reinforcement learning. CoRR, abs/1806.10729 2018.
[19] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of generalisation in deep reinforcement learning. arXiv e-prints, pages arXiv-2111, 2021.
[20] Karol Kurach, Anton Raichuk, Piotr Stańczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: novel reinforcement learning environment. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4501-4510, 2020.

---

[21] Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Ed- ward Grefenstette, and Tim Rocktäschel. The NetHack Learning Environment, In Proceedings of the Conference Neural Information Processing Systems (NeurIPS), 2020.
[22] Mare Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds. Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Sacta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schrittwisser, Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan-Davis. OpenSpiel: A framework for reinforcement learning in games. CoRR, URL htttttttp//aii....o www./orrggga/aaoooooooooooe abs/ 1908. 09453
2019.
[23] Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi- agent reinforcement learning in sequential social dilemmas, 2017. https:ts:://axt abs/1702.03037
[24] Joel Z. Leibo, Edgar Dué nez Guzmán, Alexander Sasha Vezhnevets, John P. Agapiou, Peter Sunehag, Raphael Koster, Jayd Matyas, Charles Beattic, Igor Mordatch, and Thore Graepel. Scalable evaluation of multi-agent reinforcement learning with melting pot. PMLR, 2021.
[25] Ryan Lowe, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed processing systems, 30, 2017.
environments. Advances in neural information
[26] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems, 32, 2019.
[27] Anuj Mabajan, Mikayel Samvelyan, Lei Mao, Viktor Makoviychuk, Animesh Garg, Jean Kossaifi, Shimon Whiteson, Yuke Zhu, and Animashree Anandkumar. Tesseract: Tensorised actors for multi-agent reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Pro- ceedings of Machine Learning Research, pages 7301-7312. PMLR, 18-24 Jul 2021. URL mlr press/v133/mahajaanra
html
[28] Anuj Mahajan, Mikayel Samvelyan, Tarun Gupta, Benjamin Ellis, Mingfei Sun, Tim Rock- täschel, and Shimon Whiteson. Generalization in cooperative multi-agent systems. arXiv preprint arXiv:2202.00104 2022.
[29] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5600 2013.
[30] Frans A. Olichoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs. Springer Publishing Company, Incorporated, 1st edition, 2016. ISBN 3319289276.
[31] Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Plerre-Alexandre Kamienny, Philip Tomr, Wendelin Boehmer, and Shimon Whiteson. FACMAC: Factored multi-agent centralised policy gradients. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL httphttts//pperrriiiie wttpesnrrriee..nefforummm net/forum? #d==wH777400EEu
[32] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Muchine Learning, pages 4295-4304. PMLR, 2018.
[33] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Ex- panding monotonic value function factorisation for deep multi-agent reinforcement learn- ing- In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Ad- vances in Neurul Information Processing Systems, volume 33, pages 10199-10210. Cur- ran Associates, Inc., 2020. /ttp:////////rroceeeeeiii wwwpediiinnnnneuiiipee nwwrips..... ptt.rrpps.ccpppppe/0/////ii./ 72447haabboee2ace2211155500 Pap.ssaapppppppp. pdf
12

---

[34] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint 1rXiv:2205.06175 2022.
[35] Cinjon Resnick, Wes Eldridge, David Ha, Denny Britz, Jakob Foerster, Julian Togelius, Kyunghyun Cho, and Joan Bruna. Pommerman: A multi multi-agent playground. arXiv preprint WrXivv1809.07122 2018.
[36] Sebastian Risi and Julian Togelius. Increasing generality in machine learning through procedural content generation. Nature Machine Intelligence, 2, 08 2020. doi: 101100//42256-0200020000
[37] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr. Ja Jakob Foerster, and Shimon White- son. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 2186-2188, 2019.
[38] Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Kuttler, Edward Grefenstette, and Tim Rocktäschel. Minihack the planet: A sandbox for open-ended reinforcement learning research. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
[39] Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards-just map them to actions. arXiv preprint 1912. 02875,, 2019.
[40] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5887-5896. PMLR, 2019.
[41] Joseph Suarez, Yilun Du, Phillip Isola, and Igor Mordatch. Neural mmo: A massively multiagent game environment for training and evaluating intelligent agents. arXiv preprint arXiv: 2019.
[42] Mingfei Sun, Vitaly Kurin, Guoqing Liu, Sam Devlin, Tao Qin, Katja Hofmann, and Shimon Whiteson. You may not need ratio clipping in ppo. arXiv preprint arXiv:2202.00079, 2022.
[43] Mingfei Sun, Sam Devlin, Jacob Beck, Katja Hofmann, and Shimon Whiteson. Trust region bounds for decentralized ppo under non-stationarity. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, AAMAS '23, page 5-13, Richland, SC, 2023. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450394321.
[44] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning based team reward. In Pro- ceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 2085-2087 2018.
[45] Gabriel Synnaeve, Nantas Nardelli, Alex Auvolat, Soumith Chintala, Timothée Lacroix, Zeming Lin, Florian Richoux, and Nicolas Usunier. Torchcraft: a library for machine learning research on real-time strategy games, 2016. Utt/htss::/aarxi
httg/abs/161111
[46] Justin K Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez- Vicente et al. Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[48] Eugene Vinitsky, Natasha Jaques, Joel Leibo, Antonio Castenada, and Edward Hughes. An open source implementation of sequential social dilemma games. httpps://iithhh com/ euggeneriintttky//eqeeettt.... sociallls__l_____ooooaaa www.ddlllmmmmmmmmmmmmmmmm weemaaaamesiissseeeeeeeeeeeeeese 182 2019. GitHub repository.
13

---

[49] Oriol Vinyals, Timo Ewalds. Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittweeser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, Hado van Hasselt, David Silver, Timothy Lillicrap, Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp, and Rodney Tsing. Starcraft ii: A new challenge for reinforcement learning, 2017. URL http:///aaaaaiv..gaaas//a/a/a///
04782
[50]
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang. Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg. Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Çaglar Gülçehre, Ziyu Wang, Tobias PfafT, Yuhuai Wu, Roman Ring. Dani Yogatama, Dario Wünsch, Katrina McK- inney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap. Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nat., 575(7782)3350-354 2019. doi: (038//41586-0111
[51] Jianhao Wang. Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. International Conference on Learning Representations, 2020.
[52] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforce- ment learming with emergent roles. arXiv preprint arXiv:2003.08039, 2020.
[53] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng. Shimon Whiteson, and Chongjic Zhang. Rode: Learning roles to decompose multi-agent tasks. In International Conference on Learning Representations, 2020.
[54] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 5571-5580, PMLR, 10-15 Jul 2018. URL htttttt//procceddiiii
mlr. pptps///0//ese/p/pppe
v80/ yrress//v00/ang1Bee waaann1d.... html
[55] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2 2021.
14

---

A Links
SMACv2: http://ggiubb.coommowwwwi//eee/ MIT License.
MAPPO implementation: https//gghhub.commmmmoollll//ppp
QMIX implementation: https://gthhubccommm//mlllll//// Apache-2 License.
B
Further Background
B.1
StarCraft
StarCraft Il is real-time strategy game featuring 3 different races, Protoss, Terran and Zerg, with different properties and associated strategies. The objective is build an army powerful enough to destroy the enemy's base. When battling armies, players must ensure army units acting optimally. This is called micromanagement. An important micromanagementt strategy is focus firing, which is ordering all allied units jointly target the enemies one by one to ensure that damage taken is minimised.
Another important strategy is kiting, where units flee from the enemy and then pick them off one by one as they chase.
B.2 QMIX
QMIX can thought of as an extension DQN (29] to the Dec-POMDP setting. The joint optimal action is found by forcing the joint Q to adhere to the individual global max (IGM) principle[40) which states that the joint action can found by maximising individual agents' Q, functions:
arg max, Q1(11, a1) arg max, Q2(T2, a2)
arg max Q( a))
arg max, Qu an)
This central Q is trained to regress to target 7Q a) where Q is target network that updated slowly. The central Q estimate is computed by mixing network, whose weights are conditioned on the state, which takes as input the utility function Q, of the agents. The weights of the mixing network are restricted to be positive, which enforces the IGM principle(40) by ensuring the central Q is monotonic in each Qi.
B.3 Independent and Multi-agent PPO
Proximal Policy Optimisation (PPO) is method initially developed for single-agent reinforcement learning which aims address performance collapse in policy gradient methods. does this by heuristically bounding the ratio of action probabilities between the old and new policies. To this end, optimises the below objective function.
E Egndt a
min
(#(aas)
A= a). clip (T(ass)
i(aas)) T[aas)"
- 1, As(saa)
where clip(t, a, b) is function that outputs if a, bit b and otherwise.
Extending this to the multi-agent setting can easily done in two ways. The first use independent learning, where each agent treats the others part of the environment and learns critic using its observation and action history. The second is to use a centralised critic conditioned on the central state. This is called multi-agent PPO. Both independent PPO (IPPO) and multi-agent PPO (MAPPO) have demonstrated strong performance on SMAC[55]
g Note that we not apply the observation [55 anywhere in this paper. This is because these changes
and state changes suggested by Yu et al. were not implemented wrapper on top of SMAC, but instead by modifying SMAC directly, leading to the environment code becoming unmanageably complicated.
15

---

27m 30m
5m 6m
8m
bane bane
corridor
Mean Train win Rate
0.3
0.8
0.6
0.4
0.1
0.2
0.-
0.00
0.0
10M
0
10M
10M
10M
10M
Step
Step
Step
Step
Step
2c, vs. 64zg
253z
3s52
10m vs. 11m
1c3522
Mean Train Win Rate
1.0
1.0
9.
0.8
0.8
0.6
0.6
94-
0.4
0.4
0.2
0.2
02
0.0
0.0
10M
100M
10M
Stop
Step
Stop
Step
Stop
closed-lopp MAPPO
open-loop MAPOO
(a) MAPPO open-loop and closed-loop results
2c_vS_642g
MMM2
27m_ v5_30m
8m, vs V5_
3s52_ $552_vs_33563 3962
Mean Test Win Rate
1.00
0.75
0.50
0.25
0.00
10M
10M
0
10M
10M
10M
Step
Step
Step
Step
Step
closed-loop QMIX
open-loop QMIX
(b) QMIX open-loop and closed-loop results
Figure 8: Plot of selected SMAC scenarios treated as an open-loop planning problem by limiting the observation to the current timestep and agent ID. Plots show the mean win rate and standard deviation across 3 training seeds for MAPPO and QMIX.
C Experimental Details
In this section we describe extra details of the experiments run as part of the paper.
C.1 Stochasticity
This section describes details of the experiments run in Section 5.1 the main paper. Both the closed-loop and open-loop algorithms were based the MAPPO implementation from Sun et al. [43. The code used for these experiments can be found here Both the open-loop and closed-loop algorithms use the same neural network architecture as in 43 Both were also provided with access to the available actions mask the environment and conditioned the critic on the state. We used the hyperparameters from Sun al. 43]. with the exception of the actor's learning rate, which we set to 0.0005 and decayed linearly throughout training. This is because learning rate decay has been found to be important for bounding PPO's policy updates 42 Full hyperparameters can be found in
Table 5 Full results are shown in Figure g
C.2 SMAC Feature Inferrability & Relevance
This section describes details of the experiments in Section 5.2 and Section 7.3 in the main paper. The code used for these experiments can be found here
For each scenario, we had 3 QMIX policies trained using the implementation and hyperparameters from Hu al. [13, each with different network and SMAC initialization. Training results for the policies for SMAC are shown in Figure 9 Each policy constitutes seed and is used to collect
16

---

Mean Test Win Rate
27m_v5_350 v5m30m
6n_v5_8c
VS /s_3662
1.00
1.00
1.00
1.00
0.75
0.75
0.30
0.50
0.50
0.50
0.50
0.25
0.00
0.00
0.00
0.00
0.00
10M
10M
0
10M
10M
10M
Step
Step
Step
Mean Test Win Rate
5m, VS. 6m
3s Vs,52
20 V5. 642g
8m, VS 9m
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
10M
10M
10M
10M
Step
Step
Step
Step
QMIX
Figure 9: QMIX training results on SMAC
dataset of episodes for the reature-relevance experiment on the scenario. We call these policies expert policies. QMIX hyperparameters used are given in Table A dataset consists of two folds: 8192 episodes used for training and 4096 episodes used for evaluation.
For given mask, the experiment then consists of training a new QMIX network, termed regression network, whose input is trajectories with observations and states masked according the mask, and whose task is to predict the Q-values output by the expert policy on the unmasked trajectories.
The nothing mask does not apply any effect the observations and states in trajectory. Otherwise, a mask, say 'health (ally)', masks the health feature every ally in the observation of each agent (except its own health) and masks the health feature of all the units controlled by QMIX in the state. Table Jshows feature sets zeroed out for different masks.
The regression network has the same architecture as the expert network. We use the mean squared error (MSE) as a loss function and optimise the network via Adam with batch size of 512 episodes and learning rate equal to 0.005. (The other Adam hyper-parameter, are the default PyTorch values.) Training is performed with early stopping according to the validation fold with an evaluation every epochs and patience of 10 tries (i.e. training is stopped when the MSE on the validation fold does decrease 50 consecutive epochs and performance at the best epoch is retained). Models for all masks and scenarios hit early stopping and trained for about 200 epochs average, except a few ones which trained to limit 500 epochs.
All hyper-parn parameters were tuned on few different scenarios from both SMAC and SMACv2 to minimise validation MSE using datasets and expert policies not used for the reported results. The sizes of the training and validation folds are respectively 1.6 times and 0.8 times the size of the QMIX replay buffer (5000 episodes) and have been chosen to be large enough minimize validation MSE while keeping experiments practical.
Figure 10 shows the full results of the feature quality experiments for SMAC and Figure 11 shows the full results for SMACv2. Table shows summary results for the everything and nothing masks for SMACv2 and Table 2 shows the data for SMAC.
Experiments for this section were conducted on 80-core CPU machines with NVIDIA GeForce RTX 2080 Tî Tesla 1100-XXM2-16GB GPUs. A single (scenario, seed, mask) combination took around I hour to train. In total, the experiments this section took about 1500 GPU hours.
C.3 SMACv2 Runs
Here we describe the hyperparameter and training procedure used in section 7.1. As in previous experiments, we used the implementation by Hu et al. [3] for QMIX and by Sun et [4] for MAPPO. These implementations have both achieved very strong results on SMAC. We therefore tuned the hyperparameters by taking deviations from these for a few key parameters. For MAPPO, tuned the actor's 's learning rate and the clipping range. Additionally, we added linear learning rate
17

---

Table I: Masked features for each setting of the feature quality experiment. V means feature is
masked and Xmeans a feature not masked.
Mask
Ally
Enemy
Health
Shield
Distance
Actions
Health
Shield
Distance
everything nothing health (ally)
V
V
X
X
X
X
/
X
X
X
X
X
X
shield (ally)
X
X
X
X
X
distance (ally)
X
V
X
X
X
X
health and shield (ally)
V
X
X
X
X
X
actions only
X
X
X
V
X
X
X
X
X
all except actions
ally all health (enemy)
V
V
V
V
V
V
X
XXXX
X
X
XXXX
X
X
X
J
X
X
shield (enemy)
X
distance (enemy) enemy all
X
X
6n any
5m nurrig
Jede
3dts 3ets
Jude
Dmg RAKE
Damp RICE
50
50
100
300
100
Sm tin
Sm as Dmmenny
gBnuas geotess meriing 100
th
thus Sconetre
toeesesss reteing
Ska RMC
L00
100
110
100
20 ve Gring sky
Ging many
2c Glop wploocep. nenrigg
contier sky
conteer enemy
centstr-plidic (umddeemmmmn
Stap VAE
SA POSE
6:9
250
250
253
Swe
22m w/y
70m 3im eremy
37m
30m
zurnirg
sky
MNNO enemy
V9610 rposdes raming
S001 MMSC
Stoce RMSt
200
shy
smemy
sptodda naming
100 K20
Eppcate
Oyp
CAn
d shici
exceept actinn
Samm: ssi_edoor_nng only
excratling
nothing
Figure 10: Results of the feature quality experiments for all tested SMAC scenarios not in the main paper.
18

---

Table 2: Mean Q values in the different feature quality experiments on SMAC scenarios
Medk
everyling
7.05000
0.65 0.35
0 = 0.02
0.00 1-335
0 00300
ally propppacaiin actines
0.31 1.26
G1M7 B.005
01.11 -304
1.:11 1-118
aly Jeaki
0.2 0.35
0 = 00003
0.10 2/12
0,005 0.254
0.27 0.20
0 2 8.002
0.376-2 3/003
0.001 =0,00
04:002 0.2
0.09 = 0:21
0.25 3-03
0,02 0.25
STATES Jeal and (lackl
0.20= 0.35
0 = 00000
62200.06
0,035 0.204
0.221 0151
unng 1 DU092
0.26 91335
(U363
CtATEy Jlach
0:2 0.38
0J06 2: 0003
0.101
2-035
BCX0 + 1O08
0.206
0.85
54330 0.363
Jilt, 2i, proppt actoee acoces
0281 0.55
UIAM 2 0.003
DL:03 0.00
GJAKST (LOCO
aky All_Jhaah_nne soads
6.328 1-01
UINT 2
203
Tealh
0.301
ally dosonee
0.29 0.25
0001 = +.000
0.161 501
-CC 0.00
aby sTNIE
0.201 1L3CT
GING 2 BV00
1.30 2 2014
ILIIE 1L3004
eneryy beal and Jhcl
0.20 0.35
0 = 0.003
0:55- 503
0.015 0,300
cxemyy 20at
0.0 1120
dIR1 DV003
0122 2/11
ILES S-UIN
ererry sheed
0.202 0.304
0 + 0.000
0.188 5 2035
0023 0,3006
cones all
0. 0:0
0.61 = 80007
03 3/11
0.00 + 000
salt. cooapt acticss
1.17 0.54
2
0.392 3/55
GJNI 01204
aky AlyLhwshhjae atd 2hol
1 0.26
0.12 2: 0012
0.71 2/15
0 (L,02
vity
0.35
0.15- 0:13
3.05
0.031 0.364
1.12 6.50
0.19 2: 8.22
0.71 2/13
UIIIL 0L002
=
2003
divi
0.50
0.16 6.07
0.51
0.361
enenty bolfh
17 0.35
0.16 0/13
0282- 3.05
0 0.204
CISTE ditteass
0.22
0.15 0002
0.92 --22
0.307
-DC
coneryy all metling
1:45- 0.36
1818 8013
0 5-07
0.0E 6 0.00
0/02
0.71
123 000
0.25 1027
01K1 67305
0-0 303
OTES 1X0T
0.05 0.55
0 0.006
0.277
3/94
0 0,302
wby Teath
0.59 11.0
0855 0.103
01:00 3.03
0.47 0.35
0 = 0.02
0.277 2 5.12
0 0.365
0.78 0.37
0.26 0.107
0.65-200 2015
0 0.00%
0.55 0.55
01302 5113
01991 (1361
0.90 0.37
01654 = 0/005
027 2014
0 0,005
CURSET Aichl
0.61 0.56
0162 0/155
0136 5/5
0101G 0.355
eetiing
0.44 0.37
0/00 0.005
0.4 2/14
exerytiing
UD 301
0/21 1031
UIMI 10305
110 3711
olyy seoupt actces
0.47 11.55
01N : DU30"
0128 2014
UII4 (L263
aly Tealh
0.45 0.36
0 = 0.007
0.26 3-04
0.01 0.300
all dareee
0.0 1126
0107 5 Dui0e
11:21 2004
1II1 11303
olly
0.000.00
0.00 = 0.005
0:00= 500
0.015 0.305
CRERy bostt deea
0.05 0.34
dIN 2 D0005
0130 2011
4IR2 1L304
0.41 0.30
0002 = 0.055
0.55 2- 901
0,007 0.305
overty Jichl
0.35= 0.36
0.00 = 0/007
0.1 + 201
0,009 0, 0,0066
ceeling
0.34
0109 5 00000
022
1012
0201
231L
CJNN TUXUS
allt peoptt acices
0.15 0.24
0116
018.2 2/12
IUCOOS 2 J-JNLL
ay beath
Teath
0:2-4
5.0
0.11 0.25
UNIT 00093
MAT 2011
-DOOCT 2 MA
0.14
0108 2:
121 14520
STERRY hwwit anc_
0L45 2 210
01002 BU022
HSIT2 2014
LIIS ELOCS
encmyy_bcitt beidts
0-22 = 03355
00355-200000
0S14 2002
0.301
everry dieit
0110 0.33
0m1a -000
0.7 501
6.-4 1000
aal all
0 0.22
0.27 11102
0.11 2/11
GJNIZ 1L365
alle coappt acices
0.42 12 0.32
0.60 = 0005
0.51 3/15
O 0.302
aby Teathh shedd
0.11 -1.30
0106 B/15S
0 2:11
ally Johnes
0.4 (.35
0160 = 0005
0.29 2 5015
0 (L303
aby Sandd
0 0.30
GIGI = 50005
026-220 201
(L001 2.1000
Jfly
0.45 0.32
0.38 0.005
0.00 = 01301
CXERYY Scala
0,00 0.30
0235220011
035:220 201
GNLL 0.301
deelt dhed CHNTE, ail melling
0.21 (150
9011
G101 0.30R
0 0.32
QURS 2 B0005
0100
INC254-2 2 0.00L
0 0.30
0 203
dlle Cveplling
XIE
03200
O'IZE CLOOS
bly.hwahh and sould
017 0.30
BOK: 600
-LUCE 1 111
aby beath
0.17 6130
M 201
-TUC 1 1ING
ally Sistd
0.88 0.35
0 = D002
0 1088-99 4037
0.005-000 0.00
113 111
ITT 201
GHIE (LUCT
CHeRty boals
0.25 0.35
0 0000
0 905
0 0.300
10 1.20
HIM 2012
-10005 1 MAINNN
0.30 0.30
0 S 0004
0T 901
O'ASS 0.301
seding
112 0012
0 0.003
0.06 2011
18022
eventtang
54:01
07 0.3
DJR 2012
B 2013
UJERI ELOCS
alle exeett
0.30 EL53
11224 5155
GARES EL302
ally Jeshh
0221 1L.4
UIRZ 2 DUDRS
0121 2/14
anly disuee
-BOKI
sll
0412 0L2
ONT 2 0012
0120223300 3/13
GLIE (LOCS
CTETE beatt aod Jock
0.41 1126
GIN0 2: 0/005
0129 2/14
8.300
everyy Caassoo
0.32 0.35
0 8 0/006
10 3.04
-BCD0S-4 O DIOO
cremi dachd
0:01 1124
0105 2 DUS55
0.18 2/14
-ELOC
weiing
0.35 0.35
0.08 6 6/007
02-400 0.00
19

---

5 gen. terran ally
5 gen terran enemy
5_gen terran episodes running
4000
Episodes Running
Step RMSE
2.0
3000
1.5
1.5
1.0
1000
0.5
100
200
100
200
100
200
Step
Step
Step
ree_proooos ally
5 gen_protoss enemy
5_geenprotoos protoss episodes running
2.0
2.0
4000
Episodes Running
Step RMSE
1.5
1.5
3000
1.0
1.0
1000
0.0
100
100
100
Step
Step
Step
5_ 5_gen_ _eggeeenrgggggg ally
5_gen __ggenzeerg zerg enemy
5_gen __ge_zeerg episodes runnnig
4000
Episodes Running
Step RMSE
3000
2000
50
100
50
100
50
100
Step
Step
Step
own health
distance shield health
all pecept_actions nothing own_pasition
own_for team_al all
everything
health_ ani_ed
last aast_action_oo aattttonnoonlllyoyoooeeeeeeooyye
Figure 11: Feature Quality Experiments for the 5 and 10 unit SMACv2 scenarios.
decay to the MAPPO implementation because this has been shown to be important to bound the policy updatef42) [22. For QMIX, we tuned the c-annealing time and À in the eligibility trace Q(A). All other hyperparameters, including neural network architecture, were unchanged from the implementations mentioned previously.
The MAPPO code for these experiments can be found here, and the QMIX code here The QMIX code distributed under the Apache license, and the MAPPO code under the MIT license. The only differences in these branches and the implementations used for previous experiments on SMAC are important to changing the environment from SMAC SMACv2.
The open-loop policy is identical to MAPPO, except the policy receives as input only the timestamp and agent ID, as in Section 5.1.
We tuned hyperparameters by running each set of hyperparameters on scenarios and then choosing the best results. The grids of hyperparameters for QMIX and MAPPO can be found in Table [ For IPPO, we used mostly the same hyperparameters as MAPPO. For QPLEX we used the hyperparameters from [51] All algorithms were trained for 10M environment steps with evaluations
20

---

Table 3: SMACv2 feature quality experiment results
Map
Mask
Q
Fanse
?
Gass
s-gen_protoom
everything
8.4 0.05
1.5 + 0.05
0.179 0.006
1.18 + 0.04
0.119 0.004
Sal_allll_e__ cxcc_cccp__tac actions
0.09 0.02
0.082 = 0.001
0.52 1.02=0.01 0.01
0.022
0.002
ally health and shield
0.62
0.074 0.004
0.45
ally health
0.51
0.061 = 0.005
0.39 0.02
0,001 0.001
ally distance
0.52
0,0019
ally shield
0.52 1 0.04
0.062 0.006
0.38 1 0.03
0.0018 0.0008
ally_ail all
0.85
0,012
enemy health and shield
0.79 1. 0.03
0.084 0.004
0.6 L 0.03
0.034 0.003
enemy bealth
D,022
enemy distance
0.56 1 0.03
0.067 0.006
0.41 1 0.03
0.0074
0.0002
enemy shield
0.57
0.068 0.005
0.42
0,008 0,002
enemy all
0.96 1 0.02
0.116 1. 0.004
0.74 0.02
0.055
0.002
nothing
0.5 0.03
0.06 0.006
0.37 0.03
5_ 5_gen_ berran
everything
0.1
2. 0.1
0,243 0.000
0.1
0.148 0.0US
_lly_all all excepttactionns
0.05
0.1221 0.001
0.75 170=0.00 0.01
0,028 0,002
ally health and and shield
0.82 0.04
0.1 = 0.001
0.6 0.03
0.0U52 0,0009
ally health
0.83 = 0.04
0.1007 = 0.0003
0.6 0.04
0.0061 0.0007
ally distance
0.78 1: 0.04
0.0945 = De US
0.56 0.03
0.0072 = 0.0007
ally shield
0.75 = 0.02
0.092 = 0.002
0.55 0.02
-0.003 0.001
ally_sll
1.18 1 0.06
0.145 = 0.002
0.92 = 0.06
0.05 1 0.003
enemy health and shield
0.98 1: 0.06
0.12 0.003
0.72 0.05
0,025 0.003
enemy bealth
0.97 1: 0.07
0,118 0.003
0.71 1 0.05
0,023 = 0.003
enemy distance
0.82 1: 0.04
0.1002 0.0002
0.59 0.03
0.0055 0.0006
enemy shield
0.75 1: 0.03
0.091 - 0.005
0.54 0.04
-0.004 0.004
enemy alll
1.2 1.1
0.151 0.007
0.95 0.09
0.057 0.008
nothing
0.78 0.00
0.0945
0.0007
0.56 + 0.03
5_ gege__aegg
everything
0.20
2.11
0.33 = 0.02
1.86 0.01
0.1 = 0.009
ally_all except, actions
1.36
0.188 0.008
0.99
ally_healle health and shield
1.15 0.03
0.16
0.83 = 0.03
0.002 0.001
ally health
1.17
0.005
ally distance
1.18 0.02
0.164 + 0.007
0.85 1 0.02
0,006 0.001
ally_shiio shield
1.13
0.137 0,006
0,0000
ally_ai all
1.46 1: 0.02
0,202 1 0.01
1.09 1 0.02
0.044 0.002
enemy enemy_healee health shield
0.95 0.03
enemy bealth
1.3 0.02
0.179 1 0.007
0.94 1 0.02
0.022 = 0.001
enemy distance
1.24
0.173 0.008
0.88
0.014
enemy enemy_shhelld shield
1.14 0.02
0.159 1 0.006
0.82 15211..00 0.02
0.0005 0.0008
enemy
0.23
1.24
0.008
nothing
1,14 0.02
0.137 + 0.007
0.81 351+00000 0.02
of 20 episodes every 2k steps. The hyperparameters used for QMIX are given in Table 4 for MAPPO and IPPO in Table 3 and for QPLEX in Table
C.4
EPO Runs
This section provides the implementation details for the Extended Partial Observability challenge in SMACv2 and outlines the corresponding training procedure.
For the EPO baselines in Section 7.2, used the same hyperparameters Section 7.1. Imple- mentations of MAPPO and QMIX were also consistent with those in Section 7.1. Each training run and corresponding evaluations were conducted across 3 random seeds. Experiments for this section were conducted on 80-core CPU machines with NVIDIA GeForce RTX690 2080 Ti or Tesla V100-SXM2-16GB GPUs. Both MAPPO and QMIX experiments took between 36-48 hours to complete, each running on 8 GPUs, for 3 p values, across 3 seeds and 3 races.
D Environment Additional Details
D.1 SMACv2 Additional Details
In this section we describe the generation of teams in SMACv2. For each race, different types of units are used. Each race has a special unit that should not generated too often. For Protoss this is the colossus. This is very powerful unit. If team has many colossi, the battle will devolve into a war about who can use their colossi most effectively. Terran has the medivac unit. This is healing unit that cannot attack the enemy and so is only spawned sparingly. Zerg has the baneling unit. This a suicidal unit which deals area-of-effect damage to enemy units by rolling into them and exploding. In scenarios with many banelings, the agents learn to spread out and hide in the corners
21

---

Table 4: QMIX hyperparameters used for experiments. Parameters with (SMAC) or (SMACv2) after them denote that parameter setting was only used for SMAC or SMACv2 experiments respectively. These are the values in the corresponding configuration file in PyMarl 37 13. Mac is the code responsible for marshalling inputs to the neural networks, learner is the code used for learning and runner determines whether experience collected in serial or parallel.
Parameter Name
Value
Action Selector
epsilon greedy
Start
1.0
Finish
0.05
Anneal Time
100000
Runner
parallel
Batch Size Run
4
Buffer Size
5000
Batch Size
128
Optimizer
Adam
/max
10050000
Target Update Interval
200
Mac
mac n rn
Agent
Agent Output Type
q
Learner
nq Jearner qmix
Mixer
Mixing Embed Dimension
32
Hypernet Embed Dimension
64
Learning Rate
0.001
(SMAC)
0.6
A (SMACv2)
0.4
with the hope that the enemy banelings explode and the allies win by default. All of these special units are spawned with probability of 10%. The other units used spawn with probability of 45%. This is summarised in Table
There are two changes to the observation space from SMAC. First, each agent observes their own field-of-view direction. Secondly, each agent observes their own position in the map as x- and y-coordinates. This is normalised by dividing by the map width and height respectively. The only change to the state from SMAC was to add the field-of-view direction of each agent to the state.
Additionally, we made one small change the reward function in SMACv2. This to fix a bug where the enemies healing can give allied units reward. There are more details available about this problem in the associated Github issue SMACv2 also has an identical API to the original SMAC, allowing for very simple transition between the two frameworks.
The code for SMACv2 can be found in the Github repo where there is a README detailing how to run the benchmark with random agents.
D.2 EPO Additional Details
In EPO, the first ally agent to observe an enemy is guaranteed to observe the usual SMAC features associated with that enemy. Upon an enemy being observed for the first time, by any agent on the ally team, random binary draw occurs for all other agents (i.e., those that had not yet observed this particular enemy). Random tie breaking ensures only one agent is guaranteed see the enemy should two or more observe it for the first time on the same timestep. The draw is weighted by a tunable environmnet parameter p corresponding to the probability of success, If the draw successful for particular agent, any future instances of the agent observing the enemy will occur as normal, without masking. If the draw unsuccessful, that agent will not be able to observe the enemy on future timesteps, irrespective of whether or not it is within sight range. If the first agent to have observed the enemy dies, the next time the enemy falls within an ally agent's 's sight range that agent guaranteed to see the enemy and the random draw occurs again for all other agents.
22

---

Table 5: MAPPO and IPPO hyperparameters used for the experiments on SMAC and SMACv2. Parameters with (SMAC) or (SMACv2) after them denote that parameter setting was only used for SMAC or SMACv2 experiments respectively.
Hyperparameter
Value
Action Selector
multinomial
Mask Before Softmax
True
Runner
parallel
Buffer Size
8
Batch Size Run
8
Batch Size
8
Target Update Interval
200
Learning Rate Actor (SMAC)
0.001
Learning Rate Actor (SMACv2)
0.0005
Learning Rate Critic
0.001
0.995
0.99
Agent Output Type
pi pi_logits
Learner
trust wwwgreennlaarreeee
Critic (MAPPO)
centralV_rmi Sarilli____n_riiieee
Critic (IPPO)
decentral_rme lai____n__iiii
Detach Every
20
Replace Every
None
Mini Epochs Actor
10
Mini Epochs Critic
10
Entropy Loss Coeff
0.0
Advantage Cale Method
GAE
Bootstrap Timcouts
False
Surrogate Clipped
True
Clip Range
0.1
Is Advantage Normalized
True
Observation Normalized
True
Use Popart
False
Normalize Value
False
Obs Agent Id
True
Obs Last Action
False
E
Analysis of Changes in SMACv2
To investigate the impact of the changes introduced in SMACv2, we perform three ablation studies. We focus only on the 5 unit scenarios, and only train MAPPO on the ablations because it is significantly faster to train than QMIX. We use the same hyperparameters and setup as in Section 7.1. We ablate different team compositions by using a single unit type for each of the non-special units in each race. We investigate random start positions by only using the surround reflect scenarios and by removing the random start positions entirely. We also ablate the effect of using the true unit ranges. Additionally, we investigate partial observability by comparing feed-forward and recurrent networks.
The results of these experiments are shown in Figure 12 The position ablations show that stochastic starting positions and hence stochasticity itself contributes greatly to the challenge of SMACv2. Without random start positions, MAPPO achieves win rates of close to 100%. Both surround and reflect scenarios have a similar win rate, suggesting similar difficulty. There does not appear to be additional difficulty from combining the two. This is logical as the scenarios can be disambiguated using start positions at the beginning of the episode.
Our experiments of unit type ablations, shown in Figure 12b indicate that unit variety significantly impacts the difficulty of the tasks. All three races show large differences between the easiest unit distribution and the baseline, suggesting that diverse range of units contributes to SMACv2's difficulty. In both the Zerg and Protoss scenarios, the melcc-only scenarios are casier than the ablation
23

---

Table 6: QPLEX hyperparameters used for experiments on SMACv2.
Hyperparameter
Value
Action Selector
posilon_greede
Epsilon Start
1.0
Epsilon Finish
0.05
Epsilon Anneal Time
50000
Runner
parallel
Batch Size Run
4
Buffer Size
5000
Batch Size
128
Optimizer
adam
Target Update Interval
200
Agent Output Type
q
Learner
Double Q
True
Mixer
dmaq qatten
Mixing Embed Dim
32
Hypernet Embed
64
Ady Hypernet Layers
Ady Hypernet Embed
64
Td Lambda
0.8
Num Kemel
4
Is Minus One
True
Is Adv Attention
True
Is Stop Gradient
True
Head
4
Attend Reg Coef
0.001
State Bias
True
Mask Dead
False
Weighted Head
False
Nonlinear
False
Burn In Period
100
Name
bile__qatten_s''
Table 7: QMIX hyperparameter tuning grid
Hyperparameter
Values Tried
annealing time
[100 103, 500 103]
[0.6, 0.8, 0.4]
Table 8: MAPPO hyperparameter tuning grid
Hyperparameter
Value
Actor Learning Rate
[0.0007, 0.0004, 0.0001] [0.15, 0.05, 0.1]
Clip Range
24

---

Table 9: Unit types used per-race in the SMACv2 scenarios.
Race
Unit Types
Probability of Generation
Terran
Marine
0.45
Marauder
0.45
Medivac
0.1
Zerg
Zergling
0.45
Baneling
0.1
Hydralisk
0.45
Protoss
Stalker
0.45
Zealot
0.45
Colossus
0.1
protom
5.x.5
protess 5.v..
terran 5.05.
Hoan Text We hote
33
Stop
Stop
Stop
Step
Surrond
Reflect
Surround Rellect
No Random Start
Welphrsefo 0.45,
(a) Position ablations
(b) Unit type ablations
vS
protoss vs.5
terran 5 vs
zera
Mean Test, Win Rate
Mean Test Win Rate
2.4
22
0.2
10M
10M
10M
30M
Step
Step
Stop
FeedForward
RNN
(c) Field-of-view ablations
(d) Feed-forward vs RNN ablation
Ablation SMACv2 unit type, ffld-of-view and starting position features. Figure 12d shows the performance of fred-forward policy compared to RNN bascline. The unit weights relative to a fixed order of units. For Zerg this is zergling, hydralisk, baneling, for Terran marine, marauder, medivac and for Protoss it is stalkers, zealots and colossi. Plots show mean and standard deviation across three seeds.
Figure 12:
25

---

Mean Test Win Rate
zerg 5__
protoss vs
terran 5 5
zerg vs...
protoss 5
terran
1.00
0.75
0.50
0.25
0.00
10M
10M
10M
10M 0
10M 0
10M
Step
Step 0
Step
Step
Step
Step
0.5
Figure 13: Comparison of the mean win rate when p 0, 0.5, for QMIX (left) and MAPPO (right) on SMACv2. Here, the available action mask is still present.
Mean Test Win Rate
_er__5_vs_
protoss $_vs_5
terran_5_0 5__v__
zerg_5_vs_5
$_vs_5
protoss_5 3_vs_
terran_5 ___vv___
1.00
0.75
0.50
0.25
0.00
10M
10M
10M 0
10M 0
10M 0
10M
Step
Step
Step
Step
Step
Step
Figure 14: Comparison of mean test win rate when = 0.5, 1 with 5 agents against 5 enemy units, for QMIX (left) and MAPPO (right). Here, the available actions mask has been removed.
with ranged units. From observing episodes, the enemy AI tends to aggressively pursue allied units, which allows the allies to lure it of position. This is easier exploit with melee units than ranged ones. Terran scenarios have no melee units, but the marine scenarios are slightly easier. Overall these results show that generalising to different unit types is significant part of the challenge in SMACv2.
Our field-of-view ablations, shown in Figure I2c compare the fixed unit sight and attack ranges to the SMACv2 versions. There is increase in difficulty from varying the unit ranges, but this effect small. This suggests that the difficulty of SMACv2 better explained by the diversity start positions and unit types than the change to the true sight and attack ranges.
Finally, to evaluate the effect that the changes had on the partial observability of SMACv2, we compare feedforward network with the performance of the RNN baseline, shown in Figure [2d There is a significant decrease in performance from using the feedforward network. However, the feedforward network can still learn reasonable policies. This suggests that although being able to resolve partial observability useful SMACv2, the primary difficulty consists in generalising to range of micro-ne maaeroo-magemmenn scenarios. This is in contrast to the extended partial observability challenge, where resolving partial observability through communication is the primary difficulty. The zerg scenario has larger relative performance difference. Partial observability may be more of problem here because the splash damage done to allies by banelings creates an incentive for allies to spread out more.
Overall, the results in Figures| 12a 12b show that current MARL methods struggle with the increased stochasticity due to map randomisation. To solve this problem, agents must be able to use their observations to make inferences about relevant state or joint observation features in the Dec-POMDP in more general settings. Current inference capabilities could be improved by research on more effective sequence models, such as specialised transformers 16 55) The work in this area either applies transformer models to traditional RL methods (15), or focuses paradigms similar to upside-down RL
[39 []] While the latter work shows promising generalisation [34]. is mostly in the offline setting. Hu et al. [4] demonstrate promising sequence model that can be applied to any MARL algorithm, but only demonstrate their work on marine units, and rely on the Dec-POMDY having certain structure. Expanding such work to handle more diverse scenarios is interesting avenue of future work.
The code for the ablations can be found here This is distributed under an MIT license. The changes on this branch enable easy running of the ablations and add associated environment configurations for this purpose.
26

---

Table 10: SMACv2 Scenarios with their number of allies, number of enemies, and unit types.
Scenario Name
Number of Allies
Number of Enemies
Unit Types
protoss_5. _oooos___vv____ protoss_10 v__
Stalker, Zealot, Colossus
10
10
Stalker, Zealot, Colossus
protoss ________________________________
20
20
Stalker, Zealot, Colossus
protoss_10_vs___
10
11
Stalker, Zealot, Colossus
protoss_20_vs__
20
23
Stalker, Zealot, Colossus
serran_5_vs_3
Marine. Marauder, Medivac
1erran_10_vs_10
10
10
Marine, Marauder, Medivac
20
20
Marine, Marauder, Medivac
10rr___10_vs_11
Marine, Marauder, Medivac
terran_20, 2erran_00_vs_2_
20
23
Marine, Marauder, Medivac
serg_5_va_5
Zergling, Hydralisk, Baneling Zergling. Hydralisk, Baneling
zerg_10_we 2rrg_10_vs_10
10
10
zerg_ ver__20_vs_20
Zergling, Hydralisk, Baneling
zerg_ 10__10__s_11
10
11
Zergling, Hydralisk, Baneling
zerg_20_vs_22
20
23
Zergling. Hydralisk, Baneling
F
Limitations and Broader Impact
The main limitation of SMACv2 is that it is confined to scenarios within the game of StarCraft II. This is an environment which, while complex, cannot represent the dynamics of all multi-agent tasks. Evaluation of MARL algorithms therefore should not be limited to one benchmark, but should target variety with range of tasks.
Whilst the scenarios in SMACv2 involve battles between two armies of units, only one side can be controlled by RI. agents. It is technically possible to control two armies using two StarCraft II clients that communicate LAN, which would allow train two groups of decentralised agents against each other, e.g., via self-play. We leave the implementation of this functionality as future work.
SMACv2 aims contribute to the development of MARL algorithms. As with any machine learning field, it is possible that improving the capabilities of these algorithms could lead to unethical uses. However, there are also many potential benefits to better cooperative AI, such applications in automated driving among others. We believe that the potential benefits of developing more capable and cooperative Al outweigh the potential risks.
27


Reviews:
[{"review_id": "44uqxfrzHI", "reviewer": "Reviewer_9mMA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "lYILQtYHrk", "reviewer": "Reviewer_QzEt", "summary": "", "questions": "", "limitations": "I have specific suggestions.\n\nThe Feature Inferrability and Relevance sections (5.2 and 7.3) look interesting, but I had some trouble understanding. Specifically, \n* In 7.3, important features (e.g., enemy health) were only briefly mentioned. If the detailed results for each mask can be presented in the style of Appendix Table 3 (SMACv2 feature quality experiment results, but only with everything and nothing masks), it would be very helpful.\n* Different mask conditions in the main figure 3, and appendix figures 3-4 are not clearly distinguished, and I could not find what the y-axis “Episode Running” meant. \n* “The Root-mean-squared error as a proportion of mean Q value” metric made more sense to me in comparing the effects of feature masking. If this metric was consistently used when comparing SMAC vs. SMACv2 (5.2 vs. 7.3), it would be helpful.\n* The columns between the Appendix Tables 2 and 3 should be matched, by adding the last column of the Appendix Table 3 to Table 2.\n\nWhen referring to the Appendix, clearly identify the relevant sections or tables to facilitate easy access and comprehension for readers.\n", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "This study is of excellent quality and would substantially benefit the RL research community. I very much look forward to seeing the next generation algorithms that can crack SMACv2! \n\nThe demonstration that open-loop algorithms, which were conditioned only on agent ids and timestep, can solve the SMAC was convincing and striking, highlighting the critical need for a new, challenging benchmark.\n\nThe introduction and successful validation of stochasticity in SMACv2, using the QMIX and MAPPO algorithms, confirmed that the new benchmark effectively addressed the weakness of SMAC. \n\nThe clever inclusion of the extended partial observability challenge in SMACv2, which promotes communication among agents, has opened up intriguing avenues for further research.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "H4tQzMfGHB", "reviewer": "Reviewer_sKYb", "summary": "", "questions": "", "limitations": "-", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Uncovers a fundamental issue in a popular MARL benchmark\n- Fixes it\n- Clear experiments showing the flaw and that the fix seems to work", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This work points out certain problems with the StarCraft testbed (SMAC) used frequently in co-operative MARL literature and provides a new benchmark SMACv2, where scenarios are procedurally generated and require agents to generalize to previously unseen settings. 
The authors further show that open-loop policies do not work well on the proposed SMACv2 testbed which poses new challenges. This work is solid and the authors will add promised new results to the revision to make it more comprehensive.
Decision: Accept (Poster)