SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Teams in SMACv2 are procedurally generated. Since StarCraft II does not allow units from different races to be on the same team, scenarios are divided depending on whether they feature Protoss, Terran or Zerg units. In scenarios with the same number of enemy and ally units, the unit types on each team are the same. Otherwise the teams are the same except for the extra enemy units, which are drawn identically to the other units. We use three unit types for each race, chosen to be the same unit types that were present in the original SMAC. We generate units in teams algorithmically, with each unit type having a fixed
Figure 4: Examples of the two different types of start positions, reflect and surround. Allied units are shown in blue and enemy units in dark red.
probability. The probabilities of generating each unit type are given in Table 9 in the appendix. These probabilities are the same at test time and train time.
Random start positions in SMACv2 come in two different flavours. In reflect scenarios, the allied units are spawned uniformly at random on one side of the scenario. The enemy positions are the allied positions reflected in the vertical midpoint of the scenario. This is similar to how units spawned in the original SMAC benchmark but without clustering them together. In surround scenarios, allied units are spawned at the centre and surrounded by enemies stationed along the four diagonals. Figure 4 illustrates both flavours of start position.
The final change is to the sight range and attack range of the units. In the original SMAC, all unit types had a fixed sight and attack range, allowing short- and long-ranged units to choose to attack at the same distance. This did not affect the real attack range of units within the game – a unit ordered to attack an enemy while outside its striking range would approach the enemy before attacking. We change this to use the values from SC2, rather than the fixed values. However, we impose a minimum attack range of 2 because using the true attack ranges for melee units makes attacking too difficult.
Taken together, these changes mean that before an episode starts, agents no longer know what their unit type or initial position will be. Consequently, by diversifying the range of different scenarios the agents might encounter, they are required to understand the observation and state spaces more clearly, which should render learning a successful open-loop policy not possible. We also define a convenient interface for defining distributions over team unit types and start positions. By only implementing a small Distribution class one can easily change how these are generated.
Akin to the original version of SMAC [37], we split the scenarios into symmetric and asymmetric groups. In the symmetric scenarios, allies and enemies have the same number and type of units. In the asymmetric scenarios, the enemies have some extra units chosen from the same distribution as the allies. There are 5 unit, 10 unit and 20 unit symmetric scenarios, and 10 vs 11 and 20 vs 23 unit asymmetric scenarios for each of the three races.
6.1 Extended Partial Observability Challenge
We now introduce the Extended Partial Observability (EPO) challenge, where we make additional modifications to SMACv2 to ensure an extension of SMAC with meaningful partial observability. While SMAC is clearly partially observable in the sense that agents do not observe the global state, this partial observability is not particularly meaningful. SMAC certainly occludes information that would be relevant to each agent through the use of fixed sight ranges. However, the ranges of attack actions are always less than sight ranges and so ally teams can do without communication by moving within firing range and attacking any observed enemies. Finding enemies is not itself a significant challenge, as ally and enemy units spawn closely together. Therefore, much of the information that is relevant to the action selection of a given agent is within its observation.
To create a benchmark setting with meaningful partial observability, we introduce two further changes to SMACv2. First, enemy observations are stochastically masked for each agent. Once an enemy has been observed for the first time within an episode, the agent that was first to see it is guaranteed to observe it as normal. If two or more agents observe an enemy that had not yet been observed on the same timestep, there is a random tie break. As soon as this initial sighting is made, a random binary draw occurs for all other agents, which determines whether an agent will be able to observe that enemy for the remainder of the episode if it enters the agent's sight range. Agents for which the draw is unsuccessful have their observations masked such that they cannot observe any information about that enemy for the rest of the episode, even if it enters their sight range. The draw has a tunable probability  $p$  of success. We suggest  $p = 0$  in the challenge. Figure 5 illustrates this setup.
Figure 5: Example EPO enemy sighting. Allied units that do not observe the enemy are shown in blue, those that do are shown in green and the enemy unit in dark red. Initially, an ally spots an enemy. Later (right), when the enemy is within all allied sight ranges, only the first ally to observe the enemy and the ally for which the draw was successful can see it.
Figure 6: Comparison of the mean test win rate of QMIX, MAPPO, IPPO, QPLEX and an open-loop policy on SMACv2. Plots show the mean and standard deviation across 3 seeds.
In SMAC, the environment provides an available action mask that guarantees agents can only choose attack actions if the corresponding enemy is within shooting range. In this setting only, we remove this available actions mask. This makes deciding on a target more difficult because agents cannot rely on the available actions to infer which enemies they can attack. Hence they must either infer or communicate this information. If an invalid action is chosen, it becomes a no-op. We also found that without removing the available actions, there was little separation between the  $p = 1$  (i.e. all allies can target all enemies) and  $p = 0$  (i.e. only the first ally to see an enemy can target it) cases, as shown in Figure 13 in the appendix. We believe that this is because the available actions mask encourages heavy weight to be placed on attack actions, knowing that they will not be chosen unless available. In EPO, because of the increased difficulty, 6 allied agents battle against 5 enemy units. We found that this increased the gap between the  $p = 1$  and  $p = 0$  settings. We believe this is because in the more difficult 5_vs_5 setting, agents are too preoccupied with other more basic challenges for target prioritisation to be a large issue. Results for the 5_vs_5 setting are shown in Figure 14 in the appendix and for the 6_vs_5 setting in Figure 7. In EPO, there are 3 maps, one for each race, with 6 allies and 5 enemies per team. All other settings such as unit types and start positions are identical to SMACv2. We recommend a setting of  $p = 0$ .
7 SMACv2 Experiments
In this section, we present results from a number of experiments on SMACv2. We first train state-of-the-art SMAC baselines on SMACv2 to assess its difficulty. Currently, both value-based and policy optimisation approaches achieve strong performance on all SMAC scenarios [55, 13]. As baselines we use QMIX [32], a strong value-based method and MAPPO [55], a strong policy optimisation method. We also investigate the effect of the new SMACv2 environment features with ablations. These results are available in Figure 12 in the appendix.
7.1 SMACv2 Baseline Comparisons
First we compare the performance of MAPPO, QMIX, QPLEX[51] and IPPO[8] and an open-loop policy, based on MAPPO, on SMACv2. We run each algorithm for 10M training steps and 3 environment seeds. For MAPPO, we use the implementation from Sun et al. [43] and for QMIX the one from Hu et al. [13]. The results of the runs of the best hyperparameters are shown in Figure 6.
These results shown in Figure 6 reveal a few trends. First, QMIX generally performs better than MAPPO across most scenarios. QMIX strongly outperforms MAPPO in two Protoss scenarios (protoss_20_vs_20 and protoss_10_vs_11), but otherwise final performance is similar, although QMIX is more sample efficient. However, QMIX is memory-intensive because of its large replay buffer and so requires more compute to run than MAPPO. Additionally, MAPPO appears to still be
Figure 7: Comparison of mean test win rate when  $p = 0, 0.5, 1$ , with 6 agents against 5 enemy units, for QMIX (left) and MAPPO (right). Here, the available actions mask has been removed.
increasing its performance towards the end of training in several scenarios, such as the 10_vs_11 and 20_vs_20 maps. It is possible that MAPPO could attain better performance than QMIX if a larger compute budget were used. MAPPO and IPPO perform nearly identically across the maps, suggesting MAPPO gains little benefit from the centralised state. QPLEX performs worse than QMIX across a number of maps, particularly the 20_vs_20 maps.
Roughly even win rates are attained on the symmetric maps. By contrast, on the asymmetric maps, win rates are very low, particularly on the 20_vs_23 scenarios. Additionally, there is no real change in difficulty as the number of agents scales. However, the maps do not seem uniformly difficult, with all algorithms struggling with the Zerg scenarios more than other maps. Reassuringly the open-loop method cannot learn a good policy on any maps - even those where QMIX and MAPPO both achieve high win rates. This is strong evidence that SMAC's lack of stochasticity has been addressed.
7.2 EPO Baselines
We now compare QMIX and MAPPO on EPO. A comparison between the  $p = 1$ ,  $p = 0$  and  $p = 0.5$  settings is shown in Figure 7. In this setting,  $p = 0$  struggles to attain even modest performance, whereas  $p = 1$  comes close to solving the task in a number of instances. This leaves a lot of room for the development of algorithms that learn to use implicit communication as a means of overcoming meaningfully restrictive partial observability and reduce the performance gap between the  $p = 1$  and  $p = 0$  settings. The performance in the  $p = 0.5$  case is similar to the  $p = 0$  case. This suggests that any communication protocol would have to be reasonably robust to have an impact on performance. This makes sense – an additional  $50\%$  chance to observe an enemy would still make it difficult to perform group target prioritisation. In general QMIX has a smaller gap between the  $p = 0$  and  $p = 1$  cases, suggesting it is perhaps better able to use the global state to resolve partial observability differences than MAPPO.
7.3 SMACv2 Feature Inferrability & Relevance
In this section we repeat the experiments from Section 5.2 on the 5 unit SMACv2 scenarios. We use three previously trained QMIX policies to generate 3 different datasets for training, and compute error bars in the regression across these three policies. The results are given in Table 3 and Figure 11 in the appendix.
When comparing results between SMAC and SMACv2, we focus on the difference between the everything (all features not visible) and nothing (no features hidden) masks. This is to account for some scenarios having much higher errors in the nothing mask than others. For example,  $5\mathrm{m\_vs\_6m}$  has an error as a proportion of the mean Q-value in the nothing mask of 0.14, compared to 0.018 for  $2\mathrm{c\_vs\_64zg}$ . The SMACv2 scenarios have higher errors than any of the SMAC scenarios when subtracting  $\frac{\epsilon_{\mathrm{rms}}}{Q}$  for the everything and nothing masks. For example, this value is 0.12 for 5_gen_protoss, which has the lowest value for any tested SMACv2 map. This is significantly higher than the 0.07 for  $5\mathrm{m\_vs\_6m}$ , which has the highest value for any SMAC map. Tables 2 and 3 in the appendix list the full values for SMAC and SMACv2. These results suggest that SMACv2 is significantly more stochastic than SMAC. Second, examining stepwise results in Figure 11 in the appendix shows that SMACv2 maps have similar patterns of feature importance to SMAC maps. Enemy health is the most important individual feature and is important throughout an episode for all scenarios tested. Importantly, both the ally and enemy all masks reach significant thresholds of the
average  $Q$ -value, suggesting that it is important in SMACv2 for methods to attend to environment features.
Using a set of novel experimental evaluations, we showed that the popular SMAC benchmark for cooperative MARL suffers from a lack of stochasticity and meaningful partial observability. We then proposed SMACv2 and repeated these evaluations, demonstrating significant alleviation of this problem. We also demonstrated that baselines that achieve strong performance on SMAC struggle to achieve high win rates on the new scenarios in SMACv2. We performed ablations and discovered that the stochasticity of the unit types and random start positions combine to explain the tasks' difficulty. We also introduced and evaluated current methods on the extended partial observability (EPO) challenge and demonstrated that meaningful partial observability is a significant contributor to the challenge in this environment.
Gorsane et al. [11] found that it was most common for MARL papers accepted at top conferences to only evaluate on one benchmark, relying on the multiple scenarios within that benchmark to provide sufficient variety. However, in addition to the issues shown in that paper with cherry-picking of scenarios within a benchmark, this evaluation protocol is vulnerable to issues in environment dynamics, such as SMAC's lack of stochasticity, and community-wide overfitting. Given the difficulty in designing both benchmarks and sufficiently diverse scenarios within them, we recommend that evaluating on multiple benchmarks should become the norm where possible to help avoid issues.
We hope that SMACv2 can contribute to MARL research as a significantly challenging domain capturing practical challenges.
Benjamin Ellis and Jonathan Cook are supported by the EPSRC centre for Doctoral Training in Autonomous and Intelligent Machines and Systems EP/S024050/1. The experiments were made possible by a generous equipment grant from NVIDIA.
[1] Torchcraftai: A bot platform for machine learning research on starcraft: Brood war. https://github.com/TorchCraft/TorchCraftAI, 2018. GitHub repository.
[2] Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280:103216, 2020.
[3] Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4): 819-840, 2002.
[4] Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/f5b1b89d98b7286673128a5fb112cb9a-Paper.pdf.
[5] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 2021.
[6] Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for OpenAI Gym. https://github.com/maximecb/gym-minigrid, 2018.
[49] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Kuttler, John Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, Hado van Hasselt, David Silver, Timothy Lillicrap, Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp, and Rodney Tsing. Starcraft ii: A new challenge for reinforcement learning, 2017. URL https://arxiv.org/abs/1708.04782.
[50] Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michael Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Remi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Çaglar Güçehre, Ziyu Wang, Tobias Pfaff, Yuhui Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nat., 575(7782):350-354, 2019. doi: 10.1038/s41586-019-1724-z.
[51] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. In International Conference on Learning Representations, 2020.
[52] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforcement learning with emergent roles. arXiv preprint arXiv:2003.08039, 2020.
[53] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. In International Conference on Learning Representations, 2020.
[54] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5571-5580. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/yang18d.html.
[55] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.
A Links
- SMACv2: https://github.com/oxwhirl/smacv2. MIT License.
- MAPPO implementation: https://github.com/benellis3/mappo.
- QMIX implementation: https://github.com/benellis3/pymarl2. Apache-2 License.
B Further Background
B.1 StarCraft
StarCraft II is a real-time strategy game featuring 3 different races, Protoss, Terran and Zerg, with different properties and associated strategies. The objective is to build an army powerful enough to destroy the enemy's base. When battling two armies, players must ensure army units are acting optimally. This is called micromanagement. An important micromanagement strategy is focus firing, which is ordering all allied units to jointly target the enemies one by one to ensure that damage taken is minimised.
Another important strategy is kiting, where units flee from the enemy and then pick them off one by one as they chase.
B.2 QMIX
QMIX can be thought of as an extension of DQN[29] to the Dec-POMDP setting. The joint optimal action is found by forcing the joint  $Q$  to adhere to the individual global max (IGM) principle[40], which states that the joint action can be found by maximising individual agents'  $Q_{i}$  functions:
$$
\operatorname * {a r g   m a x} _ {\boldsymbol {a}} Q (s, \boldsymbol {\tau}, \boldsymbol {a}) = \left\{ \begin{array}{l} \arg \max  _ {a} Q _ {1} (\tau_ {1}, a _ {1}) \\ \arg \max  _ {a} Q _ {2} (\tau_ {2}, a _ {2}) \\ \dots \\ \arg \max  _ {a} Q _ {n} (\tau_ {n}, a _ {n}) \end{array} \right.
$$
This central  $Q$  is trained to regress to a target  $r + \gamma \hat{Q}(s, \tau, \mathbf{a})$  where  $\hat{Q}$  is a target network that is updated slowly. The central  $Q$  estimate is computed by a mixing network, whose weights are conditioned on the state, which takes as input the utility function  $Q_{i}$  of the agents. The weights of the mixing network are restricted to be positive, which enforces the IGM principle[40] by ensuring the central  $Q$  is monotonic in each  $Q_{i}$ .
B.3 Independent and Multi-agent PPO
Proximal Policy Optimisation (PPO) is a method initially developed for single-agent reinforcement learning which aims to address performance collapse in policy gradient methods. It does this by heuristically bounding the ratio of action probabilities between the old and new policies. To this end, it optimises the below objective function.
$$
\mathbb {E} _ {s \sim d ^ {\pi}, a \sim \pi} \left[ \min  \left(\frac {\tilde {\pi} (a | s)}{\pi (a | s)} A ^ {\pi} (s, a), \operatorname {c l i p} \left(\frac {\tilde {\pi} (a | s)}{\pi (a | s)}, 1 - \epsilon , 1 + \epsilon\right) A ^ {\pi} (s, a)\right) \right]
$$
where  $\operatorname{clip}(t, a, b)$  is a function that outputs  $a$  if  $t < a$ ,  $b$  if  $t > b$  and  $t$  otherwise.
Extending this to the multi-agent setting can easily done in two ways. The first is to use independent learning, where each agent treats the others as part of the environment and learns a critic using its observation and action history. The second is to use a centralised critic conditioned on the central state. This is called multi-agent PPO. Both independent PPO (IPPO) and multi-agent PPO (MAPPO) have demonstrated strong performance on SMAC[55, 8]. Note that we do not apply the observation and state changes suggested by Yu et al. [55] anywhere in this paper. This is because these changes were not implemented as a wrapper on top of SMAC, but instead by modifying SMAC directly, leading to the environment code becoming unmanageably complicated.
(a) MAPPO open-loop and closed-loop results
(b) QMIX open-loop and closed-loop results
Figure 8: Plot of selected SMAC scenarios treated as an open-loop planning problem by limiting the observation to the current timestep and agent ID. Plots show the mean win rate and standard deviation across 3 training seeds for MAPPO and QMIX.
In this section we describe extra details of the experiments run as part of the paper.
C.1 Stochasticity
This section describes details of the experiments run in Section 5.1 in the main paper. Both the closed-loop and open-loop algorithms were based on the MAPPO implementation from Sun et al. [43]. The code used for these experiments can be found here. Both the open-loop and closed-loop algorithms use the same neural network architecture as in [43]. Both were also provided with access to the available actions mask of the environment and conditioned the critic on the state. We used the hyperparameters from Sun et al. [43], with the exception of the actor's learning rate, which we set to 0.0005 and decayed linearly throughout training. This is because learning rate decay has been found to be important for bounding PPO's policy updates [42]. Full hyperparameters can be found in Table 5. Full results are shown in Figure 8.
C.2 SMAC Feature Inferrability & Relevance
This section describes details of the experiments run in Section 5.2 and Section 7.3 in the main paper. The code used for these experiments can be found here.
For each scenario, we had 3 QMIX policies trained using the implementation and hyperparameters from Hu et al. [13], each with a different network and SMAC initialization. Training results for the policies for SMAC are shown in Figure 9. Each policy constitutes a seed and is used to collect a
Figure 9: QMIX training results on SMAC
dataset of episodes for the feature-relevance experiment on the scenario. We call these policies expert policies. QMIX hyperparameters used are given in Table 4. A dataset consists of two folds: 8192 episodes used for training and 4096 episodes used for evaluation.
For a given mask, the experiment then consists of training a new QMIX network, termed regression network, whose input is trajectories with observations and states masked according to the mask, and whose task is to predict the  $Q$ -values output by the expert policy on the unmasked trajectories.