SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Benjamin Ellis $^{1*}$  Jonathan Cook $^{1}$  Skander Moalla $^{1,5}$  Mikayel Samvelyan $^{2,3}$  Mingfei Sun $^{4}$  Anuj Mahajan $^{1}$  Jakob N. Foerster $^{1}$  Shimon Whiteson $^{1}$
<sup>1</sup>University of Oxford <sup>2</sup>University College London <sup>3</sup>Meta AI <sup>4</sup>University of Manchester <sup>5</sup>EPFL
The availability of challenging benchmarks has played a key role in the recent progress of machine learning. In cooperative multi-agent reinforcement learning, the StarCraft Multi-Agent Challenge (SMAC) has become a popular testbed for the centralised training with decentralised execution paradigm. However, after years of sustained improvement on SMAC, algorithms now achieve near-perfect performance. In this work, we conduct new analysis demonstrating that SMAC lacks the stochasticity and partial observability to require complex closed-loop policies (i.e., those that condition on the observation). In particular, we show that an open-loop policy conditioned only on the timestep can achieve non-trivial win rates for many SMAC scenarios. To address this limitation, we introduce SMACv2, a new benchmark where scenarios are procedurally generated and require agents to generalise to previously unseen settings during evaluation. We show that these changes ensure the benchmark requires the use of closed-loop policies. We also introduce the extended partial observability challenge (EPO), which augments SMACv2 to ensure meaningful partial observability. We evaluate state-of-the-art algorithms on SMACv2 and show that it presents significant challenges not present in the original benchmark. Our analysis illustrates that SMACv2 addresses the discovered deficiencies of SMAC and can help benchmark the next generation of MARL methods. Videos of training are available on our website.
In many real-world cases, control policies for cooperative multi-agent reinforcement learning (MARL) can be learned in a setting where the algorithm has access to all agents' observations, e.g., in a simulator or laboratory, but must be deployed where such centralisation is not possible. Centralised training with decentralised execution (CTDE) [10, 25] is a paradigm for tackling these settings and has been the focus of much recent research [32, 44, 12, 26, 27]. In CTDE, the learning algorithm sees all observations during training, as well as possibly the Markov state, but must learn policies that can be executed without such privileged information.
As in other areas of machine learning, benchmarks have played an important role in driving progress in CTDE. Examples of such benchmarks include the Hanabi Learning Environment [2], Google football [20], PettingZoo [46], and Multi-Agent Mujoco [31]. Perhaps the most popular one is the StarCraft Multi-Agent Challenge (SMAC) [37], which focuses on decentralised micromanagement challenges in the game of StarCraft II. Rather than tackling the full game with centralised control [50], SMAC tasks a group of learning agents, each controlling a single army unit, to defeat the units of the
*Correspondence to benellis@robots.ox.ac.uk.
$^{2}$ Code is available at https://github.com/oxwhirl/smacv2
(a) Terran
(b) Zerg
Figure 1: Screenshots from SMACv2 showing agents battling the built-in AI.
(c) Protoss
enemy army controlled by the built-in heuristic AI. SMAC requires learning several complex joint action sequences such as focus fire, $^{3}$  and kiting enemy units, $^{4}$ . For these reasons, many prominent MARL papers [51, 26, 52, 33, 9, 55, 53] rely on SMAC to benchmark their performance. $^{5}$
However, there is significant evidence that SMAC is outliving its usefulness [55, 13, 11]. Recent work reports near-perfect win rates on most scenarios [55, 13] and strong performance without using the centralised state via independent learning [55, 8]. This suggests that, due to ceiling effects, SMAC may no longer reward further algorithmic improvements and that new benchmarks are needed.
In this paper, we present new, more fundamental problems with SMAC and hence reason for new benchmarks. In particular, we show that an open-loop policy, which conditions only on the timestep and ignores all other observations, performs well on many SMAC scenarios. We also extend this analysis by examining the estimates of the joint  $Q$ -function learnt by QMIX. We demonstrate that, even with all features masked, regression to the joint  $Q$ -function is possible with less than  $10\%$  error in all but two scenarios from just timestep information.
Together these results suggest that SMAC is not stochastic enough to necessitate complex closed-loop (i.e. conditioned on the observation) control policies on many scenarios. Therefore, although SMAC scenarios may require difficult to discover action sequences such as focus fire, agents do not need to adapt to a diverse range of situations, but can largely repeat a fixed action sequence with little deviation. Additionally, meaningful partial observability (see Section 4) is minimal in SMAC due to large fields-of-view. For partial observability to be meaningful, one agent must observe information that is relevant to the current or future action selection of another agent, unknown to that other agent, and uninformable from the other agent's observation. Meaningful partial observability is crucial to decentralisation and the NEXP-completeness of Dec-POMDPs [3].
To address these shortcomings, we propose SMACv2, a new benchmark that uses procedural content generation (PCG) [36] to address SMAC's lack of stochasticity. In SMACv2, for each episode we randomly generate team compositions and agent start positions. Consequently, it is no longer sufficient for agents to repeat a fixed action sequence, but they must learn to coordinate across a diverse range of scenarios. To remedy the lack of meaningful partial observability, we introduce the extended partial observability challenge (EPO), where we only allow the first agent to spot an enemy to see it for certain when it is in range. This requires agents to implicitly communicate enemy information to prioritise targets. Additionally, we update the sight and attack ranges to increase the diversity of the agents and complexity of scenarios.
Our implementation is extensible, allowing researchers to implement new distributions easily and hence further expand on the available SMACv2 scenarios. We also analyse SMACv2 and demonstrate that the added stochasticity prevents an open-loop policy from learning successfully, but instead requires policies to condition on ally and enemy features.
We evaluate several state-of-the-art algorithms on SMACv2 and show that they struggle with many scenarios, confirming that SMACv2 poses substantial new challenges. Finally, we perform an ablation on the new observation features to demonstrate how each contributes to the difficulty, highlighting the specific areas on which future MARL research should focus.
3 In focused fire, allied units jointly attack and kill enemy units one after another.
${}^{4}$  Kiting draws enemy units to give chase while maintaining distance so that little damage is incurred.
SMAC has been cited over 600 times on Google Scholar.
The MARL community has made significant use of games for benchmarking cooperative MARL algorithms. The Hanabi Learning Environment [2] tasks agents with learning to cooperate in the card-game Hanabi. Here, each agent observes the cards of its teammates but not its own, which must be implicitly communicated via gameplay. Hanabi is partially observable and stochastic, but only features teams of 2 to 5 players, which is fewer than all but the smallest SMACv2 scenarios. Kurach et al. [20] propose Google Football as a complex and stochastic benchmark, and also have a multi-agent setting. However, it assumes fully observable states, which simplifies coordination.
Peng et al. [31] propose a multi-agent adaptation of the MuJoCo environment featuring a complex continuous action space. Multi-Particle Environments (MPE) [25] feature simple communication-oriented challenges where particle agents can move and interact with each other using continuous actions. In contrast to both multi-agent MuJoCo and MPE, SMACv2 has a discrete action space, but challenges agents to handle a wide range of scenarios using procedural content generation. A plethora of work in MARL uses grid-worlds [25, 23, 54, 4, 48, 35, 24] where agents move and perform a small number of discrete actions on a 2D grid, but these tasks have much lower dimensional observations than SMACv2. OpenSpiel [22] and PettingZoo [46] provide collections of cooperative, competitive, and mixed sum games, such as grid-worlds and board games. However, the cooperative testbeds in either of these suites feature only simple environments with deterministic dynamics or a small number of agents. Neural MMO [41] provides a massively multi-agent game environment with open-ended tasks. However, it focuses on emergent behaviour within a large population of agents, rather than the fine-grained coordination of fully cooperative agents. Furthermore, none of these environments combines meaningful partial observability, complex dynamics, and high-dimensional observation spaces, whilst also featuring more than a few agents that need to coordinate to solve a common goal.
StarCraft has been frequently used as a testbed for RL algorithms. Most work focuses on the full game whereby a centralised controller serves as a puppeteer issuing commands for the two elements of the game: macromanagement, i.e., the high-level strategies for resource management and economy, and micromanagement, i.e., the fine-grained control of army units. TorchCraft [45] and TorchCraftAI [1] provide interfaces for training agents on StarCraft: BroodWar. The StarCraft II Learning Environment (SC2LE) [49] provides a Python interface for communicating with the game of StarCraft II and has been used to train AlphaStar [50], a grandmaster-level but fully centralised agent that is able to beat professional human players. SMAC and SMACv2 are built on top of SC2LE and concentrate only on decentralised unit micromanagement for the CTDE setting.
One limitation of SMAC is the constant starting positions and types of units, allowing methods to memorise action sequences for solving individual scenarios (as we show in Section 5.1), while also lacking the ability to generalise to new settings at test time, which is crucial for real-world applications of MARL [28]. To address these issues, SMACv2 relies on procedural content generation [PCG; 36, 18] whereby infinite game levels are generated algorithmically and differ across episodes. PCG environments have recently gained popularity in single-agent domains [7, 6, 17, 21, 38] for improving generalisation in RL [19] and we believe the next generation of MARL benchmarks should follow suit. Iqbal et al. [15] and Mahajan et al. [28] consider updated versions of SMAC by randomising the number and types of the units, respectively, to assess the generalisation in MARL. However, these works do not include the random start positions explored in SMACv2, analyse the properties of SMAC to motivate these changes, or address SMAC's lack of meaningful partial observability. They also do not change the agents' field-of-view and attack ranges or provide a convenient interface to generate new distributions over these features.
A partially observable, cooperative multi-agent reinforcement learning task can be described by a decentralised partially observable Markov decision process (Dec-POMDP) [30]. This is a tuple  $(n,S,A,T,\mathbb{O},O,R,\gamma)$  where  $n$  is the number of agents,  $S$  is the set of states,  $A$  is the set of individual actions,  $T:S\times A^n\to \Delta (S)$  is the transition probability function,  $\mathbb{O}$  is the set of joint observations,  $O:S\times A^n\rightarrow \Delta (\mathbb{O})$  is the observation function,  $R:S\times A^n\rightarrow \Delta (\mathbb{R})$  is the reward function, and  $\gamma$  is the discount factor. We use  $\Delta (X)$  to denote the set of probability distributions over a set  $X$ . At
each timestep  $t$ , each agent  $i \in \{1, \dots, n\}$  chooses an action  $a \in A$ . The global state then transitions from state  $s \in S$  to  $s' \in S$  and yields a reward  $r \in \mathbb{R}$  after the joint action  $\mathbf{a}$ , according to the distribution  $T(s, \mathbf{a})$  with probability  $\mathbb{P}(s'|s, \mathbf{a})$  and  $R(s, \mathbf{a})$  with probability  $\mathbb{P}(r'|s, \mathbf{a})$ , respectively. Each agent then receives an observation according to the observation function  $O$  so that the joint observation  $\mathbf{o} \in \mathbb{O}$  is sampled according to  $O(s, \mathbf{a})$  with probability  $\mathbb{P}(\mathbf{o}|s, \mathbf{a})$ . This generates a trajectory  $\tau_i \in (\mathbb{O} \times A^n)^*$ . The goal is to learn  $n$  policies  $\pi_i: (\mathbb{O}_i \times A)^* \to \Delta(A)$  that maximise the expected cumulative reward  $\mathbb{E}[\sum_i \gamma^i r_{t+i}]$ .
3.2 StarCraft Multi-Agent Challenge (SMAC)
Rather than tackling the full game, the StarCraft Multi-Agent Challenge [37, SMAC] benchmark focuses on micromanagement challenges where each military unit is controlled by a single learning agent. Units have a limited field-of-view and no explicit communication mechanism when at test time. By featuring a set of challenging and diverse scenarios, SMAC has been used extensively by the MARL community for benchmarking algorithms. It consists of 14 micromanagement scenarios, which can be broadly divided into three different categories: symmetric, asymmetric, and micro-trick. The symmetric scenarios feature the same number and type of allied units as enemies. The asymmetric scenarios have one or more extra units for the enemy and are often more difficult than the symmetric scenarios. The micro-trick scenarios feature setups that require specific strategies to counter. For example,  $3s\_ vs\_ 5z$  requires the three allied stalkers to kite the five zealots, and corridor requires six zealots to block off a narrow corridor to defeat twenty-four zerglings without being swarmed from all sides. Two state-of-the-art algorithms on SMAC are MAPPO [55] and QMIX [32]. We provide background to these methods in Section 2 of the appendix.
4 Partial Observability and Stochasticity in CTDE
If the initial state and transition dynamics of a Dec-POMDP are deterministic, then open-loop policies suffice, i.e., there exists an optimal joint policy that ignores all information but the timestep and agent ID. Such a policy amounts to playing back a fixed sequence of actions. One way of determining whether SMAC has sufficient stochasticity therefore is to measure how strongly the learned policies condition on their observations. For example, in a mostly deterministic environment, a policy may only be required to condition on a small subset of the observation space for a few timesteps, whereas a more stochastic environment might require conditioning on a large number of observation features. We use this concept in section 5.2 to evaluate SMAC's stochasticity.
Since open-loop policies do not rely on observations, the observation function and any partial observability it induces only become relevant when the environment is stochastic enough to require closed-loop policies. However, even in stochastic environments not all partial observability is meaningful. If the hidden information is not relevant to the task, or can be inferred from the observation, it is resolvable by a single agent alone, and if it is unknown and undiscoverable by any of the agents, then it cannot contribute to solving the task. Without meaningful partial observability, a task cannot be considered truly decentralised because a policy can either ignore or infer any unknown information without the intervention of other agents. Meaningful partial observability is essential to the difficulty (and NEXP-completeness) of Dec-POMDPs [3]. While SMAC is technically partially observable, there is limited hidden information because large fields-of-view render most relevant observations common among agents.
5 Limitations of SMAC
In this section, we analyse stochasticity in SMAC by examining the performance of open-loop policies and the predictability of Q-values given minimal observations. We show that SMAC is insufficiently stochastic to require complex closed-loop policies.
5.1 SMAC Stochasticity
We use the observation that if the initial state and transition functions are deterministic, there always exists an optimal deterministic policy that conditions solely on the timestep information to investigate stochasticity in SMAC. In particular, we use a range of SMAC scenarios to compare
Figure 2: QMIX (left) and MAPPO (right) open-loop and closed-loop results.
policies conditioned only on the timestep (which we refer to as open-loop) to policies conditioned on the full observation history (i.e., all the information typically available to each agent in SMAC), which we refer to as closed-loop. The open-loop policies observe no ally or enemy information, and can only learn a distribution of actions at each timestep. If stochasticity in the underlying state is a serious challenge when training policies on SMAC scenarios, then policies without access to environment observations should fail. However, if the open-loop policies can succeed at the tasks, this demonstrates that SMAC is not representing the challenges of Dec-POMDPs well and so would benefit from added stochasticity.
We use MAPPO and QMIX as our base algorithms because of their widespread adoption and very strong performance and train open- and closed-loop versions of each. We train the open-loop policies on SMAC, but only allow the policies to observe the agent ID and timestep, whereas the closed-loop policies are given the usual SMAC observation as input with the timestep appended. The open-loop policies still have access to the central state as usual during training since this is only used in CTDE to aid in training and not during execution. The QMIX open-loop policy was trained only on the maps which the MAPPO open-loop policy was not able to completely solve. For QMIX, the hyperparameters used are the same as in [13] and for MAPPO the hyperparameters are listed in Appendix C.
Figure 2 shows the mean win rates and standard deviation of both policies on selected scenarios. The full results are available in Figure 8 in the appendix. Overall, the open-loop policies perform significantly worse than their closed-loop counterparts. However, this performance difference varies widely across scenarios and algorithms. For MAPPO, some scenarios, such as bane_vs_bane, 3s5z, 1c3s5z and 2s3z achieve open-loop performance indistinguishable from that of closed-loop. In other scenarios, such as 8m_vs_9m and 27m_vs_30m, there are large performance differences, but the open-loop policy still achieves high win rates. There are also scenarios, such as corridor and 2c_vs_64zg, where the MAPPO open-loop policy does not learn at all, whereas the closed-loop policies can learn successfully. QMIX also achieves good performance on 2c_vs_64zg, MM2, 27m_vs_30m and 8m_vs_9m, although for the latter maps the closed-loop policy strongly outperforms the open-loop QMIX. Altogether, there are only four SMAC maps where the open-loop approach cannot learn a good policy at all: 3s5z_vs_3s6z, corridor, 6h_vs_8z and 5m_vs_6m. Overall, open-loop policies perform well on a range of scenarios in SMAC. The easy scenarios show no difference between the closed-loop and the open-loop policies, and some hard and very hard scenarios show either little difference or non-trivial win rates for the open-loop policy. These successes are striking given the restrictive nature of the open-loop policy. This suggests that stochasticity is not a significant challenge for a wide range of scenarios in SMAC.
These results highlight an important deficiency in SMAC. Stochasticity is either not evaluated or not part of the challenge for the vast majority of SMAC maps. Not testing stochasticity is a major flaw for a general MARL benchmark because without stochasticity the policy only needs to take optimal actions along a single trajectory. Additionally, without stochasticity, there can be no meaningful partial observability. Given widespread use of SMAC as a benchmark to evaluate algorithms for Dec-POMDPs, this suggests that a new benchmark is required to evaluate MARL algorithms.
5.2 SMAC Feature Inferrability & Relevance
In this section, we look at stochasticity from a different perspective. We mask (i.e. "zero-out") all features of the state and observations on which QMIX's joint Q-function conditions. If the Q-values can still be easily inferred via regression, then SMAC does not require learning complex closed-loop
Figure 3: Comparison of the loss for different feature masks when regressing to a trained QMIX policy. The x-axis plots steps within an episode. 'Episodes Running' is the number of episodes that have not terminated by the given timestep. everything masks the entire observation and nothing masks no attributes. team_all masks all attributes of the team. last_action_only masks the last actions of all the allies in the state. The all Except Actions masks all the allied information except the last actions in the state. The mean is plotted for each mask and standard deviation across 3 seeds used as error bars. The low error rates for the everything mask imply that Q-values can be effectively inferred given only the timestep.
policies as Dec-POMDPs in general do, but is solvable by ignoring key aspects of StarCraft II gameplay. If the central  $Q$ -value could be inferred only from timestep information, this would also imply the task is not decentralised because all agents implicitly know the timestep and so could operate with the same central policy.
We also extend this experiment to masking subsets of the observation and state features. By only masking a subset of features, we measure the impact of different features on the learned policy. The more observation features the  $Q$ -function conditions on, the more confident we can be that SMAC requires understanding decentralised agent micromanagement rather than learning action sequences. We measure this conditioning by regressing to  $Q$ -values of a trained policy when masking different sets of features. The observation features hidden by each mask are in Table 1 in the appendix. Since we evaluate the regression performance of the joint  $Q$ -function, we always mask features in the state as well as the observation. More details of the regression procedure are in Appendix C.
Some interesting trends can be observed in Figure 3, which shows Q-value regression loss for each mask and scenario against steps in the RL episode. First, the root-mean-squared error of masking everything is low, reaching about  $15\%$  of the mean  $Q$ -value at peak, and  $5 - 10\%$  for most of the episode. Mean root-mean-squared error as a proportion of mean  $Q$ -value, as well as other metrics, is given in the appendix in Table 2. For all scenarios except  $5\mathrm{m\_vs\_6m}$  this value is below 0.12. These are not much higher than the baseline nothing masks, suggesting the observation features do not really inform the Q-value. These results highlight the lack of stochasticity and meaningful partial observability in SMAC and the necessity of novel benchmarks to address these shortcomings.
6 SMACv2
As shown in the previous section, SMAC has some significant drawbacks. To address these, we propose three major changes: random team compositions, random start positions, and increasing diversity among unit types by using the true unit attack and sight ranges. These changes increase stochasticity and so address the deficiency discovered in the previous section.