SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The nothing mask does not apply any effect on the observations and states in a trajectory. Otherwise, a mask, say 'health (ally)', masks the health feature of every ally in the observation of each agent (except its own health) and masks the health feature of all the units controlled by QMIX in the state. Table 1 shows the feature sets zeroed out for different masks.
The regression network has the same architecture as the expert network. We use the mean squared error (MSE) as a loss function and optimise the network via Adam with a batch size of 512 episodes and learning rate equal to 0.005. (The other Adam hyper-parameters are the default PyTorch values.) Training is performed with early stopping according to the validation fold with an evaluation every 5 epochs and a patience of 10 tries (i.e. training is stopped when the MSE on the validation fold does not decrease in 50 consecutive epochs and performance at the best epoch is retained). Models for all masks and scenarios hit early stopping and trained for about 200 epochs on average, except a few ones which trained to a limit of 500 epochs.
All hyper-parameters were tuned on a few different scenarios from both SMAC and SMACv2 to minimise validation MSE using datasets and expert policies not used for the reported results. The sizes of the training and validation folds are respectively 1.6 times and 0.8 times the size of the QMIX replay buffer (5000 episodes) and have been chosen to be large enough to minimize validation MSE while keeping experiments practical.
Figure 10 shows the full results of the feature quality experiments for SMAC and Figure 11 shows the full results for SMACv2. Table 3 shows summary results for the everything and nothing masks for SMACv2 and Table 2 shows the same data for SMAC.
Experiments for this section were conducted on 80-core CPU machines with NVIDIA GeForce RTX 2080 Ti or Tesla V100-SXM2-16GB GPUs. A single (scenario, seed, mask) combination took around 1 hour to train. In total, the experiments in this section took about 1500 GPU hours.
C.3 SMACv2 Runs
Here we describe the hyperparameter and training procedure used in section 7.1. As in previous experiments, we used the implementation by Hu et al. [13] for QMIX and by Sun et al. [43] for MAPPO. These implementations have both achieved very strong results on SMAC. We therefore tuned the hyperparameters by taking deviations from these for a few key parameters. For MAPPO, we tuned the actor's learning rate and the clipping range. Additionally, we added linear learning rate
Table 1: Masked features for each setting of the feature quality experiment.  $\checkmark$ means a feature is masked and  $x$ means a feature is not masked.
Mask	Ally	Enemy
Health	Shield	x	y	Distance	Actions	Health	Shield	x	y	Distance
everything	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
nothing	X	X	X	X	X	X	X	X	X	X	X
health (ally)	✓	X	X	X	X	X	X	X	X	X	X
shield (ally)	X	✓	X	X	X	X	X	X	X	X	X
distance (ally)	X	X	✓	✓	✓	X	X	X	X	X	X
health and shield (ally)	✓	✓	X	X	X	X	X	X	X	X	X
actions only	X	X	X	X	X	✓	X	X	X	X	X
all except actions	✓	✓	✓	✓	✓	X	X	X	X	X	X
ally all	✓	✓	✓	✓	✓	✓	X	X	X	X	X
health (enemy)	X	X	X	X	X	X	✓	X	X	X	X
shield (enemy)	X	X	X	X	X	X	X	✓	X	X	X
distance (enemy)	X	X	X	X	X	X	X	X	✓	✓	✓
enemy all	X	X	X	X	X	X	✓	✓	✓	✓	✓
Figure 10: Results of the feature quality experiments for all tested SMAC scenarios not in the main paper.
Figure 11: Feature Quality Experiments for the 5 and 10 unit SMACv2 scenarios.
— own_health	— distance	— all Except Actions	— own_fov
— everything	— shield	— nothing	— team_all
— health_and_shield	— health	— own_position	— last_action_only
decay to the MAPPO implementation because this has been shown to be important to bound the policy update[42]. For QMIX, we tuned the  $\epsilon$ -annealing time and  $\lambda$  in the eligibility trace  $Q(\lambda)$ . All other hyperparameters, including neural network architecture, were unchanged from the implementations mentioned previously.
The MAPPO code for these experiments can be found here, and the QMIX code here. The QMIX code is distributed under the Apache license, and the MAPPO code under the MIT license. The only differences in these branches and the implementations used for previous experiments on SMAC are important to changing the environment from SMAC to SMACv2.
The open-loop policy is identical to MAPPO, except the policy receives as input only the timestamp and agent ID, as in Section 5.1.
We tuned hyperparameters by running each set of hyperparameters on all scenarios and then choosing the best results. The grids of hyperparameters for QMIX and MAPPO can be found in Table 7 and 8. For IPPO, we used mostly the same hyperparameters as MAPPO. For QPLEX we used the hyperparameters from [51]. All algorithms were trained for 10M environment steps with evaluations
Table 3: SMACv2 feature quality experiment results
Map	Mask	Q	εrmse	εrmse/Q	εabs	εmask/εrmse/Q
5_gen_protoss	everything	8.4 ± 0.05	1.5 ± 0.05	0.179 ± 0.006	1.18 ± 0.04	0.119 ± 0.004
ally_all Except_actions	0.69 ± 0.02	0.082 ± 0.004	0.52 ± 0.01	0.022 ± 0.002
ally_health_and_shield	0.62 ± 0.02	0.074 ± 0.004	0.45 ± 0.02	0.014 ± 0.002
ally_health	0.54 ± 0.03	0.064 ± 0.005	0.39 ± 0.02	0.004 ± 0.001
ally_distance	0.52 ± 0.04	0.062 ± 0.006	0.38 ± 0.03	0.0019 ± 0.0005
ally_shield	0.52 ± 0.04	0.062 ± 0.006	0.38 ± 0.03	0.0018 ± 0.0008
ally_all	0.85 ± 0.009	0.102 ± 0.004	0.656 ± 0.01	0.042 ± 0.002
enemy_health_and_shield	0.79 ± 0.03	0.094 ± 0.004	0.6 ± 0.03	0.034 ± 0.003
enemy_health	0.69 ± 0.05	0.082 ± 0.007	0.51 ± 0.04	0.022 ± 0.003
enemy_distance	0.56 ± 0.03	0.067 ± 0.006	0.41 ± 0.03	0.0074 ± 0.0002
enemy_shield	0.57 ± 0.03	0.068 ± 0.005	0.42 ± 0.02	0.008 ± 0.002
enemy_all	0.96 ± 0.02	0.116 ± 0.004	0.74 ± 0.02	0.055 ± 0.002
nothing	0.5 ± 0.03	0.06 ± 0.006	0.37 ± 0.03
5_gen_terran	everything	8.2 ± 0.1	2.0 ± 0.1	0.243 ± 0.009	1.6 ± 0.1	0.148 ± 0.008
ally_all Except_actions	1.0 ± 0.05	0.1221 ± 0.001	0.75 ± 0.04	0.028 ± 0.002
ally_health_and_shield	0.82 ± 0.04	0.1 ± 0.001	0.6 ± 0.03	0.0052 ± 0.0009
ally_health	0.83 ± 0.04	0.1007 ± 0.0003	0.6 ± 0.04	0.0061 ± 0.0007
ally_distance	0.78 ± 0.04	0.0945 ± 9e-05	0.56 ± 0.03	0.0002 ± 0.0007
ally_shield	0.75 ± 0.02	0.092 ± 0.002	0.55 ± 0.02	-0.003 ± 0.001
ally_all	1.18 ± 0.06	0.145 ± 0.002	0.92 ± 0.06	0.05 ± 0.003
enemy_health_and_shield	0.98 ± 0.06	0.12 ± 0.003	0.72 ± 0.05	0.025 ± 0.003
enemy_health	0.97 ± 0.07	0.118 ± 0.003	0.71 ± 0.05	0.023 ± 0.003
enemy_distance	0.82 ± 0.04	0.1002 ± 0.0002	0.59 ± 0.03	0.0055 ± 0.0006
enemy_shield	0.75 ± 0.05	0.091 ± 0.005	0.54 ± 0.04	-0.004 ± 0.004
enemy_all	1.2 ± 0.1	0.151 ± 0.007	0.95 ± 0.09	0.057 ± 0.008
nothing	0.78 ± 0.03	0.0945 ± 0.0007	0.56 ± 0.03
5_gen_zerg	everything	7.2 ± 0.05	2.41 ± 0.05	0.33 ± 0.02	1.86 ± 0.04	0.176 ± 0.009
ally_all Except_actions	1.36 ± 0.02	0.188 ± 0.008	0.99 ± 0.02	0.031 ± 0.002
ally_health_and_shield	1.15 ± 0.03	0.16 ± 0.005	0.83 ± 0.03	0.002 ± 0.001
ally_health	1.17 ± 0.03	0.161 ± 0.005	0.83 ± 0.02	0.0037 ± 0.0008
ally_distance	1.18 ± 0.02	0.164 ± 0.007	0.85 ± 0.02	0.006 ± 0.001
ally_shield	1.13 ± 0.02	0.157 ± 0.006	0.81 ± 0.02	-0.0005 ± 0.0003
ally_all	1.46 ± 0.02	0.202 ± 0.01	1.09 ± 0.02	0.044 ± 0.002
enemy_health_and_shield	1.31 ± 0.03	0.18 ± 0.008	0.95 ± 0.03	0.024 ± 0.002
enemy_health	1.3 ± 0.02	0.179 ± 0.007	0.94 ± 0.02	0.022 ± 0.001
enemy-distance	1.24 ± 0.01	0.173 ± 0.008	0.88 ± 0.02	0.014 ± 0.002
enemy_shield	1.14 ± 0.02	0.159 ± 0.006	0.82 ± 0.02	0.0005 ± 0.0008
enemy_all	1.67 ± 0.01	0.23 ± 0.01	1.24 ± 0.02	0.074 ± 0.008
nothing	1.14 ± 0.02	0.157 ± 0.007	0.81 ± 0.02
of 20 episodes every 2k steps. The hyperparameters used for QMIX are given in Table 4, for MAPPO and IPPO in Table 5, and for QPLEX in Table 6.
C.4 EPO Runs
This section provides the implementation details for the Extended Partial Observability challenge in SMACv2 and outlines the corresponding training procedure.
For the EPO baselines in Section 7.2, we used the same hyperparameters as Section 7.1. Implementations of MAPPO and QMIX were also consistent with those in Section 7.1. Each training run and corresponding evaluations were conducted across 3 random seeds. Experiments for this section were conducted on 80-core CPU machines with NVIDIA GeForce RTX690 2080 Ti or Tesla V100-SXM2-16GB GPUs. Both MAPPO and QMIX experiments took between 36-48 hours to complete, each running on 8 GPUs, for  $3p$  values, across 3 seeds and 3 races.
D Environment Additional Details
D.1 SMACv2 Additional Details
In this section we describe the generation of teams in SMACv2. For each race, 3 different types of units are used. Each race has a special unit that should not be generated too often. For Protoss this is the colossus. This is a very powerful unit. If a team has many colossi, the battle will devolve into a war about who can use their colossi most effectively. Terran has the medivac unit. This is a healing unit that cannot attack the enemy and so is only spawned sparingly. Zerg has the baneling unit. This is a suicidal unit which deals area-of-effect damage to enemy units by rolling into them and exploding. In scenarios with many banelings, the agents learn to spread out and hide in the corners
Table 4: QMIX hyperparameters used for experiments. Parameters with (SMAC) or (SMACv2) after them denote that parameter setting was only used for SMAC or SMACv2 experiments respectively. These are the values in the corresponding configuration file in PyMarl[37, 13]. Mac is the code responsible for marshalling inputs to the neural networks, learner is the code used for learning and runner determines whether experience is collected in serial or parallel.
Parameter Name	Value
Action Selector	epsilon greedy
ε Start	1.0
ε Finish	0.05
ε Anneal Time	100000
Runner	parallel
Batch Size Run	4
Buffer Size	5000
Batch Size	128
Optimizer	Adam
tmax	10050000
Target Update Interval	200
Mac	n_mac
Agent	n_rnn
Agent Output Type	q
Learner	nq_learner
Mixer	qmix
Mixing Embed Dimension	32
Hypernet Embed Dimension	64
Learning Rate	0.001
λ (SMAC)	0.6
λ (SMACv2)	0.4
with the hope that the enemy banelings explode and the allies win by default. All of these special units are spawned with a probability of  $10\%$ . The other units used spawn with a probability of  $45\%$ . This is summarised in Table 9.
There are two changes to the observation space from SMAC. First, each agent observes their own field-of-view direction. Secondly, each agent observes their own position in the map as x- and y-coordinates. This is normalised by dividing by the map width and height respectively. The only change to the state from SMAC was to add the field-of-view direction of each agent to the state.
Additionally, we made one small change to the reward function in SMACv2. This was to fix a bug where the enemies healing can give allied units reward. There are more details available about this problem in the associated Github issue. SMACv2 also has an identical API to the original SMAC, allowing for very simple transition between the two frameworks.
The code for SMACv2 can be found in the Github repo, where there is a README detailing how to run the benchmark with random agents.
D.2 EPO Additional Details
In EPO, the first ally agent to observe an enemy is guaranteed to observe the usual SMAC features associated with that enemy. Upon an enemy being observed for the first time, by any agent on the ally team, a random binary draw occurs for all other agents (i.e., those that had not yet observed this particular enemy). Random tie breaking ensures only one agent is guaranteed to see the enemy should two or more observe it for the first time on the same timestep. The draw is weighted by a tunable environmnet parameter  $p$  corresponding to the probability of success. If the draw is successful for a particular agent, any future instances of the agent observing the enemy will occur as normal, without masking. If the draw is unsuccessful, that agent will not be able to observe the enemy on future timesteps, irrespective of whether or not it is within sight range. If the first agent to have observed the enemy dies, the next time the enemy falls within an ally agent's sight range that agent is guaranteed to see the enemy and the random draw occurs again for all other agents.
Table 5: MAPPO and IPPO hyperparameters used for the experiments on SMAC and SMACv2. Parameters with (SMAC) or (SMACv2) after them denote that parameter setting was only used for SMAC or SMACv2 experiments respectively.
Hyperparameter	Value
Action Selector	multinomial
Mask Before Softmax	True
Runner	parallel
Buffer Size	8
Batch Size Run	8
Batch Size	8
Target Update Interval	200
Learning Rate Actor (SMAC)	0.001
Learning Rate Actor (SMACv2)	0.0005
Learning Rate Critic	0.001
τ	0.995
λ	0.99
Agent Output Type	pi_logits
Learner	trust_region_learner
Critic (MAPPO)	centralV_rnn_critic
Critic (IPPO)	decentral_rnn_critic
Detach Every	20
Replace Every	None
Mini Epochs Actor	10
Mini Epochs Critic	10
Entropy Loss Coeff	0.0
Advantage Calc Method	GAE
Bootstrap Timeouts	False
Surrogate Clipped	True
Clip Range	0.1
Is Advantage Normalized	True
Observation Normalized	True
Use Popart	False
Normalize Value	False
Obs Agent Id	True
Obs Last Action	False
E Analysis of Changes in SMACv2
To investigate the impact of the changes introduced in SMACv2, we perform three ablation studies. We focus only on the 5 unit scenarios, and only train MAPPO on the ablations because it is significantly faster to train than QMIX. We use the same hyperparameters and setup as in Section 7.1. We ablate different team compositions by using a single unit type for each of the non-special units in each race. We investigate random start positions by only using the surround or reflect scenarios and by removing the random start positions entirely. We also ablate the effect of using the true unit ranges. Additionally, we investigate partial observability by comparing feed-forward and recurrent networks.
The results of these experiments are shown in Figure 12. The position ablations show that stochastic starting positions and hence stochasticity itself contributes greatly to the challenge of SMACv2. Without random start positions, MAPPO achieves win rates of close to  $100\%$ . Both surround and reflect scenarios have a similar win rate, suggesting similar difficulty. There does not appear to be additional difficulty from combining the two. This is logical as the scenarios can be disambiguated using start positions at the beginning of the episode.
Our experiments of unit type ablations, shown in Figure 12b, indicate that unit variety significantly impacts the difficulty of the tasks. All three races show large differences between the easiest unit distribution and the baseline, suggesting that a diverse range of units contributes to SMACv2's difficulty. In both the Zerg and Protoss scenarios, the melee-only scenarios are easier than the ablation
Table 6: QPLEX hyperparameters used for experiments on SMACv2.
Hyperparameter	Value
Action Selector	epsilon_greedy
Epsilon Start	1.0
Epsilon Finish	0.05
Epsilon Anneal Time	50000
Runner	parallel
Batch Size Run	4
Buffer Size	5000
Batch Size	128
Optimizer	adam
Target Update Interval	200
Agent Output Type	q
Learner	dmaq_qatten_learner
Double Q	True
Mixer	dmaq_qatten
Mixing Embed Dim	32
Hypernet Embed	64
Adv Hypernet Layers	1
Adv Hypernet Embed	64
Td Lambda	0.8
Num Kernel	4
Is Minus One	True
Is Adv Attention	True
Is Stop Gradient	True
N Head	4
Attend Reg Coef	0.001
State Bias	True
Mask Dead	False
Weighted Head	False
Nonlinear	False
Burn In Period	100
Name	qplex_qatten_sc2
Table 7: QMIX hyperparameter tuning grid
Hyperparameter	Values Tried
ε annealing time λ	[100 × 103, 500 × 103] [0.6, 0.8, 0.4]
Table 8: MAPPO hyperparameter tuning grid
Hyperparameter	Value
Actor Learning Rate	[0.0007, 0.0004, 0.0001]
Clip Range	[0.15, 0.05, 0.1]
Table 9: Unit types used per-race in the SMACv2 scenarios.
Race	Unit Types	Probability of Generation
Terran	Marine	0.45
Marauder	0.45
Medivac	0.1
Zerg	Zergling	0.45
Baneling	0.1
Hydralisk	0.45
Protoss	Stalker	0.45
Zealot	0.45
Colossus	0.1
(a) Position ablations
(c) Field-of-view ablations
Figure 12: Ablation of SMACv2 unit type, field-of-view and starting position features. Figure 12d shows the performance of a feed-forward policy compared to an RNN baseline. The unit weights are relative to a fixed order of units. For Zerg this is zergling, hydralisk, baneling, for Terran it is marine, marauder, medivac and for Protoss it is stalkers, zealots and colossi. Plots show mean and standard deviation across three seeds.
(b) Unit type ablations
(d) Feed-forward vs RNN ablation
Figure 13: Comparison of the mean test win rate when  $p = 0, 0.5, 1$  for QMIX (left) and MAPPO (right) on SMACv2. Here, the available action mask is still present.
Figure 14: Comparison of mean test win rate when  $p = 0, 0.5, 1$  with 5 agents against 5 enemy units, for QMIX (left) and MAPPO (right). Here, the available actions mask has been removed.
with ranged units. From observing episodes, the enemy AI tends to aggressively pursue allied units, which allows the allies to lure it out of position. This is easier to exploit with melee units than ranged ones. Terran scenarios have no melee units, but the marine scenarios are slightly easier. Overall these results show that generalising to different unit types is a significant part of the challenge in SMACv2.
Our field-of-view ablations, shown in Figure 12c, compare the fixed unit sight and attack ranges to the SMACv2 versions. There is an increase in difficulty from varying the unit ranges, but this effect is small. This suggests that the difficulty of SMACv2 is better explained by the diversity of start positions and unit types than the change to the true sight and attack ranges.
Finally, to evaluate the effect that the changes had on the partial observability of SMACv2, we compare a feedforward network with the performance of the RNN baseline, shown in Figure 12d. There is a significant decrease in performance from using the feedforward network. However, the feedforward network can still learn reasonable policies. This suggests that although being able to resolve partial observability is useful in SMACv2, the primary difficulty consists in generalising to a range of micro-management scenarios. This is in contrast to the extended partial observability challenge, where resolving partial observability through communication is the primary difficulty. The zerg scenario has a larger relative performance difference. Partial observability may be more of a problem here because the splash damage done to allies by banelings creates an incentive for allies to spread out more.
Overall, the results in Figures 12a and 12b show that current MARL methods struggle with the increased stochasticity due to map randomisation. To solve this problem, agents must be able to use their observations to make inferences about relevant state or joint observation features in the Dec-POMDP in more general settings. Current inference capabilities could be improved by research on more effective sequence models, such as specialised transformers [47, 16, 5]. The work in this area either applies transformer models to traditional RL methods [15], or focuses on paradigms similar to upside-down RL [39, 5]. While the latter work shows promising generalisation [34], it is mostly in the offline setting. Hu et al. [14] demonstrate a promising sequence model that can be applied to any MARL algorithm, but only demonstrate their work on marine units, and rely on the Dec-POMDP having a certain structure. Expanding such work to handle more diverse scenarios is an interesting avenue of future work.
The code for the ablations can be found here. This is distributed under an MIT license. The changes on this branch enable easy running of the ablations and add associated environment configurations for this purpose.
Table 10: SMACv2 Scenarios with their number of allies, number of enemies, and unit types.
Scenario Name	Number of Allies	Number of Enemies	Unit Types
protoss_5_vs_5	5	5	Stalker, Zealot, Colossus
protoss_10_vs_10	10	10	Stalker, Zealot, Colossus
protoss_20_vs_20	20	20	Stalker, Zealot, Colossus
protoss_10_vs_11	10	11	Stalker, Zealot, Colossus
protoss_20_vs_23	20	23	Stalker, Zealot, Colossus
terraran_5_vs_5	5	5	Marine, Marauder, Medivac
terraran_10_vs_10	10	10	Marine, Marauder, Medivac
terraran_20_vs_20	20	20	Marine, Marauder, Medivac
terraran_10_vs_11	10	11	Marine, Marauder, Medivac
terraran_20_vs_23	20	23	Marine, Marauder, Medivac
zerg_5_vs_5	5	5	Zergling, Hydralisk, Baneling
zerg_10_vs_10	10	10	Zergling, Hydralisk, Baneling
zerg_20_vs_20	20	20	Zergling, Hydralisk, Baneling
zerg_10_vs_11	10	11	Zergling, Hydralisk, Baneling
zerg_20_vs_23	20	23	Zergling, Hydralisk, Baneling
F Limitations and Broader Impact
The main limitation of SMACv2 is that it is confined to scenarios within the game of StarCraft II. This is an environment which, while complex, cannot represent the dynamics of all multi-agent tasks. Evaluation of MARL algorithms therefore should not be limited to one benchmark, but should target a variety with a range of tasks.
Whilst the scenarios in SMACv2 involve battles between two armies of units, only one side can be controlled by RL agents. It is technically possible to control two armies using two StarCraft II clients that communicate via LAN, which would allow to train two groups of decentralised agents against each other, e.g., via self-play. We leave the implementation of this functionality as future work.
SMACv2 aims to contribute to the development of MARL algorithms. As with any machine learning field, it is possible that improving the capabilities of these algorithms could lead to unethical uses. However, there are also many potential benefits to better cooperative AI, such as applications in automated driving among others. We believe that the potential benefits of developing more capable and cooperative AI outweigh the potential risks.