SlowGuess/NeurIPS_2023_OCR huggingface dataset:
It's hard to directly measure the models' robustness because of the vast space of cyclic, non-transitive strategies in StarCraft II. Instead, we apply the Round Robin tournament on the set of models to be evaluated, with any two models in the set playing 100 matches. Based on the performance of the models in these matches, we conducted robustness evaluations on different models and populations.
We conducted tournaments on two model sets. The first set includes the first 5 days' MA models of all the ablation experiments, including:
- AlphaStar: The original AlphaStar we replicated.
- AlphaStar+exploiters with random  $\mathbf{z}$ : Reform the unconditional exploiters in AlphaStar to learn random strategies sampled from strategy set  $D$ .
- AlphaStar+EIE+ERE: Reform the unconditional exploiters in AlphaStar to EIE and ERE.
- AlphaStar+ERE: Reform the unconditional exploiters in AlphaStar to ERE.
- AlphaStar+opponent modeling: AlphaStar add opponent modeling.
Roa-Star
We demonstrate the robustness evaluations on the first set in Section 5.1, Section 5.2, and Appendix C.1.
The second model set contains all the first 10 days' models in the league of AlphaStar and our ROA-Star, including the models generated by MA and exploiters. The robustness evaluations on the second set are shown in Section 5.3.
$^{6}$ http://aligulac.com/periods/343/?page=1&race=p&nats=all&sort=vp
C Supplementary Experimental Results
C.1 The Ablation Study about EIE and ERE
We conduct an ablation experiment to verify both EIE and ERE could contribute to the robustness of MA.
AlphaStar+ERE: During the training, we reset the exploiters to the configuration of ERE with a probability of  $80\%$ , and to the original unconditional exploiters with a probability of  $20\%$ .
We exhibit the Elo curves of each MA with different exploiters in Figure 9, ERE is superior to random selection in strategy but MA can still benefit from EIE to get further improvement.
Figure 9: Comparison of the different settings of exploiters.
C.2 Learning Process of Explorative Exploiters
Explorative Exploiters are designed to learn the strict build orders in  $z$ , which is especially useful for  $z$  that are currently under-explored. In this section, we select a set of  $z$  with high execution deviation and compare the learning efficiency of exploiters on these  $z$  with various settings. For a specific  $z$ , we exhibit the learning process of each entity in the sequence by showing the increase in their execution precision. The execution precision of the  $n$ -th entity refers to the ratio of successful executions of the  $n$ -th entity after the successful execution of the first  $n - 1$  entities. We represent each entity in a different color in sequence in the following figures in this section.
Take the strategy Proxy Stargate for example, with its build order as "Gateway->Assimulator->Assimator->Gateway->Cyberneticscore->Stargate->Adept->Adept->Gateway->Voidray->Shieldbattery->Shieldbattery->Shieldbattery->Voidray->Voidray->Nexus->Voidray". Figure 10 compare the learning efficiency of the exploiter on this strategy with different learning settings. With the help of z-related curricular reward and strategy-guided loss, the agent learns to execute the strategy defined by  $z$  effectively and accurately.
In Figure11, we provide the learning process of the build orders on another six  $z$  as shown in Table 4 under the setting of Explorative Exploiters. Within  $4 * 10^{8}$  steps, which is consistent with the training steps of ME, all the entities specified by target  $z$  achieve an execution precision above  $50\%$ . In Table 4, we also provide a specific comparison of the last entity execution precision of the AlphaStar  $z$ -related pseudo-rewards setting and the ERE setting.
Figure 10: Comparison of the learning process on an under-explored strategy (Proxy Stargate) with different exploiter settings. The top one is the exploiter with the original z-related pseudo-rewards in AlphaStar. The middle one is the exploiter equipped with the z-related curricular reward. The bottom one is the final Explorative Exploiter with the z-related curricular reward and strategy-guided loss.
Table 4: The execution precision of the last entity in target build order after training of  $4*10^{8}$  steps. AlphaStar refers to training the exploiter with z-related pseudo-rewards as in AlphaStar. ERE refers to the exploiter settings in ERE, including the z-related curricular reward and strategy-guided loss.
index	target build order	last entityexecution precision
AlphaStar	ERE
0	Forge->Assimator->Gateway->Assimator->Gateway->Cyberneticscore->photonicannon->Gateway->Stalker->Stalker->Stargate->Shieldbattery->Shieldbattery->Shieldbattery->phonocannon->Stargate->Voidray->Shieldbattery->Shieldbattery->Voidray	0	0.69
1	Gateway->Assimator->Assimator->Gateway->Cyberneticscore->Roboticsfacility->Stalker->Stalker->Warpprism->Roboticsbay->Stalker->Stalker->Shieldbattery->Nexus->Disruptor->Stalker->Stalker->Shieldbattery->Disruptor->Stalker	0	0.69
2	Gateway->Assimator->Assimator->Gateway->Cyberneticscore->Adept->Adept->Adept->Nexus->Adept->Adept->Shieldbattery->Adept->Adept->Roboticsfacility->Sentry->Sentry->Sentry	0	0.80
3	Gateway->Assimator->Assimator->Gateway->Cyberneticscore->Twilightcouncil->Adept->Adept->Adept->Darkshrine->Gateway->Nexus->Shieldbattery->Darktemplar->Darktemplar->Roboticsfacility->Stalker->Stalker->Stalker	0	0.50
4	Gateway->Assimator->Assimator->Gateway->Cyberneticscore->Adept->Adept->Stargate->Stalker->Stalker->Shieldbattery->Stalker->Oracle->Stalker->Stalker->Shieldbattery	0	0.60
5	Gateway->Assimator->Assimator->Gateway->Cyberneticscore->Adept->Adept->Stargate->Stalker->Oracle->Nexus->Oracle->Roboticsfacility->Twilightcouncil->Stalker->Stalker->immortal->Stalker->Stalker	0	0.67
Figure 11: 6 cases of the Explorative Exploiters' learning process on under-explored strategies.
C.3 The Impact of Opponent Modeling on Scouting Ability
We measure the effectiveness of each scouting behavior with the opponent prediction model and reward the agent accordingly. To show the impact of scouting rewards, we compare the time consumed for the agents to discover the opponent's newly-built buildings. We made two 5-day MA models trained with/without opponent modeling play against each other for 2000 matches. To encounter diverse opponents, both two models randomly pick  $z$  from set  $\mathcal{D}$  to execute. Then we calculate the average interval from the opponent constructing a new building until the agent discovers it. As shown in Figure 12, the scouting reward remarkably reduces the time for discovering the building of opponent under the fog.
Figure 12: The consumed time to discover opponent's newly-built buildings.
C.4 Opponent Prediction Model Performance
In this section, we exhibit the performance of opponent prediction model on the test set of 3000 human replays.
We make quantity predictions for the current opponent's entities, including those in production. We categorize the quantity of each entity type into buckets, where buildings are grouped into 0, 1, 2, and greater than 2 categories, and military units are divided into 0, 1, 2, 3-4, and greater than 4 categories. We use multi-classification heads to predict these categories and calculate the macro f1-score of these tasks on the test set, as shown in Figure 13.
Another critical piece of the opponent's strategy is its technical route. Once the performance of the opponent military unit is observed, it is possible to directly infer whether the opponent has upgraded a certain technology, such as "Charge" which can increase the attack speed of Zealots. However, at this time the optimal timing to respond is often missed. Instead, it is more meaningful to predict the technologies under research. As they are binary classification tasks with extremely imbalanced data, we measure the ability of the opponent prediction model with average precision (AP), as shown in Figure 13. The mean average precision (mAP) of prediction on all technologies is 0.73.
The construction location of key buildings, specifically whether they are constructed outside of the base, determines if the opponent is using proxy strategies. Therefore, we utilize binary classification to predict whether the opponent is constructing or has already constructed proxy buildings. The AP of each type is shown in Figure 13 and the mAP of all proxy building types is 0.75.
Building Quantity Prediction
Unit Quantity Prediction
Technologies Under Research Prediction
Proxy Buildings Prediction
Figure 13: The performance of opponent prediction model on the test set.
C.5 Strategy Diversity in Exploiters
Intuitively, the robustness of MA would benefit from the strategy diversity in exploiters. In this section, we evaluate the strategy diversity in exploiters of two leagues, ROA-Star and AlphaStar, in terms of both qualitative and quantitative measures.
To get a vectorized description of each model generated by the exploiters during the first 10 days, we can have them play 100 matches against a common opponent, such as the 3-day MA model in
AlphaStar. For each model, we calculate the average statistics of entities and technologies on the matches to generate a vectorized description. Then we analyze their strategies by applying k-means clustering to these vectors. The clustering result, shown in Figure 14, displays each model as a point in the 2d space after their vector dimensions have been reduced using t-SNE. The exploiters in ROA-Star can explore more strategies than AlphaStar within the same training time and resources, e.g. the proxy strategies.
We also quantitatively analyze the diversity in exploiters like Determinantal Point Process (DPP, Kulesza and Taskar 2012) dose. DPP measures the diversity of a candidate set by calculating the determinant of a kernel matrix that describes the similarity between each item pair in the candidate set. To get the similarity matrix, we calculate an L2-distance  $d$ , then use the kernel function  $\exp \left( -\frac{d}{T} \right)$  to transform the distance into similarity,  $T$  is a scale factor which we set to 3 here. The final DPP scores of ROA-Star and AlphaStar are shown in Table 5. The numerical comparison of the DPP support that exploiters in ROA-star are more diverse than AlphaStar.
Figure 14: Left visualizes the models in exploiters of AlphaStar and ROA-Star in the 2d space with t-SNE. Right colors the left points with the result of K-means clustering on the models.
Table 5: Diversity in Exploiters
AlphaStar	ROA-Star
DPP score	0.002	0.229
C.6 Latent Space of Opponent Prediction Model
In this section, we visualize the latent space of the opponent prediction model when the agent fights against four different opponents, as shown in Figure 15(b). From top to bottom, we present the distribution of  $h_t$  (the latent opponent strategy embedding) at different game stages: at the 30th second of the game, all the opponents only produce some Probes (the workers) and can't be distinguished. In the 3rd minute, the opponents build the first tech building and choose different tech routes, but they still have similar early military units (like Stalkers), so there are overlaps in the hidden space. In the 6th minute, the opponents produced different units and developed different technologies, resulting in a distinct distribution in the hidden space. As a comparison, Figure 15(a) visualizes the latent space of the policy network in AlphaStar. When facing different opponents, there is no obvious distinguishability in the distribution of hidden states.
The 3rd Minute
The 6th Minute
Figure 15: Let AlphaStar and ROA-Star fight against four specific opponents, each for 150 matches, and visualize (a). the hidden states of the LSTM layer in AlphaStar's policy network (b). the latent opponent strategy embedding  $h_t$  generated by ROA-Star's opponent prediction model. From top to bottom, we collect the corresponding hidden states at different game stages and show their distributions in the 2d space using t-SNE. We color each sub-figure according to the agent's opponents, which includes 150 points for each opponent.
b.
The 30th Second
The 6th Minute