SlowGuess/NeurIPS_2023_OCR huggingface dataset:
AlphaStar [Vinyals et al., 2019] became the first StarCraft II AI that achieves the GrandMaster level performance. After that, to enhance the strategy diversity in the league of AlphaStar, TStarBot-X [Han et al., 2020] leverages various human replays to create diverse initialization checkpoints for the exploiters to reset. Handcraft rules were also employed to help the MA explore the strategy space. Another work after AlphaStar is StarCraft Commander (SCC) [Wang et al., 2021], which makes great efforts to optimize the training efficiency by filtering the human data used in imitation learning and compressing the network structure. They branch off new main agents learning specified strategies. SCC displayed comparable performance to professional players in human evaluations, but its APM surpasses humans by a large margin due to the lack of APM limits. Recently, A hierarchical RL method has been proposed to train a StarCraft II AI that can defeat the built-in AI using very few computational resources [Liu et al., 2022b]. To summarize, none of these StarCraft II AIs is able to achieve the level of professional players while adhering to the APM limits and utilizing fewer computational resources than AlphaStar.
After 50 days of league training, ROA-star generates a total of 768 models, of which 221 are MA models. We evaluate the robustness of the 50-day MA model under the condition of no  $z$  through human evaluation. As a comparison, we replicated AlphaStar and trained it for 10 days. We compared the first 10 days of training of ROA-Star and AlphaStar from multiple dimensions. As ROA-Star contains improvement in both exploiters' training method and opponent modeling, we conduct the ablation experiments to demonstrate the effectiveness of each component. Each ablation experiment was trained for 5 days. Notably, all the comparative experiment and ablation experiments use the same computational resources as ROA-Star, as shown in the Appendix A.5.
5.1 The Effectiveness of Goal-conditioned Exploiters in ROA-Star
Figure 2: Comparison of each composition of ROA-Star using Elo score.
Figure 3: Worst-case win rate of the 5-day MA models
Based on the foundation experimental setup of AlphaStar, we modify the original unconditioned exploiters and design the following two experiments to test the effect of goal-conditioned exploiters.
AlphaStar+exploiters with random z: We make exploiters learn  $z$  in  $\mathcal{D}$  with uniform probability. For each reset, the exploiter randomly selects one  $z$  from  $\mathcal{D}$  to learn with an  $80\%$  probability and learns unconditionally with a  $20\%$  probability. When learning a selected strategy, the exploiter would be rewarded by the  $z$ -related pseudo-rewards measuring edit distance between the target and executed build orders.
AlphaStar+EIE+ERE: On the basic setting of AlphaStar, we utilize our goal-conditioned exploiters in Section 3.1, the details of the experimental setup are in the Appendix A.3.
To examine the relative strength of the MA models in different ablation experiments, we consider the rating metric Elo score. Elo scores reflect the relative skill levels of the players based on the win rates between them. We play 100 matches between any two of the first 5-day models of each MA and plot the Elo curves in Figure 2. Besides, we calculate the worst-case win rate of each 5-day MA model defending against all other MA models as a metric of robustness, the result is shown in Figure 3. As we can see, the exploiters that learn random strategies contribute to the robustness of MA, but our goal-conditioned exploiters obviously outperform random selection on  $\mathcal{D}$ .
Additionally, we demonstrate the capability of ERE to learn the precise build orders of target strategies through the use of curricular reward and strategy-guided loss. Our supplementary video and Appendix C.2 showcase the remarkable performance of ERE in imitating non-mainstream strategies.
5.2 The Effectiveness of Opponent Modeling in ROA-Star
Table 2: Win rate against different opponents.
model win rate vs	AlphaStar	+opponent modeling
Void Ray push	54%	60%
Proxy VD	50%	56%
VC Stalker Play	22%	52%
Disruptor push	48%	58%
We conduct an ablation study to validate the effectiveness of opponent modeling.
Table 3: Comparison with AlphaStar
AlphaStar	ROA-Star
worst win rate	1%	44%
avg win rate	70.2%	84.1%
RPP score	0.2157	0.7843
Figure 4: Distribution of tech-building routes against specific opponents. Building acronyms: VR (Robotics Facility), VS (Stargate), BF (Forge), VC (Twilight Council), VD(Dark Shrine).
AlphaStar+opponent modeling: With the aid of the opponent prediction model, we incorporate opponent strategy embedding for the original AlphaStar agents, as well as add scouting rewards. The worst-case robustness is in Figure 3 and the Elo curve is in Figure 2.
To better demonstrate the impact of opponent modeling, we test two 5-day MA models, one trained with opponent modeling and one without, against opponents with specific strategies. We generated these opponents by training MA only on the fixed  $z$  we selected, without using exploiters. Figure 4 displays the distribution of tech-building routes when the models encounter various opponents. Although both models employ a similar dominant strategy, namely VS-VR, the model trained with opponent modeling demonstrates greater flexibility and has prompt responses even on its first tech-building when encountering different opponents, thereby exhibiting superior robustness compared to the original AlphaStar. For example, it increases the probability of VR opening to defend against the Blink Stalkers and build more BF and Photon Cannon to defend the cloaked Dark Templars. The models' win rates against these opponents are shown in Table 2.
In the Appendix C.6, we present visualizations of the latent space of the opponent prediction model, showcasing ROA-Star's "awareness" of the opponent's strategy when facing various opponents. We also compare the time consumed for the agent to discover the newly-built buildings in Appendix C.3. Figure 12 shows that scout reward remarkably reduces the time to discover the opponents' building under the fog.
5.3 Overall Comparison between ROA-Star and AlphaStar
Figure 5: Elo scores of agents in the league of AlphaStar and ROA-star during 10-day training. Curves represent two main agents and points represent the models from exploiters. UE refers to unconditioned exploiters.
Figure 6: Payoff matrix calculated in (a). our ROA-Star and (b). AlphaStar. Blue means a row agent wins and red loses. The exploiters in AlphaStar lose efficacy in the later stage of training.
The primary objective of ROA-Star is the improvement in robustness. However, it's scarcely possible to precisely measure the robustness of the agent due to the enormous strategy space. Similar to the evaluation method in Section 5.1, we count the win rates by a Round Robin tournament on models of two leagues, with each pair of models playing 100 matches. Based on those matches, we introduce quantitative evaluation indexes to score the robustness of the MA and the whole league for AlphaStar and ROA-Star.
We calculate the win rate in the worst case and average case that two 10-day MA models defend against all models in two leagues. Results are shown in Table 3 that ROA-Star significantly outperforms AlphaStar in the MA's robustness. Relative population performance (RPP) is a measure of the expected outcome of the game between two populations. It is calculated based on their meta-game mixture strategies after they have reached Nash equilibrium [Balduzzi et al., 2019]. Given the payoff matrix  $M_{AB} \in [0,1]^{N \times M}$  between all the models in league  $A$  and league  $B$  with two mixture strategies probabilities  $P_A, P_B$  in the Nash equilibrium, the RPP of league  $A$  is:
$$
R P P (A) = P _ {A} ^ {T} M _ {A B} P _ {B}
$$
As shown in Table 3, a higher RPP of ROA-Star demonstrates it constructs a mixture strategy that could counter any mixed strategy from AlphaStar.
As a by-product of the above matches, we evaluate the relative strength of all the players in the league of AlphaStar and our ROA-Star with the Elo rating system. The Elo score curves of the two main agents as well as points indicating the models from exploiters are plotted in Figure 5. As we can see, the strength of MA in ROA-Star is superior to AlphaStar throughout the training procedure, and ROA-Star can always generate more effective opponents for the MA.
Additionally, we evaluate the internal payoff estimation for each of the two leagues, giving the agents' relative performance against all the players in the same league. As shown in Figure 6, the exploiters of AlphaStar seem to have gradually weakened dominance to the MA in the later stage. Meanwhile, there always exist evenly matched opponents for MA in the ROA-Star league, which benefits from its goal-conditioned exploiters.
5.4 Top Human Evaluation of ROA-Star
Figure 7: Trend of ROA-Star's win rate when fighting with professional players. ROA-Star won BO3 and BO5 matches against all the players and maintained final win rates no less than  $50\%$ .
To make a comprehensive evaluation of ROA-Star without access to the official Battle.net, we invite three top professional players: Cyan, Macsed and Jieshi to play against our agent on the Kairos Junction map. Although most of the current AI trained in the full games of StarCraft II are evaluated with human players in best-of-three (BO3) or best-of-five (BO5) matches [Vinyals et al., 2019, Wang et al., 2021], we realize that human has the ability to identify the weaknesses of an opponent in repeated competitions. Therefore, we ask each professional player to play 20 matches with ROA-Star to validate its robustness. We didn't make any demands on the strategies they use except encourage them to try more methods to find the weakness of our agent (Appendix B.1).
Figure 7 shows the trend of ROA-star's win rates as the games proceeds. ROA-star gains the upper hand over all the opponents at the beginning, but its win rate drops as the professional players learn more about it and test more strategies. Finally, as a mixed strategy, ROA-star maintains a win rate of no less than  $50\%$ , which means human players didn't find a winning strategy that can continuously defeat it, indicating the robustness of the agent.
We submit a comparison video with AlphaStar as supplementary material, showing how AlphaStar was defeated by the Cannon Rush strategy in Battle.net evaluation and how we defend against the same strategy played by the professional player through effective scout and prompt response.
In this paper, we develop a new league training method, i.e., ROA-Star, for approximating a NE in very large-scale games such as StarCraft II. ROA-Star improves the influential AlphaStar league training in two significant aspects. ROA-Star trains goal-conditioned exploiters (as opposed to unconditioned exploiters in AlphaStar), which greatly improves the exploiters' ability in identifying weaknesses at both the micro and macro level for the MA and the entire league. In addition, a novel opponent modeling task is introduced in ROA-Star, which helps the agent respond to the opponent's real-time strategy more effectively. Meanwhile, a novel "scouting" reward is constructed based on the prediction of the opponent strategy. Extensive experiments demonstrate the effectiveness of ROA-Star over AlphaStar. Overall, ROA-Star produced a better and superhuman AI with orders of magnitude less resources than AlphaStar on StarCraft II.
There are several future directions of this work. During league training, we kept the parameters of the opponent prediction model fixed to retain the prior knowledge of game strategies in the human data. This approach proved effective in our experiments, yet it is worth investigating the impact of the diversity of strategies in the human data. Also, the policy diversity [Wu et al., 2022, Yao et al., 2023] during league training is worth pursuing as well. In this paper, we only tested ROA-Star on StarCraft II, and future experiments on other large scale games with hidden information are needed to further validate the effectiveness and generalization of ROA-Star. Finally, even though ROA-Star provides a general guideline that goal-conditioned exploiters and opponent modeling help in terms of approximating a NE in very large-scale games, recent research advances on both goal-conditioned RL and opponent modeling in general are definitely worth exploiting to further improve ROA-Star's performance.
We thank all the human players who participated in this work and helped us conduct professional evaluations. We would like to express our gratitude to Tencent AI Arena (https://aiarena.tencent.com/) for providing the powerful computing capability that facilitated the training of ROA-Star.
S. V. Albrecht and P. Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-95, 2018.
M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
D. Balduzzi, M. Garnelo, Y. Bachrach, W. Czarnecki, J. Perolat, M. Jaderberg, and T. Graepel. Open-ended learning in symmetric zero-sum games. In International Conference on Machine Learning, pages 434-443. PMLR, 2019.
E. Chane-Sane, C. Schmid, and I. Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International Conference on Machine Learning, pages 1430-1440. PMLR, 2021.
Y. Du, L. Han, M. Fang, J. Liu, T. Dai, and D. Tao. Liir: Learning individual intrinsic reward in multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. First return, then explore. Nature, 590(7847):580-586, 2021.
L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pages 1407-1416. PMLR, 2018.
C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pages 1515-1528. PMLR, 2018.
A Experiment Setup
The training procedure of ROA-Star includes a supervised learning stage and a 50-day multi-agent reinforcement learning stage. We also implement an experiment with AlphaStar's setup for comparison and trains with the same computation resources. Due to the limitation of budget, the comparison experiment lasts for 10 days which is still a valid baseline as the main agents always get transitive improvement [Vinyals et al., 2019]. Besides, we conduct a few ablation experiments to evaluate the impact of each component in ROA-Star, which are enumerated in Appendix B.2. Each ablation experiment was trained for 5 days. All these experiments are applied in the race Protoss and all the training of reinforcement learning is restricted on the Kairos Junction map. In this section, we introduce the basic settings of ROA-Star.
A.1 Human Dataset
Blizzard is releasing a large number of 1v1 replays played on the ladder. The instructions for how to download the replay files can be found at https://github.com/Blizzard/s2client-proto. We extracted a dataset from these replays which contains 120,938 Protoss vs. Protoss replays from StarCraft II versions 4.8.2 to 4.9.3. These replays were played by human players with MMR scores greater than 4100.
We utilize the dataset of human replays to learn a good initiation checkpoint for reinforcement learning. After 5 days of supervised learning, the model trained on the full game of StarCraft II can defeat the built-in elite AI with a win rate of  $90\%$ . We also train an opponent prediction model on this dataset. The opponent prediction model converges after half-day training, its performance on the test set will be exhibited in Appendix C.4.
Human replays are also used to construct the strategy set  $\mathcal{D}$  for the league training. In order to select a set that can cover the effective strategies, we extract strategy statistic  $z$  from each human replay in the dataset and cluster all  $z$  using the edit distance between their build orders. We sample from each cluster equally to ensure the diversity of selected  $z$ . Finally, we obtain 193 different  $z$  which constitute the strategy set  $\mathcal{D}$ .
A.2 Reinforcement Learning
In the reinforcement learning stage, we reward the agents with the win-loss outcome,  $z$ -related pseudo-rewards and scouting reward. Similar to AlphaStar's configuration, the  $z$ -related pseudo-rewards measure the edit distance between executed and target build orders, as well as the Hamming distance between executed and target cumulative statistics on the units, buildings, and technologies. When agents condition on no extra  $z$ , we disable all  $z$ -related pseudo-rewards. It's worth noting that ERE replace the edit distance reward with the curricular reward on the build orders.
We apply RL techniques similar to those used in AlphaStar. To perform asynchronous and off-policy updates, we use V-trace algorithm [Espeholt et al., 2018], as well as the self-imitation algorithm (UPGO, Oh et al. 2018). We also apply a standard entropy regularization loss and a policy distillation loss distilling from the last reset target, i.e. the historical MA model for EIE, and the supervised model for other agents. We apply an additional strategy-guided loss for ERE to help learn under-explored strategies. The overall loss we used in the reinforcement learning stage is shown below.
$$
\mathcal {L} _ {R L} = \mathcal {L} _ {V - t r a c e} + \mathcal {L} _ {U P G O} + \mathcal {L} _ {e n t r o p y} + \mathcal {L} _ {d i s t i l l} + \mathcal {L} _ {E R E} ^ {*}
$$
A.3 League Setting
ROA-Star consists of four simultaneously training agents in its league: one MA, one ME, and two LEs, where the exploiters are categorized into ME and LE by the different ways of getting opponents. The MA trains with strategy statistic  $z$  sampled from our strategy set  $\mathcal{D}$ , and we set  $z$  to zero  $10\%$  of the time. A frozen copy of MA is added as a new player to the league with a period of every  $2 \times 10^{8}$  steps. The LE fights with the whole league and adds a frozen copy into the league when it defeats all the players in the league with a win rate above  $70\%$  or reaches the timeout threshold of  $2 \times 10^{8}$  steps. At this point, its parameters will be reset with a  $25\%$  probability. ME aims to find the weakness of MA, it adds the frozen copy in the league and reset parameters when defeating MA in more than  $70\%$  of games or after a timeout of  $4 \times 10^{8}$  steps.
So far, the league setting is almost the same with AlphaStar. In ROA-Star, we reform both ME and LE to goal-conditioned exploiters. During the training, each exploiter will be reset to various configurations with a proportion of  $20\%$  origin unconditional exploiter,  $30\%$  EIE, and  $50\%$  ERE. EIE reset to the current MA model. It samples from the  $z$  set with the top  $10\%$  win rate. ERE reset to the supervised model and condition on top  $15\% z$  in execution deviation. All the MA and the exploiters combine the opponent strategy embedding into their observations and get rewards from the scouting behaviors.
A.4 APM Limits
There exists a physical limit for human players on the actions per minute (APM) they can execute. To ensure fairness, we set limitations on the operating frequency for AI that the agent should successively execute actions with a minimum decision interval of 3 frames (around 130 milliseconds). The average APM of any of our final agents' models is less than 240 (with a peak APM below 800), which is close to the human players on Battle.net according to AlphaStar [Vinyals et al., 2019].
A.5 Infrastructure
To scale up league training, we utilize a distributed learner-actor framework depicted in Figure 14. Actors are deployed on CPU machines to interact with the StarCraft II environments, perform action inference and generate training samples. Meanwhile, learners are deployed on GPU machines to update the model parameters using these samples. League Manager is distributed across both the GPU and CPU, maintains the win rates of all historical models, and allocates training tasks to learners and actors, including the agent's own model, the opponent's model, and  $z$  of both sides.
For each agent, the full scale of computational resources contains 64 NVIDIA v100 GPUs and 4600 CPU cores. Each actor worker occupies two CPU cores. About 2400 StarCraft II environments are used simultaneously to provide training samples for an agent. An agent processes about 11000 environment steps per second.
Figure 8: The framework of league training in ROA-Star.
A.6 Summary of ROA-Star's Training Process
Here we present the summary of the 50-day training process of our agents, where the basic settings of each agent can be found in Appendix A.3. With the scale of our computational resources, an agent can process about 11,000 environment steps per second. Our MA was continuously trained for 50 days, which consumed about  $4.42 \times 10^{10}$  steps. During training, MA added a frozen copy to the league every  $2 \times 10^{8}$  steps, resulting in a total of 221 models. Our ME reset its parameters after  $4 \times 10^{8}$  steps at most, which generated 113 models in total. Our LE reset its parameters after  $2 \times 10^{8}$  steps at most, and the two concurrent-training LEs generated 216 and 218 models respectively.
In comparison, AlphaStar's MA was continuously trained for 44 days and consumed around  $1.9 * 10^{11}$  steps (around 50,000 environment steps per second) [Vinyals et al., 2019].
B Evaluation Details
B.1 Human Evaluation
To evaluate the robustness of ROA-Star, we invite three top professional players: Jieshi, Cyan and Macsed to play 20 matches each with ROA-Star. The matches take place on the Kairos Junction map, with both sides using the race Protoss. All of these professional players major in Protoss, and according to Aligulac<sup>6</sup>, their world rankings in Protoss are 19, 25, and 39, respectively. They are champions of many StarCraft II professional competitions, including Dreamhack StarCraft II Masters China and StarCraft II World Championship Series China.
The 20 matches against every professional player were divided into 2 times, with an interval of one week in between. 10 matches were played at a time, with a 3-minute break between matches and a 20-minute break after 5 consecutive matches. Professional players could watch replays against ROA-Star and think about their strategy during each break. The average duration of each game was approximately 10 minutes, with the shortest being 3 minutes and the longest being 18 minutes.
To express our gratitude towards the professional players and motivate them to win, we offered them two options for calculating test fees before the test and allowed them to choose the one that suited them best. The first method is that the test fee for each match is 100 RMB, and the second method is based on the result of each match: professional players receive 150 RMB when they win the game, otherwise, they will only receive 50 RMB. Finally, Cyan chose the first method, while Macsed and Jieshi chose the second method.
In the end, we invited herO, the champion of DreamHack SC2 Masters 2022 Atlanta and the second-ranked professional player in Protoss according to Aligulac<sup>6</sup>, to play two best-of-three (BO3) matches against our agent as the final benchmark. Prior to the competition, we made an agreement that herO would be rewarded with 100 dollars for every BO3 victory, and no payment would be made if he loses. In the end, we won the first BO3 with a score of 2:0, and lost the second BO3 with a score of 1:2, showing ROA-Star is competitive with the best human player in the world.
B.2 Robustness Evaluation between AIs