A Robust and Opponent-Aware League Training Method for StarCraft II
Authors: Ruozi Huang
Xipeng Wu
Hongsheng Yu
Zhong Fan
Haobo Fu
QIANG FU
Yang Wei
Venue: NeurIPS 2023 spotlight (2023)
Keywords: StarCraft II
league training
AlphaStar
opponent-modeling
reinforcement learning
TL;DR: A league training method with goal-conditioned exploiters and opponent modeling that aims to improve the robustness of StarCarft II AI.

Abstract: It is extremely difficult to train a superhuman Artificial Intelligence (AI) for games of similar size to StarCraft II. AlphaStar is the first AI that beat human professionals in the full game of StarCraft II, using a league training framework that is inspired by a game-theoretic approach. In this paper, we improve AlphaStar's league training in two significant aspects.  We train goal-conditioned exploiters, whose abilities of spotting weaknesses in the main agent and the entire league are greatly improved compared to the unconditioned exploiters in AlphaStar. In addition, we endow the agents in the league with the new ability of opponent modeling, which makes the agent more responsive to the opponent's real-time strategy. Based on these improvements, we train a better and superhuman AI with orders of magnitude less resources than AlphaStar (see Table 1 for a full comparison). Considering the iconic role of StarCraft II in game AI research, we believe our method and results on StarCraft II provide valuable design principles on how one would utilize the general league training framework for obtaining a least-exploitable strategy in various, large-scale, real-world games.

A Robust and Opponent-Aware League Training Method for StarCraft II
Ruozi Huang
Xipeng Wu
Hongsheng Yu
Zhong Fan
Haobo Fu"
Qiang Fu
Wei Yang
Tencent AI Lab, Shenzhen, China
(rosiehuang, [rg.s zhongfnn beof
Abstract
It is extremely difficult to train superhuman Artificial Intelligence (AI) for games similar size to StarCraft II. AlphaStar is the first AI that beat human professionals in the full game of StarCraft II, using league training framework that is inspired by a game-theoretie approach. In this paper, we improve AlphaStar's 's league training in two significant aspects. We train goal-conditioned exploiters, whose abilities of spotting weaknesses in the main agent and the entire league are greatly improved compared the unconditioned exploiters in AlphaStar. In addition, endow the agents in the league with the new ability of opponent modeling, which makes responsive to the opponent's ' real-time strategy. Based these improvements, we train a better and superhuman AI with orders of magnitude less resources than AlphaStar (see Table for full comparison). Considering the iconic role of StarCraft II in game Al research, we believe our method and results on StarCraft provide valuable design principles on how one would utilize the general league training framework for obtaining a veast-exploitable strategy in various, large-scale, real-world games.
the agent
Introduction
As one of the most famous real-time strategy games, StarCraft II poses great challenges, in terms of developing a superhuman Artificial Intelligence (AI), to existing Reinforcement Learning (RL. techniques (Vinyals et al. 2017 2019 Han et al. 2020, Wang al., 2021]. which mainly two-folds. The first challenge is the enormous space of strategies. There are approximately 1026 possible choices of actions at each time step, and each game normally consists of tens of thousands of time steps. The second challenge is the non non-transitive strategy space with hidden information. Non-transitive strategies mean strategy A winning against B and strategy B winning against C does not necessarily mean strategy A winning against C. This implies simple self-play RL algorithms may exhibit strategic cycles Lanctot al. [2017]. Fu et al. [202]] making little progress in terms of the strength of the trained agent. For the hidden information. opponent entities are observable only when they are currently seen by the agent's own entities, and the information of opponent entities is extremely important in StarCraft II: it reveals the opponent strategy. Yet, it is worth-mentioning that the agent is allowed to actively demystify (known as scouting) the hidden information StarCraft II, which is in contrast to other games with hidden information, such as poker.
in
As the first AI that defeated top humans in the full game of StarCraft II, AlphaStar [Vinyals al. 2019 copes with the challenge of enormous strategy space by using human data. The RL agents in
"Coresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table 1: A full comparison between AlphaStar and our method ROA-Star. The computational cost is for a single training agent in the league, and the evaluation results are based on Protoss vs. Protoss matches. ROA-Star uses significantly less computational resources than AlphaStar, and the evaluation against top humans is more comprehensive in ROA- than AlphaStar.
AlphaStar
ROA-Star
TPU or GPU
256 3rd-generation TPU cores
64 NVIDIA VIOD GPUs
CPU
4100 preemptible CPU cores
4600 standard CPU cores
computational cost
actor
16000 concurrent games
2400 concurrent games
learner
50000 steps per second
11000 steps per second
amaleur
11 wins and 3 losses
20 wins and 0 losses
3 wins 2 losses (vs herO")
human evaluation
wins O losses (vs TLO 5 wins 1 losses (vs MaNa
10 wins losses (vs Jiesh
professional
12 wins 8 losses (vs Cyar
10 wins losses (vs Mac
AlphaStar are all initialized by Supervised Learning (SL) on human replays. Also, KL-divergence (from human replays) regularization is imposed during RL in AlphaStar. To deal with the non- transitive strategy space with hidden information, AlphaStar employs league training framework that is inspired by game-theoretic approaches (Heinrich et al., 2013 Lanctot et al. 2017| with convergence guarantees to Nash Equilibrium (NE), which is an optimal solution concept AlphaStar agents: one main agent, one main exploiter, and two league exploiters. The main agent the one output for evaluation after training. The main exploiter aims to find weaknesses in the main agent, while the league exploiters aim to find weaknesses in the entire league.
aims for. The AlphaStar league consists of four (yet three types) constantly-learnig
Despite achieving Grandmaster level on net and defeating several top humans, AlphaStar computationally prohibitive (see Table 1), and its main agent was later found fragile to some uncommon counter-strategies framework in terms of approximating a NE strategy large-scale games such as StarCraft II. In this paper, we improve the AlphaStar league training framework in two substantial ways. We found that (as shown in Figure
This implies the inefficiency of the AlphaStar league training
the later training iterations of AlphaStar, the exploiters tend to lose the ability to identify the weaknesses in the main agent and the entire league. To alleviate this problem, we train goal-conditioned (as opposed to unconditioned exploiters in AlphaStar) exploiters that exploit the weaknesses in certain directions. To alleviate the problem that AlphaStar does not respond to the opponent real-time strategy effectively, we introduce a novel opponent modeling auxiliary training task, which conveys explicit training signal for the feature extraction process to focus on the area observation that reveals the opponent strategy most. In addition, we construct a novel "scouting" reward based on the prediction of the opponent strategy, which greatly encourages the scouting behaviors and thus helps the agent respond to the opponent real real-time strategy more effectively. We term new method as robust and opponent-awan learning training method for StarCraft II (Rar-Star).
We validate our improvements by comparing ROA-Star to AlphaStar: Extensive experiments demon- strate that the exploiters in ROA-Star are more effective in detecting the weaknesses of the main agent and the entire league; that the main agent in ROA-Star responds to the opponent strategy more effectively; and overall that the main agent in ROa-Star is significantly stronger. We also conducted by far the most comprehensive AI vs. top human evaluations in StarCraft II, where our agent trained by ROA-Star achieved a winning rate above 50% in repeated games. A detailed comparison, in terms of the computational cost and human evaluation, between AlphaStar and ROA-Star is given in Ta In light of the significant improvement of ROA-Star over AlphaStar, we believe ROA-Star provides two insightful design principles, i.c., the con-icondi exploiters and the opponent modeling auxiliary task, on how one would utilize a league style training framework for obtaining a least-exploitable strategy in various, large-scale, real-world games.
'TLO is a professional player majored in Zerg not Protoss.
'The rankings of herO, MaNa, Jieshi, Cyan and MacSed among all professional players of Protoss are 2, 13, 19, 25 and according to http://aligulac httpeeawa.ccccmaeee www.cco////eeee 'In their released human evaluation replays (https:/wwwwnattl)) www.rt....eaommmmmmmmmmmm com/articless/4415866 018866009-17--- (2)322.) AlphaStar was easily defeated by Grandmaster players using some uncommon strategies, such Cannon Rush.

---

2 Preliminary on StarCraft II and AlphaStar
2.1 StarCraft II
In StarCraft (Vinyals et al. 2017]. players act as battlefield commanders and start with a few workers who can gather resources. As resources accumulate, players need to allocate them to build buildings, train military units, and research new technologies. StarCraft II offers players the choice of dozens of unit types, each with different spells and abilities. It requires players to construct their armies strategically and assign different real-time tasks to these military units, which can be roughly divided into scout, defense, and attack. When the offensive units of two players meet, each player must quickly direct their units to engage in combat and effectively control them. To win the game, players require to control units at the micro level while making strategic choices at the macro level. Another fundamental setting in StarCraft II is the "fog of war", which limits a player's 's vision of the map to only those areas that are within the visual range of their entities (buildings and armies).
2.2 Supervised Learning in AlphaStar
Each agent in AlphaStar is firstly trained through supervised learning on human replays. Formally, an agent's policy denoted by π marlo1 41: where represented by a deep neural network. At each time step the agent receives a partial observation or of the complete game state and selects an action dr. Or consists of its own entities, visible enemy entities, and a mini-map depicting the terrain of the battlefield. ae includes the choice of action type, the action executors, and the targets. The policy conditions on an extra statistic z, which is vectorized description of strategy. encodes the first 20 build order (buildings or units) and some cumulative statistics present in game. In supervised learning, AlphaStar minimizes the KL divergence between human actions and the conditioned policy π human data.
2.3 RL and League Training in AlphaStar
To address the game-theoretic challenges, AlphaStar proposes a multi-agent RL algorithm, named league training. It assigns the learning agents three distinct types (main agent, main exploiters, and league exploiters), each corresponding to different training and opponent selection mechanisms. All agents are initialized with the parameters of the supervised learning agents and trained with RL. During the league training, these agents periodically add their checkpoints into the league.
The Main Agent (MA) takes the whole league as opponents, with a Prioritized Fictitious Self-Play (PFSP) mechanism that selects the opponent for RL training with probabilities in proportion to the opponent's win rate against MA. Also, the MA trained by randomly conditioned on either zero or a statistic Z sampled from D, where D {21. 22..... , Zm is set of strategy statistics extracted from human data. Similar to goal-con coal-conditioned RL (Schaul et al. 2015 Veeriah et al. 2018). the agent, when conditioned on z, would receive pscudo-rewards that measure the edit distance between the target build orders in 2 and the executed ones.
There are two types of exploiters that are designed to identify the weaknesses of their opponents. The Main Exploiter (ME) trains against the MA. The League Exploiter (LE) trains against the whole league. Both exploiters are unconditionally trained. Exploiters add a checkpoint to the league when they achieve a sufficiently high win rate or reach a timeout threshold. At that point, there a certain probability that exploiters are reset to the supervised agent.
3 ROA-Star
In this section, we present the technique details of ROA-Star. As we discussed before, the purpose of the exploiters in AlphaStar is to find weaknesses of the MA and the entire league. Yet, as we found in the experiments, exploiters in AlphaStar (we re-implement AlphaStar ourselves) gradually lose the ability to counter the MA or the entire league as the training proceeds. This may be because is increasingly difficult for the exploiters to counter strong agents in later training iterations via exploring the whole strategy space freely (other than initialized from the SL agents). Hence, we propose to alleviate this problem by training exploiters that exploit in certain directions with goal- conditioned RL. In real-time games like StarCraft II, the opponent's 's strategy, involving both the
3

---

Opponent Modeling
Op11
rel reliforcement lcorning
sample 2 Boc. to
hidden
opporier OFC stratady emb
LITM
Cont
Transformer
minimoo
opporient
visbll region
eokorree Pegistooie
Ryplstatio Fobecttin
visible
Main Explotte
Main Agent
League Exploiter (X2)
Berforrmees Leaning
AoroStar's expiote
Figure 1: a. The design of goal-conditioned exploiters in ROA-Sta -Sar. Blue rectangles highlight the original unconditioned exploiters in AlphaStar. Small circles are the agent models. Green histograms display the win rate and the execution deviation of æ in MA during a specific time slice in training. b. Opponent Modeling in ROA-Star. Left is the overview of the architecture of the agent in ROA-Star, which combine the embedding of opponent strategy. Right is the overview of the opponent prediction model.
composition of the entity and the technological development, can change rapidly. AlphaStar exhibits somewhat slow response to these dynamic changes, resulting in the decrease in the playing strength. Our straightforward solution is introducing an opponent modeling training task. Also, based on the opponent modeling prediction, we construct novel "scouting" reward to encourage efficient scouting behaviors, which helps respond to the opponent real-time strategy more effectively.
3.1
Goal-conditioned RL for the Exploiters
The exploiters (either the main exploiter or the league exploiter) in AlphaStar are trained uncondi- tionally, i.e., the policy x <a, 01.t, 01:--1 depending on only previous observations and actions. In addition to the unconditioned exploiters in AlphaStar, we introduce another two ways of training the exploiters (a, (01., d1:t-1. that are extra conditioned on the statistic z. More specifically, we train Exploitative Exploiters (EIE) that are conditioned on those z, which are associated with high win-rate in the MA. In other words, EIE try to find the weaknesses of either the main or the league based on "good" known far. Also, we train Explorative Exploiters (ERE) that are conditioned on those 2, which are under-explored so far in the MA. In other words, ERE try to find the weaknesses of either the main or the league based on "under-explored" z so far. Both the EIE or ERE are trained with goal goal-conditioned RL, the details which are described below.
The MA in ROA-Star is trained the same way as that in AlphaStar, where the MA is trained either unconditionally or conditioned on a statistic randomly selected from the set D. our case, we maintain moving average of win rate of the MA for each sampled during the training process. This gives us an estimation of the "performance" of different z. Yet, it is worth mentioning that MA conditioned on certain is not guaranteed to generate plays that are consistent with the corresponding 2. In other words, a with bad performance could be due to the fact that the MA is unable to execute the z successfully. For this reason, we calculate the edit distance between the actual executed z and the target z as the execution deviation, and we maintain a moving average of the execution deviation for each in D as well.
We only sample from those with the Top-N win rate of the MA for EIE learn. Similar to the training of MA in AlphaStar, EIE are rewarded by the z-related pseudo-reward measuring the edit distance between the target and executed Different from the exploiters AlphaStar, EIE are always reset to the latest MA parameters whenever a reset occurs. As a result, EIE inherit the strongest strategies MA. Also, once a z is sampled, it is fixed till the next reset EIE, which is in contrast to the MA training setting that samples for each game. Instead of evenly distributing computing

---

resources among all 2 with the Top-N win rate, training with a fixed for each reset allows EIE to concentrate all its computational resources to refine the micro-level operations on that 2 and uncover potential weaknesses at the micro level.
In comparison to EIE, ERE are conditioned on those that are currently under-explored according to the moving average exccution deviation statistics of MA. We initialize ERE with the SL agents when reset occurs. We notice that the z-related pseudo-reward measured by the edit distance would be easily hacked [Ecoffet et al. 2021 on those: z with high execution deviation. For instance, the agent tends to produce the common military entities in target to gain reward but skip the critical ones. To reduce the execution deviation on these we design reward that is similar to the trajectory connettoryconddiioonne learning algorithm [Guo al. 2020), which formulates a curriculum on how to imitate the target 2. More specifically. ERE are rewarded for each correctly built entity in the target 2 till the first mistaken entity happens, where we ignore the left entities in the target z. For the first mistaken entity (say 6m) in the target 2, we additionally apply a strategy-guice gugdddedggdd loss:
d
Lag =
Σ 1=1
( m
log (aae 01:t, Q1:t-1,)),
where cm E {0, 11" ar E R d is the dimension of action space, which encompasses all the actions related to entity construction. The strategy -ggdded loss is disabled for production production-irreleannt actions, therefore it won't 't harm the learning of micro-opernee opeertioeer
We use both the curricular reward and Etrategy-guides loss in ERE training, which is similar to the idea of Go-Explore [Ecoffet et al. 202T]: the agent can always return to the promising state with the curricular reward and then explore the intended direction guided by the strategy-guidde loss. Again, we only sample from those Z with the Top-N execution deviation of the MA. Once a is sampled, it is fixed till the next reset of ERE. Also, as ERE are conditioned on the with high execution deviation, ERE aim to uncover potential weaknesses at the strategy (macro) level. An illustration of our exploiters in comparison to AlphaStar is given in Figure I a).
3.2 Opponent Modeling for the Entire League
Knowing the opponent strategy gives the agent a huge advantage in Starcraft II. Yet, with the basic setting "fog of war" in StarCraft II, it is difficult for agents to predict the opponent strategy. Nonetheless, it possible to infer the opponent strategy to some extent based on the observed opponent entities. More importantly, because of the "scouting" mechanism in StarCraft II, the agent can actively send units to gather more information about the opponent. AlphaStar does not explicitly predict the opponent strategy, and there has been evidence I showing that AlphaStar does not respond to the opponent real-time strategy effectively, which greatly affect the agent's performance as demonstrated in our experiments.
In order improve an agent's ability of responding to the opponent strategy promptly and effectively, we introduce an opponent modeling auxiliary task to infer the opponent strategy. In particular, during the supervised learning of human data, we train a probabilistic encoder to obtain a latent opponent strategy embedding h, which serves as input to a probabilistic decoder that explicitly predicts the opponent strategy. The encoder and the decoder are trained by maximizing the Evidence Lower Bound (ELBO) used in B-- V-riiaiiona Autoencoders (5-VAE) (Kingma and Welling [014]] Higgins jet al. 2014]. Afterwards, in our league training, the probabilistic encoder is fixed and serves as special feature extractor that focuses the parts of observation that mostly reflects the opponent strategy.
To construct the input features for the encoder go (heloge)) of our opponent modeling task, we filter out opponent irrelevant information (e.g., own entities) in on and focus on visible opponent information, which includes opponent armies, buildings, and technologies. The encoder (h, loge) predicts a Gaussian distribution of h, with mean μ4 and variance dt. A KL divergence term is optimized between the predicted Gaussian and the standard Normal distribution. The decoder Po(ya (h predicts (a classification problem) the opponent invisible information at each time step. which includes the
$ FFrom their released human evaluation replays https://www.natur/ cow.aaaeecmmmtttee/s/s//// 11811119--1111111 1099-122-24SSoe it's relatively easy to observe that AlphaStar rarely conducts effective scouting and lacks knowledge of the opponent's 's time strategy, making it fragile to uncommon strategies like the Cannon Rush strategy in Protoss vs. Protoss matches.
5

---

quantity of each entity, the technologies being researched. and the location of important tech buildings (e.g., near the player's base or near the opponent). To summarize, we use the reparametrization trick
[Kingma and Welling 2014 to optimize the following loss:
Lom (it, 00, 0,; ()
= (ogg logs) [log (vee /4]] + BK (halog:))lV
where in practice the negative log likelihood in the left term is replaced with the focal loss Lin et al. 2017] in our case to alleviate the class imbalance problem in StarCraft II. An illustration of the opponent prediction model is given in Figure
In addition, we construct a novel "scouting" reward based on the change in the cross-entropy of the opponent strategy prediction. Other than predicting the opponent strategy behind the fog, another way to "defog" is to drive military units to periodically scout the area outside the vision. Given that the full map is too vast to scout entirely. it is crucial for the agent determine when and where to scout. As a result, good "scouting" behavior should improve the prediction accuracy of the opponent strategy. We thus the opponent prediction model to obtain such "scouting" reward. At each step of the RL process, we calculate the cross-entropy between the true opponent strategy and the predicted probabilities of the classifiers in the opponent prediction model:
NV
-tt)-
log (())
We reward the agent every 30 seconds with the decrease in cross-entropy to encourage scouting behaviors. Note that the agent will not be punished for the increase in cross-entropy, as there is sustained growth of cross-entropy as the game progresses (prediction in later stage much harder).
r roout = max H (t) - (t ),0),
where main = arg min (t')), l' E (t, 30s).
As demonstrated in our experiments, the "scouting" reward greatly encourages the effective "scouting" behavior and improves the overall performance.
4 Related Work
There is enormous literature on either goal-conditioned RL [Andrychowiez et al., 2017 Florensa
et al. 2018
Ghosh et al., 2019 Chane-Sane et al. [202 Liu et al. [022a] or opponent modeling He
ett al. 2016
Albrecht and Stone [ et al.) 2018 Raileanu et al. 2018 Willi et al. 2022 Fu
et al. 2022
As in this paper we are focused on how coal-conditioned RL and opponent modeling
would improve the league training efficiency of AlphaStar, we will describe related work only the
realm of AI for StarCraft, with more emphasis on literature that comes after AlphaStar.
Before AlphaStar, early research on StarCraft focus on traditional methods, such as planning methods associated with heuristic rules [Weber] 2010 Ontanón et al. 22013] Ontañón et al. [0008] Weber et al. 2011 Ontanon et al. (2013). Later, there have been RL methods for merroo-mmaaemmentt mini-games [Usunier et al. 2016 Vinyals et al. 2017 Du et al. 2019 Han et al. [019). Most recently, RL were applied to the full game with the help of some handeraft rules [ee al. 2018 Sun al. 2018 Pang et al. 2019 However, none of these works achieves competitive human level in the full game of
StarCraft II.
AlphaStar [Viyaals et al. 2019] became the first StarCraft II AI that achieves the GrandMaster level performance. After that, to enhance the strategy diversity in the league of AlphaStar, TStarBot-X
Han et al. 2020) leverages various human replays to create diverse initialization checkpoints for the exploiters reset. Handcraft rules were also employed to help the MA explore the strategy space. Another work after AlphaStar is StarCraft Commander (SCC) [Wang et al. 2021]. which makes great efforts to optimize the training efficiency by filtering the human data used imitation learning and compressing the network structure. They branch off new main agents learning specified strategies. SCC displayed comparable performance to professional players in human evaluations, but its APM surpasses humans by a large margin due to the lack of APM limits. Recently, A hierarchical RL method has been proposed to train a StarCraft II AT that can defeat the built-in AI using very few computational resources [Liu et al. (2026). To summarize, none of these StarCraft II AIs is able to achieve the level of professional players while adhering to the A PM limits and utilizing fewer computational resources than AlphaStar.

---

Experiments
After 50 days league training, ROA-star generates a total of 768 models, which 221 are MA models. We evaluate the robustness of the 50-day MA model under the condition of no through human evaluation. As comparison, we replicated AlphaStar and trained it for 10 days. We compared the first 10 days of training of ROA-Star and AlphaStar from multiple dimensions. As ROA-Star contains improvement in both exploiters training method and opponent modeling, we conduct the ablation experiments to demonstrate the effectiveness of each component. Each ablation experiment was trained for 5 days. Notably, all the comparative experiment and ablation experiments use the same computational resources -tarStar, as shown in the Appendix A.5
5.1
The Effectiveness of Goal-conditioned Exploiters in ROA-Star
1400
Alrns -ps-speetttddeee
Alpns Stare with rindon
15%
1000
800
810
600
AtphaStar
21%
400
oppernentt modeling
AppaSttrreepllttee with rondom
200
AtphaStar+EEE
50%
0%
10%
30%
40%
Training days
Figure 2: Comparison of each composition of Figure 3: Worst-case win rate of the S-day MA ROA-Star using Elo score.
models
Based on the foundation experimental setup of AlphaStar, modify the original unconditioned exploiters and design the following two experiments to test the effect of goal-conditioned exploiters.
AlphaSttar-exploiters with random z: We make exploiters learn in D with uniform probability. For each reset, the exploiter randomly selects one from D to learn with an 80% probability and learns unconditionally with 20% probability. When learning selected strategy, the exploiter would be rewarded by the z-related pseudo-rewards measuring edit distance between target and executed build orders.
AdphaStar=EIE-ERR On the basic setting of AlphaStar, we utilize our goal-conditioned exploiters in Section 3.1 the details of the experimental setup are in the Appendix A.3
To examine the relative strength of the MA models in different ablation experiments, consider the rating metric Elo score. Elo scores reflect the relative skill levels of the players based on the win rates between them. We play 100 matches between any two of the first 5-day models of each MA and plot the Elo curves in Figure 2 Besides, we calculate the worst-case win rate of each 5-day MA model defending against all other MA models metric of robustness, the result is shown in Figure 3 As we can see, the exploiters that learn random strategies contribute to the robustness of MA, but our goal-conditioned exploiters obviously outperform random selection on D.
Additionally, we demonstrate the capability of ERE to learn the precise build orders of target strategies through the use of curricular reward and grratey--guide loss. Our supplementary video and Appendix showcase the remarkable performance of ERE imitating non- non-mainstream strategies.
5.2 The Effectiveness of Opponent Modeling in ROA-Star
Table 2: Win rate against different opponents.
Table 3: Comparison with AlphaStar
model win rate AlphaStar
+opponent modeling
AlphaStar
ROA-Star
Void Ray push
54%
60 %
Proxy VD
50%
56%
worst win rate
1%
44%
Stalker Play
22%
%
avg win rate
70.2%
84.1 %
Disruptor push
48%
58 %
RPP score
0.2157
0.7843
We conduct an ablation study to validate the effectiveness of opponent modeling.
7

---

AlpheStar
AiphaSter+ogponentt modeling
1,0
1.3
sold Ray push
Wid Ray puch mary VD
Prasy VD
VC Stulkar play
VC Stulter play
Discuptor push
Derapiic pash
0.6
0.5
0.4
0.2
0.2
0.0
50
V5-955
V5-241
ViKeM
Vic-5
155
Vi--VIt
Figure 4: Distribution tech-building routes against specific opponents. Building acronyms: VR (Robotics Facility), VS (Stargate), BF (Forge), VC (Twilight Council), VD(Dark Shrine).
AlphaStartopponent modeling: With the aid of the opponent prediction model, we incorporate opponent strategy embedding for the original AlphaStar agents, as well as add scouting rewards. The worst-case robustness is in Figure and and the Elo curve is in Figure
To better demonstrate the impact of opponent modeling, with opponent modeling and one without, against opponents with specific strategies. We generated these opponents by training MA only on the fixed selected, without using exploiters. Figure 4 displays the distribution of tech-building routes when the models encounter various opponents. Although both models employ similar dominant strategy, namely VS-VR, the model trained with opponent modeling demonstrates greater flexibility and has prompt responses even on its first tech- building when encountering different opponents, thereby exhibiting superior robustness compared to the original AlphaStar. For example, it increases the probability of VR opening to defend against the Blink Stalkers and build more BF and Photon Cannon to defend the cloaked Dark Templars. The models' win rates against these opponents are shown in Table 2
test two 5-day MA models, trained
In the Appendix C.6l we present visualizations of the latent space of the opponent prediction model, showcasing ROA-Star's "awareness" of the opponent's 's strategy when facing various opponents. We also compare the time consumed for the agent to discover the newly-built buildings in Appendix C.3 Figure shows that scout reward remarkably reduces the time to discover the opponents' building under the fog.
5.3 Overall Comparison between ROA-Star and AlphaStar
Vain Agattx
Lague Exploter
Leque Expitter 2
Marn Exp piter
1,750
1250
ELO scoe
3200
Training days
Figure 5: Elo scores of agents in the league of AlphaStar and ROA-star during 10-day training. Curves represent two main agents and points rep- resent the models from exploiters. UE refers to unconditioned exploiters.
Figure 6: Payoff matrix calculated in (a). our ROA-Star and (b). AlphaStar. Blue means a row agent wins and red loses. The exploiters in AlphaStar lose efficacy the later stage of training.
The primary objective ROA-Star is the improvement in robustness. However, it's scarcely possible to precisely measure the robustness of the agent due to the enormous strategy space. Similar to the evaluation method in Section 5.1 we count the win rates by Round Robin tournament on models of two leagues, with each pair of models playing 100 matches. Based on those matches, we introduce quantitative evaluation indexes to score the robustness of the MA and the whole league for AlphaStar and RtA-Star.

---

We calculate the win rate in the worst case and average case that two 10-day MA models defend against all models in two leagues. Results are shown TableB]thal ROA-Star significantly outperforms AlphaStar in the MA's robustness. Relative population performance (RPP) is measure of the expected outcome of the game between two populations. is calculated based on their meta-game mixture strategies after they have reached Nash equilibrium [Balduzzi et al.) 2019). Given the payoff matrix MAB E [0. xM XM between all the models league A and league B with two mixture strategies probabilities PA.PB in the Nash equilibrium, the RPP of league A is:
RPP(A) P MABPB
As shown in Table 3 higher RPP of ROA-Staa demonstrates it constructs a mixture strategy that could counter any mixed strategy from AlphaStar.
As a by-product of the above matches, we evaluate the relative strength of all the players in the league of AlphaStar and our ROA-Star with the Elo rating system. The Elo score curves of the main agents as well as points indicating the models from exploiters are plotted in Figure 5 As we can see, the strength of MA in ROA-Star is superior AlphaStar throughout the training procedure, and ROA-Star can always generate more effective opponents for the MA.
Additionally, evaluate the internal payoff estimation for each of the two leagues, giving the agents' relative performance against all the players the same league. As shown in Figure the exploiters of AlphaStar seem to have gradually weakened dominance to the MA in the later stage. Meanwhile, there always exist evenly matched opponents for MA the ROA-Star league, which benefits from its gond-conditioned exploiters.
5.4 Top Human Evaluation of ROA-Star
1.0
Cyan
0.9
80S
B05
Macsed
Jieshi
average win rate
0.6
0.4
0.3
0.1
0.0
2
12 number games
10
14
35
18
20
Figure 7: Trend of ROA-Star's win rate when fighting with professional players. ROA-Star won BO3 and BO5 matches against all the players and maintained final win rates no less than 50%.
To make a comprehensive evaluation of ROA-Star without access to the official Battle.net, we invite three top professional players: Cyan, Macsed and Jieshi play against our agent on the Kairos Junction map. Although most of the current AI trained in the full games of StarCraft II are evaluated with human players in best-of-three thrreee (BO3) best-of-five (B05) matches (Vinyals et al. 2019 Wang et al. 2021], we realize that human has the ability to identify the weaknesses of an opponent in repeated competitions. Therefore, we ask each professional player to play 20 matches with ROA-Star to validate its robustness. We didn't 't make any demands on the strategies they use except encourage them to try more methods to find the weakness of our agent (Appendix B.II)
Figure 7 shows the trend of ROA-star's win rates as the games proceeds. ROA-star gains the upper hand over all the opponents at the beginning, but its win rate drops the professional players learn more about and test more strategies. Finally, as a mixed strategy, ROA-star maintains a win rate of no less than 50%, which means human players didn't 'n't find winning strategy that can continuously defeat it, indicating the robustness of the agent.
We submit comparison video with AlphaStar as supplementary material, showing how AlphaStar was defeated by the Cannon Rush strategy in Battle.ne net evaluation and how we defend against the same strategy played by the professional player through effective scout and prompt response.

---

6 Conclusions and Limitations
In this paper, we develop new league training method, i.e., ROA-Star, for approximating a NE in very large-scale games such as StarCraft II. ROA-Star improves the influential AlphaStar league training in two significant aspects. ROA-Star trains goal-conditioned exploiters (as opposed to unconditioned exploiters in AlphaStar), which greatly improves the exploiters ability in identifying weaknesses at both the miero and macro level for the MA and the entire league. In addition, a novel opponent modeling task is introduced in ROA-Star, which helps the agent respond to the opponent's s real-time strategy more effectively. Meanwhile, a novel "scouting" reward is constructed based on the prediction of the opponent strategy. Extensive experiments demonstrate the effectiveness of ROA-Star over AlphaStar. Overall, ROA-Star produced a better and superhuman AI with orders of magnitude less resources than AlphaStar on StarCraft II.
There are several future directions of this work. During league training, we kept the parameters of the opponent prediction model fixed to retain the prior knowledge of game strategies in the human data. This approach proved effective in our experiments, yet worth investigating the impact of the diversity of strategies in the human data. Also, the policy diversity (Wu et al. 2022 Yao et al.) 2023] during league training is worth pursuing as well. In this paper, we only tested ROA-Staa Star on StarCraft II. and future experiments on other large scale games with hidden information
needed to further validate the effectiveness and generalization of 40A-Star. Finally, even though ROA-Star provides general guideline that goal-conditioned exploiters and opponent modeling help in terms of approximating a NE in very large-scale games, recent research advances on both goal-conditioned RL and opponent modeling in general are definitely worth exploiting to further improve ROA-Star's performance.
Acknowledgements
We thank all the human players who participated in this work and helped us conduct professional evaluations. We would like express our gratitude to Tencent AI Arena httpss//aiarena.tencettccomm for providing the powerful computing capability that facilitated the training of ROA-Star.
References
S. V. Albrecht and P. Stone, Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-99 95, 2018
M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
D. Balduzzi, M. Gamelo,, Y. Bachrach, W. Czarnecki, J. Perolat, M. Jaderberg, and T. Graepel. Open-ended learning in symmetric zero-sum games. In International Conference on Machine Learning, pages 434-443, PMLR, 2019.
E. Chane-Sane, C. Schmid, and I. Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International Conference Machine Learning, pages 1430-1440. PMLR, 2021.
Y. Du, L. Han, M. Fang, J. Liu, T. Dai, and D. Tao. Liir: Learning individual intrinsic reward in multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. First return, then explore. Nature, 590(7847):580-556 2021.
L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference machine learning, pages 1407-1416. PMLR, 2018.
C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pages 1515-1528. PMLR, 2018.
10

---

H. Fu, W. Liu, S. Wu, Y. Wang. T. Yang. K. Li, J. Xing, B. Li. B. Ma, Q. Fu, et al. Actor-criti policy optimization in large-scale imperfect-inforee informcctinformatiinn game. In International Conference on Learning Representations, 2021.
H. Fu, Tian, H. Yu, W. Liu, S. Wu, J. Xiong, Wen, Li, J. Xing, Q. Fu, al. Greedy when sure and conservative when uncertain about the opponents. In International Conference on Machine Learning, pages 6829-6848. PMLR, 2022.
D. Ghosh, A. Gupta, A. Reddy, J. Fu, C. Devin, B. Eysenbach, and S. Levine. Learning to reach goals via iterated supervised learning. arXiv preprint urXiv:1912.6068. 2019.
Y. Guo, J. Choi, M. Moczulski, S. Feng. S. Bengio, M. Norouzi, and H. Lee. Memory based snlectory-conditiions policies for learning from sparse rewards. Advances in Neural Information Processing Systems, 33:4333 4345, 2020.
L. Han, P. Sun, Y. Du, J. Xiong, Q. Wang, X. Sun, H. Liu, and T. Zhang. Grid-wise control for multi-agent reinforcement learning in video game ai. In International Conference Machine Learning, pages 2576-2583 PMLR, 2019.
L. Han, J. Xiong, Sun, X. Sun, M. Fang, Q. Guo, Chen, T. Shi, H. Yu, X. Wu, al. Tstarbot-x: An open-sourced and comprehensive study for efficient league training in starcraft ii full game. arXiv preprint WrXiv:2011.13720 2020.
H. He, J. Boyd-Graber, K. Kwok, and H. Daumé III. Opponent modeling in deep reinforcement learning. International conference on machine learning, pages 1804-1813, PMLR, 2016.
J. Heinrich, M. Lanctot, and D. Silver. Fictitious self-play in extensive- form games. In International conference on machine learning, pages 805-813. PMLR, 2015.
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2014.
D. P. Kingma and M. Welling. Auto-encoding variational bayes. International Conference on Learning Representations, 2014.
A. Kulesza and B. Taskar. Determinantal point processes for machine learning. Foundations & Trends in Machine Learning, 5(2-3):xvii, 2012.
M. Lanctot, V. Zambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. Pérolat, D. Silver, and T. Graepel. A unified theoreic approach to multiagent reinforcement learning. Advances in neural information processing systems, 30, 2017.
D. Lee, H. Tang, J. O. Zhang, H. Xu, T. Darrell, and P. Abbeel. Modular architecture for starcraft with deep reinforcement learning. In Fourteenth Artificial Intelligence and Interactive Digital Entertainment Conference, 2018.
T.Y.. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988. 2017.
M. Liu, M. Zhu, and W. Zhang. Goal-conditioned reinforcement learning: Problems and solutions. arv preprint arXiv:2201.08299 2022a.
R.-Z. Liu, Z.J. Pang, Z.-Y. Meng. W. Wang. Y. Yu, and T. Lu. On efficient reinforcement learning for full-length game of starcraft ii. Journal of Artificial Intelligence Research, 75:213-260, 2022b.
J. Oh, Y. Guo, S. Singh, and H. Lee. Self- imitation learning. In International Conference on Machine Learning, pages 3878-3888 PMLR, 2018.
S. Ontanon, G. Synnaeve, A. Uriarte, and F. Richoux. A survey of real-time strategy game ai research and competition starcraft. IEEE Transactions on Computational Intelligence and Al in Games, 5(4):293-311, 2013,
11

---

S. Ontanón, G. Synnaeve, A. Uriarte, F. Richoux. D. Churchill, and M. Preuss. A survey of real-time strategy game research and competition in starcraft. IEEE Transactions on Computational Intelligence and Al in games, 5(4):293-311, 2013.
S. Ontañón, K. Mishra, N. Sugandh, and A. Ram. Learning from demonstration and case-based planning for real-time strategy games. Springer Berlin Heidelberg, 2008
Z.-J. Pang, R.-Z. full-length game of starcraft. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 4691-4698, 2019.
Liu, Z.-Y. Meng, Y. Zhang, Y. Yu, and T. Lu. On reinforcement learning for
R. Raileanu, E. Denton, A. Szlam, and R. Fergus. Modeling others using oneself in multi-agent rein- forcement learning. In International conference on machine learning, pages 4257-4266. PMLR, 2018.
T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In International conference on machine learning, pages 1312-1320. PMLR, 2015.
Sun, X. Sun, L. Han, J. Xiong, Q. Wang. B. Li, Y. Zheng, J. Liu, Y. Liu, H. Liu, et al. Tstarbots: De- feating the cheating level builtin ai in starcraft in the full game. arXiv preprint 1rXiv:1809.001 2018.
N. Usunier, G. Synnaeve, Z. Lin, and S. Chintala. Episodic exploration for deep deterministic policies: An application starcraft micromanagement tasks. arXiv preprint wxXiv:1609.0299)) 2016.
V. Veeriah, Oh, and S. Singh. Many-goals reinforcement learning. arXiv preprint arXiv:1806.096000 2018.
O. Vinyals, T. Ewalds, S. Bartunov, P. Georgiev, A. S. Vezhnevets, M. Yeo, A. Makhzani, H. Küttler, Agapiou, J. Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arxi www.17708.0777888 2017.
O. Vinyals, L. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-355 -5,4345,,553530 2019.
X. Wang, J. Song, P. Qi, P. Peng. 7. Tang, W. Zhang, W. Li, X. Pi, J. He, C. Gao, al. Sec: efficient deep reinforcement learning agent mastering the game of starcraft ii. In International Conference on Machine Learning, pages 10905-10915, PMLR, 2021.
B. Weber. Aiide 2010 stareraft competition. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 6, pages xv-xvi, 2010.
B. G. Weber, M. Mateas, and A. Jhala. Building human-level ai for real-time strategy games. Aaai Fall Symposium, 2011.
Willi, A. H. Letcher, J. Treutlein, and Foerster. Cola: consistent learning with opponent-learning awareness. In International Conference Machine Learning, pages 23804 23831. PMLR, 2022.
S. Wu, J. Yao. H. Fu, Y. Tian, C. Qian, Y. Yang. Q. Fu, and Y. Wei. Quality-similar diversity via population based reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
Yao, W. Liu, H. Fu, Y. Yang, S. McAleer, Q. Fu, and W. Yang. Policy space diversity for non-transitive games. arXiv preprint arXiv:2306. 1133366668888 16884, 2023.
Y. Zheng, Z. Meng, J. Hao, Z. Zhang, T. Yang, and C. Fan. A deep bayesian policy reuse approach against non-stationary agents. Advances in neural information processing systems, 31, 2018
12

---

A Experiment Setup
The training procedure of ROA-Star includes supervised learning stage and a 50-day multi-agent reinforcement learning stage. We also implement an experiment with AlphaStar's is setup for compari- son and trains with the same computation resources. Due to the limitation of budget, the comparison experiment lasts for 10 days which is still valid baseline as the main agents always get transitive improvement [Vinyals et al. 2019]. Besides, we conduct a few ablation experiments to evaluate the impact of each component in ROA-Star, which are enumerated in Appendix B.2 Each ablation experiment was trained for 5 days. All these experiments are applied in the race Protoss and all the training of reinforcement learning is restricted on the Kairos Junction map. In this section, introduce the basic settings of ROA-Star.
A.1 Human Dataset
Blizzard is releasing a large number of Ivl replays played on the ladder. The instructions for how to download the replay files can be found at https//gginbbbooom/i/iaaaddddddd a dataset from these replays which contains 120,938 Protoss vs. Protoss replays from StarCraft II versions 4.8.2 to 4.9.3. These replays were played by human players with MMR scores greater than 4100.
We extracted
We utilize the dataset of human replays to learn a good initiation checkpoint for reinforcement learning. After 5 days of supervised learning, the model trained on the full game of StarCraft II can defeat the built-in elite AI with a win rate of 90%. We also train an opponent prediction model on this dataset. The opponent prediction model converges after half-day training, its performance on the test set will be exhibited in Appendix C.4.
Human replays are also used to construct the strategy set 'D for the league training. In order to select set that can cover the effective strategies, in the dataset and cluster all z using the edit distance between their build orders. We sample from each cluster equally ensure the diversity of selected z. Finally, we obtain 193 different which constitute the strategy set D.
extract strategy statistic from each human replay
A.2 Reinforcement Learning
In the reinforcement learning stage, reward the agents with the win-loss outcome, 2 z-related pseudo-rewards and scouting reward. Similar to AlphaStar's configuration, the z-related pseudo- rewards measure the edit distance between executed and target build orders, as well as the Hamming distance between executed and target cumulative statistics on the units, buildings, and technologies. When agents condition on no extra z, we disable all z-related pseudo-rewards, It's worth noting that ERE replace the edit distance reward with the curricular reward on the build orders.
We apply RL techniques similar to those used in AlphaStar. To perform asynchronous and off-policy updates. we use V-trace algorithm [Espeholt et al. 2018). as well as the self-im seff-imitation algorithm (UPGO. (Oh et al. (2018). We also apply standard entropy regularization loss and a policy distillation loss distilling from the last reset target, i.c. the historical MA model for EIE, and the supervised model for other agents. We apply an additional strategy- guided loss for ERE to help learn under-explored strategies. The overall loss we used in the reinforcement learning stage is shown below.
LRI. = LV-trace + LUPGO + Centropy + Ldistill LERR
A.3 League Setting
ROA-Star consists of four simultancously training agents in its league: one MA, one ME, and two LEs, where the exploiters are categorized into ME and LE by the different ways of getting opponents. The MA trains with strategy statistic sampled from our strategy set D, and we set z to zero 10% of the time. A frozen copy of MA is added as new player to the league with a period of every 2 x 10 steps. The LE fights with the whole league and adds frozen copy into the league when it defeats all the players in the league with a win rate above 70% or reaches the timeout threshold of 2 10 steps. At this point, its parameters will be reset with 25% probability. ME aims to find the weakness of MA, it adds the frozen copy in the league and reset parameters when defeating MA in more than 70% of games after timeout of 4 x 10 steps.
13

---

So far, the league setting is almost the same with AlphaStar. In ROA-Star, we reform both ME and LE to goal-conditioned exploiters. During the training. each exploiter will be reset to various configurations with proportion of 20% origin unconditional exploiter, 30% EIE, and 50% ERE. EIE reset the current MA model. It samples from the 2 set with the top 10% win rate. ERE reset the supervised model and condition on top 15% in execution deviation. All the MA and the exploiters combine the opponent strategy embedding into their observations and get rewards from the scouting behaviors.
A.4 APM Limits
There exists a physical limit for human players on the actions per minute (APM) they can execute. To ensure fairness, we set limitations on the operating frequency for AI that the agent should successively execute actions with minimum decision interval of frames (around 130 milliseconds). The average APM of any of our final agents' models is less than 240 (with a peak APM below 800), which is close to the human players on Battle.net according AlphaStar (Vinyals et al. 2019].
A.5 Infrastructure
To scale up league training, we utilize distributed learner-actor framework depicted in Figure 14. Actors are deployed on CPU machines to interact with the StarCraft II environments, perform action inference and generate training samples. Meanwhile, learners are deployed on GPU machines to update the model parameters using these samples. League Manager is distributed across both the GPU and CPU, maintains the win rates of all historical models, and allocates training tasks learners and actors, including the agent's own model, the opponent's 's model, and of both sides.
For each agent, the full scale of computational resources contains 64 NVIDIA v100 GPUs and 4600 CPU cores. Each actor worker occupies two CPU cores. About 2400 StarCraft II environments used simultaneously to provide training samples for an agent. An agent processes about 1 11000 environment steps per second.
League Manager
opporent
lemmn tack
leamen bask
Actor
training samples
worker
adlio observadon
training sampies
Learner
learner id opponent
model
Learner
MainAgent
ExploiterAgent
modal
model
Model Pool
(model, hyper-parametent
Figure 8: The framework of league training in ROA-Star.
A.6 Summary of ROA-Star's Training Process
Here present the summary of the 50-day training process of our agents, where the basic settings of each agent can be found in Appendix AA.3] With the scale of our computational resources, an agent can process about 11,000 environment steps per second. Our MA was continuously trained for 50 days, which consumed about 4.42 * 10 10 steps. During training, MA added a frozen copy to the league every 2 10a steps, resulting in total of 221 models. Our ME reset its parameters after 4 10th steps at most, which generated 113 models in total. Our LE reset its parameters after 2 * 10 steps at most, and the two concurrent-training LES generated 216 and 218 models respectively.
In comparison, AlphaStar's MA was continuously trained for 44 days and consumed around 1.9 10 steps (around 50,000 environment steps per second) (Vinyals et al.) 2019].
14

---

B Evaluation Details
B.1 Human Evaluation
To evaluate the robustness of ROA-Star, we invite three top professional players: Jieshi, Cyan and Maesed to play 20 matches each with ROA-Star. The matches take place on the Kairos Junction map, with both sides using the race Protoss. All of these professional players major in Protoss, and according A their world rankings in Protoss are 19, 25, and 39, respectively. They are champions of many StarCraft II professional competitions, including Dreamhack StarCraft II Masters China and StarCraft II World Championship Series China.
The 20 matches against every professional player were divided into 2 times, with an interval of one week in between. 10 matches were played time, with 3-minute break between matches and a 20-minute break after 5 consecutive matches. Professional players could watch replays against ROA-Star and think about their strategy during each break. The average duration of each game was approximately 10 minutes, with the shortest being 3 minutes and the longest being 18 minutes.
To express our gratitude towards the professional players and motivate them to win, offered them two options for calculating test fees before the test and allowed them to choose the one that suited them best. The first method is that the test fee for each match is 100 RMB, and the second method is based on the result of each match: professional players receive 150 RMB when they win the game, otherwise, they will only receive 50 RMB. Finally, Cyan chose the first method, while Macsed and Jieshi chose the second method.
In the end, invited herO, the champion of DreamHack SC2 Masters 2022 Atlanta and the second- ranked professional player in Protoss according to Aligulac to play two best-of-three (BO3) matches against agent the final benchmark. Prior to the competition, we made an agreement that herO would be rewarded with 100 dollars for every BO3 victory, and no payment would be made he loses. In the end, we the first BO3 with score of 2:0, and lost the second BO3 with score of 1:2, showing ROA-Star is competitive with the best human player in the world.
B.2 Robustness Evaluation between AIs
It's hard to directly measure the models' robustness because of the vast space of cyclic, non-transitive strategies in StarCraft II. Instead, we apply the Round Robin tournament on the set of models to be evaluated, with any two models in the set playing 100 matches. Based on the performance of the models in these matches, we conducted robustness evaluations on different models and populations.
We conducted tournaments on two model sets. The first set includes the first 5 days' MA models of all the ablation experiments, including:
AlphaStar: The original AlphaStar we replicated.
AlphaStar+exploiters with random z: Reform the unconditional exploiters in AlphaStar learn random strategies sampled from strategy D.
AlphaStar=EEE-EREE Reform the unconditional exploiters in AlphaStar to EIE and ERE.
AlphaStar+ERE: Reform the unconditional exploiters in AlphaStar to ERE.
AlphaStar+opponent modeling: AlphaStar add opponent modeling.
Roa-Star
We demonstrate the robustness evaluations the first set in Section 5.1 Section 5.2 and Appendix
The second model set contains all the first 10 days' models in the league of AlphaStar and our ROA-Star, including the models generated by MA and exploiters. The robustness evaluations on the second set are shown in Section 5.3
15

---

C
Supplementary Experimental Results
C.1
The Ablation Study about EIE and ERE
We conduct an ablation experiment to verify both EIE and ERE could contribute to the robustness of MA.
During the training, we reset the exploiters to the configuration of ERE with a probability of 80%, and to the original unconditional exploiters with probability of 20%.
ERR::
exhibit Elo curves each MA with different exploiters in Figure ERE is superior to random selection in strategy but MA can still benefit from EIE to get further improvement.
1200
Elo Score
800
600
400
Alphaštar+explolters with random
200
AlphaStar+ERE
AlphaStar+EIE+ERE
2
Training days
Figure 9: Comparison of the different settings of exploiters.
C.2 Learning Process of Explorative Exploiters
Explorative Exploiters are designed learn the strict build orders in 2, which is especially useful for that are currently under-explored In this section, we select a set of 2 with high execution deviation and compare the learning efficiency exploiters on these with various settings. For specific z, we exhibit the learning process of each entity in the sequence by showing the increase in their execution precision. The execution precision of the n-th entity refers to ratio of successful executions of the n-th entity after the successful execution of the first n 1 entities. We represent each entity in different color in sequence in the following figures this section.
Take the strategy Proxy Stargate for example, with its build order as "Gateway->ssiimiltor wwwp:eeaaaaaaaayyyyeeeeeeeeeoooe www.bbatteeee-Shheeibbbeeee---ee
Voodrry->VVidryy>NNNee >eeis>>ooox-ooorrraa"" Figure IO compare the learning efficiency of the exploiter on this strategy with different learning settings. With the help of z-related curricular reward and strategy-guides loss, the agent learns to execute the strategy defined by effectively and accurately.
In IT. we provide the learning process the build orders on another six = as shown in Table 4 under the setting Explorative Exploiters. Within 4 10 steps, which is consistent with the training steps of ME, all the entities specified by target achieve an execution precision above 50%. In Table 4 we also provide specific comparison of the last entity execution precision of the AlphaStar z z-related pseudo-rewards setting and the ERE setting.
16

---

z-related pseudo-rewards in AlphaStar
1.0
the n-th entily target build order
0,55
training steps
z-related curricular reward
0,55
training
z-related curricular strategy- loss
grecssie
0.41
0.5
24
training stops
Figure 10: Comparison of the learning process on an under-explored strategy (Proxy Stargate) with different exploiter settings. The top one is the exploiter with the original z-related pseudo-rewards in AlphaStar. The middle one the exploiter equipped with the z-related curricular reward. The bottom one is the final Explorative Exploiter with the z-related curricular reward and strategy straegy-guided guided loss.
Table The execution precision
entity in target build order after training * 10 steps. AlphaStar refers to training the exploiter with z-related pseudo-rewards as in AlphaStar, ERE refers to the exploiter settings ERE, including the z-related curricular reward and strategy-guided loss.
last entity
index
target build order
execution precison
AlphaStar ERE
0.69
-Sheedbbatteey--Voddryyyy
0.69
2
0.80
> Wtt.ghaoooonoiiooogoooggogggeeee
3
0.50
4
0.60
5
0.67
Simmotal-SSttkkr-SSallkee

---

strategyO
strategy1
Strategy2
1.8
1.0
1.
0.8
type prectsion
type precision
8.
n-th entity of target order
type
0.4
9.
00.
training
training
training steps
1ed
strategy3
strategy4
strategys
1.0
Ag mundor whlyy
precision
type prectston
precision
0.0
0.0
training steps
training steps
training steps
1:8
Figure 11: cases of the Explorative Exploiters' learning process on under-explored strategies.
C.3 The Impact of Opponent Modeling on Scouting Ability
We measure the effectiveness of each scouting behavior with the opponent prediction model and reward the agent accordingly. To show the impact of scouting rewards, we compare the time consumed for the agents discover the opponent's newly-built buildings. We made two 5-day MA models trained with/without opponent modeling play against each other for 2000 matches. To encounter diverse opponents, both two models randomly pick from set D execute. Then we calculate the average interval from the opponent constructing new building until the agent discovers it. As shown in Figure [2] the scouting reward remarkably reduces the time for discovering the building of opponent under the fog.
300
AlphaStar
AlphaStartopponent modeling
250
Time (seconds)
200
150
100
50
NEXUS
ASSIMILATOR
GATEWAY
CYBERNETICSCORE
FORGE
PHOTONCANNON
ROBOTICSBAY
COBOTISFFCCLTY
STARGATE
TWILINTCCOUNCI
TEMPLERARCHIVE
DARKSHRINE
PYLON
Figure 12: The consumed time to discover opponent's 's newly-built buildings.
18

---

C.4 Opponent Prediction Model Performance
In this section, we exhibit the performance of opponent prediction model on the test set of 3000 human replays.
We make quantity predictions for the current opponent's 's entities, including those in production. We categorize the quantity of each entity type into buckets, where buildings are grouped into 0, 1, 2. and greater than 2 categories, and military units are divided into 0, 1, 2, 3-4, and greater than 4 categories. We use multi-classification heads to predict these categories and calculate the macro fl-score of these tasks on the test set, as shown in Figure 13
Another critical piece of the opponent's ' strategy its technical route. Once the performance of the opponent military unit is observed, it is possible to directly infer whether the opponent has upgraded a certain technology, such as "Charge" which can increase the attack speed Zealots. However, at this time the optimal timing to respond is often missed. Instead, it is more meaningful to predict the technologies under research. As they are binary classification tasks with extremely imbalanced data, measure the ability of the opponent prediction model with average precision (AP), shown in Figure The mean average precision (mAP) of prediction on all technologies is 0.73.
The construction location of key buildings, specifically whether they are constructed outside of the base, determines the opponent is using proxy strategies. Therefore, we utilize binary classification to predict whether the opponent is constructing or has already constructed proxy buildings. The AP of each type is shown in Figure and the mAP all proxy building types is 0.75.
0.
0.8
Macto F1-Score
Macro F1 Sccoree
00
Carrent
TEADE sopeetton
BIILLOSATTE PEON
STALeTS
all of seaamma
cottert gRAcLE
Building Quantity Prediction
Unit Quantity Prediction
3.0
1.0
Average Prectaion
0.6
tverage Precttion
0.8
unocun Coto Ceconnan coseereee MoAnANi aetellae !
lanit sull woli EVE2 2
MNSIPENTT FeEER Crestzt BLss
0.
PriON
Ald
Technolog es Under Research Prediction
Proxy Buildings Prediction
Figure 13: The performance of opponent prediction model on the test set.
C,5 Strategy Diversity in Exploiters
Intuitively, the robustness of MA would benefit from the strategy diversity in exploiters. In this section, we evaluate the strategy diversity in exploiters of two leagues, ROA-Star and AlphaStar, in terms of both qualitative and quantitative measures.
To get a vectorized description of each model generated by the exploiters during the first 10 days, can have them play 100 matches against common opponent, such as the 3-day MA model in
19

---

AlphaStar. For each model, we calculate the average statistics of entities and technologies on the matches to generate vectorized description. Then we analyze their strategies by applying k-means clustering to these vectors. The clustering result, shown in Figure 14] displays each model point in the 2d space after their vector dimensions have been reduced using t-SNE. The exploiters in ROA-Star can explore more strategies than AlphaStar within the same training time and resources, e.g. the proxy strategies.
We also quantitatively analyze the diversity in exploiters like Determinantal Point Process (DPP, Kulesza and Taskar 2012) dose. DPP measures the diversity of candidate set by calculating the determinant of a kernel matrix that describes the similarity between each item pair in the candidate set. To get the similarity matrix, we calculate an 1.2-distance d, then use the kernel function exp( + transform the distance into similarity, 7 is scale factor which we scores of ROA-Star and AlphaStar are shown in Table 5 The numerical comparison of the DPP support that exploiters in ROA-star are more diverse than AlphaStar.
here. The final DPP
Immorial Tush
AiphaStar
Prayy Stargate vo Push
Cannon Buth
Opening
Opening Cerketine
Starke flah
Arthoe Rush
Hoay Bansshiie
arupter Push
Figure 14: Left visualizes the models exploiters of AlphaStar and ROA-Star in the 2d space with t-SNE. Right colors the left points with the result of K-means clustering the models.
Table 5: Diversity in Exploiters
AlphaStar
ROA-Star
DPP score
0.002
0.229
C.6 Latent Space of Opponent Prediction Model
In this section, we visualize the latent space of the opponent prediction model when the agent fights against four different opponents, as shown in Figure [5]bb.. From top bottom, we present the distribution of h4 (the latent opponent strategy embedding) at different game stages: at the 30th second of the game, all the opponents only produce some Probes (the workers) and can't be distinguished. In the 3rd minute, the opponents build the first tech building and choose different tech routes, but they still have similar early military units (like Stalkers), so there are overlaps in the hidden space. In the 6th minute, the opponents produced different units and developed different technologies, resulting distinct distribution in hidden space. As a comparison, Figure 15 a) visualizes the latent space of the policy network in AlphaStar. When facing different opponents, there is no obvious distinguishability in the distribution hidden states.
20

---

a.
b.
The 30th Second
The 30th Second
eerenall Pash
Peay porpet-ddeitte huch
Ineertaa Rua
Proy darggse wor Part Ceetrnreetariee
The 3rd Minute
The 3rd Minute
Caran
Adep Open Dartshiw
Canron hus
Adect Opering certrine
The 6th Minute
The 6th Minute
and Poch Openiig taavitiie
Carioe
innestal Fur
Figure 15: Let AlphaStar and ROA-Star fight against four specific opponents, each for 150 matches, and visualize (a). the hidden states of the LSTM layer in AlphaStar's policy network (b). the latent opponent strategy embedding f4 generated by ROA-Star's opponent prediction model. From top bottom, collect the corresponding hidden states at different game stages and show their distributions in the 2d space using t-SNE. We color each sub-figure according to the agent's opponents, which includes 150 points for each opponent.
21


Reviews:
[{"review_id": "UoqeGRclCe", "reviewer": "Reviewer_sqN1", "summary": "The authors present a modification to the AlphaStar training algorithm that increases robustness and allows for faster responses to opponent behaviour by explicitly encoding the opponent's strategy in the model's representation along with adding scouting as an ancillary objective.", "questions": "Can you provide summary statistics or a full release of the self-play training data? 50 days does not clearly explain how many steps/games were needed to train each model\nDid the authors attempt to examine the opponent model latent space? ", "limitations": "The use of human subjects without IRB is concerning", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The method exceeds the SOTA in starcraft and should generalize to other tasks\nThey present ablation that demonstrate the individual improvements of each contribution\n", "weaknesses": "There are limited implementation details provided and the authors do not provide a code release.\nThe authors describe the model as \"opponent aware\", but this is a hypothesis, they do not investigate if the features they add are adding to \"awareness\" just if they improve the winrate, the models could just be learning a more robust set of strategies.\nAdding scouting as an additional reward objective appears not to have any grounding beyond the observation that the previous models did not have a strong enough bias towards scouting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your review. We respond to your questions in the following. \n\n**Q1**: Did the authors attempt to examine the opponent model latent space?\n\n**A1**: Thank you for your suggestion. To show the agent's 'awareness' of the opponent's strategy, we add an experiment to visualize the latent space of the opponent prediction model when the agent fights against four different opponents, as shown in Figure 1.b in the PDF we submitted. From left to right, we offer the distribution of $h_t$ (the latent opponent strategy embedding) at different game stages: at the 30th second of the game, all the opponents only produce some Probes (the workers) and can’t be distinguished. In the 3rd minute, the opponents build the first tech building and choose different tech routes, but they still have similar early military units (like Stalkers), so there are overlaps in the hidden space. In the 6th minute, the opponents produced different units and developed different technologies, resulting in a distinct distribution in the hidden space. As a comparison, Figure 1.a visualizes the latent space of the policy network in AlphaStar. When facing different opponents, there is no obvious distinguishability in the distribution of hidden states.\n\n**Q2**: Adding scouting as an additional reward objective appears not to have any grounding beyond the observation that the previous models did not have a strong enough bias towards scouting.\n\n**A2**: Scouting is essential in RTS games because of the existence of “fog of war”. A consensus among StarCraft II human players is that scouting is indispensable to detect the opponent's real-time strategy and enable proper strategic responses, which has a large impact on the prob of winning a game. Also, effective scouting is not easy: it is important to decide when and where to scout, which is difficult to write hard rules for. We design an effective and general-purpose scouting reward based on the decrease in cross-entropy between the true opponent strategy and the opponent model's predictions. We believe our scouting reward scheme can be easily applied to similar partially observable games. We will make this clearer in the revision.\n\n**Q3**: Can you provide summary statistics or a full release of the self-play training data? 50 days does not clearly explain how many steps/games were needed to train each model.\n\n**A3**: Yes, here we present the summary of the training process of our agents, where the basic settings of each agent can be found in appendix A.3. With the scale of our computational resources, an agent can process about 11,000 environment steps per second. Our main agent was continuously trained for 50 days, which consumed about $4.42 * 10^{10}$ steps. In comparison, AlphaStar's main agent was continuously trained for 44 days and consumed around $1.9 * 10^{11}$ steps (around 50,000 environment steps per second). During training, our main agent added a frozen copy to the league every $2 * 10^8$ steps, resulting in a total of 221 models. Our ME reset its parameters after $4 * 10^8$ steps at most, which generated 113 models in total. Our LE reset its parameters after $2 * 10^8$ steps at most, and the two concurrent-training LEs generated 216 and 218 models respectively. These details will be added to the revision."}, {"review_id": "2kNuc8l6Mu", "reviewer": "Reviewer_Xpov", "summary": "This paper introduces ROA-Star, an improvement to the AlphaStar training framework for StarCraft II. ROA-Star addresses two identified issues in AlphaStar: diminishing efficiency of exploiters as the training progresses and the Main Agent's inability to adapt to opponent strategies in real time. As a solution to the first problem, it incorporates goal-conditioned reinforcement learning with two types of exploiters - Exploitative Exploiters (EIE) and Explorative Exploiters (ERE). EIE fine-tunes micro-level operations targeting high win-rate strategies, while ERE uncovers macro-level strategy weaknesses. ROA-Star also utilizes opponent modeling through a probabilistic encoder-decoder, and includes a \"scouting\" reward to encourage agents to actively collect information on the opponent. Experiments demonstrate that ROA-Star enhances robustness, adaptability, and performance compared to AlphaStar.", "questions": "Do the authors have any intentions to release the game replays involving matches against human professionals? Additionally, are there plans to make the developed agent accessible to the public? These aspects could offer valuable insights and opportunities for further research and community engagement.", "limitations": "The authors acknowledge that their approach relies on certain StarCraft-specific assumptions. It would be really interesting to see how this method performs in different domains, including simpler environments such as Stratego. This extension could provide a more comprehensive understanding of the approach's applicability and limitations beyond StarCraft.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper proposes a new approach to league-based training in complex imperfect-information zero-sum games. The authors elect to apply their approach to StarCraft II, which is arguably one of the most complex environments in this domain. They offer clear comparisons against the state-of-the-art AlphaStar solution, demonstrating that their method outperforms the baseline across various metrics, including learning speed, computational resources needed, and robustness of the final agent. Compelling evidence of their approach's superiority is provided through comparisons of their agent against AlphaStar and human players in a more challenging setup than what AlphaStar employed, involving multiple games allowing humans to exploit any weaknesses in the agent. This is a particularly strong paper.", "weaknesses": "This paper is a really strong contribution, but it could be further strengthened by refining the writing. Specific examples include:\n\nline 144: The word 'In' is superfluous.\nline 171: 'seen' should be replaced with 'observed'.\nline 260: 'Besides' should be replaced with 'Additionally'.\nline 295: 'ELo' should be corrected to 'Elo'.\nlines 113-114-115 need to be rewritten for clarity and coherence.\n\nAddressing these linguistic issues would enhance the paper's readability and presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your review. We respond to your questions below. \n\n**Q1** (weaknesses): This paper could be further strengthened by refining the writing.\n\n**A1**:  Thanks very much for your suggestion. We will edit the paper according to your advice. \n\n**Q2** (Questions): Do the authors have any intentions to release the game replays involving matches against human professionals? \n\n**A2**: Thank you for your interest. We have originally included all the game replays against human professionals in the supplementary materials under the folder named 'replay'. We will make this clearer in the paper. \n\n**Q3** (Questions): Are there plans to make the developed agent accessible to the public?\n\n**A3**: We are willing to make it public, but unfortunately, it is currently technically infeasible. Blizzard released StarCraft API in https://github.com/Blizzard/s2client-proto with the highest supported game version of 4.10.0, which is the version we trained our agent on. But the current version of its client has been updated to 5.0.11. It requires configuring the player's local StarCraft II client to be able to play with our agent on an earlier version. For the same reason, we cannot release our agent on Blizzard's online matching system Battle.net like AlphaStar did, as the players play on the latest game version.\n\n**Q4** (limitation): It would be really interesting to see how this method performs in different domains, including simpler environments such as Stratego.\n\n**A4**: One direction of our future work is to apply ROA-Star to other large-scale two-player imperfect-information games. Compared to small games, ROA-Star is more likely to offer superior performance  in large games with high complexity. This is because ROA-Star can be viewed as a trade-off between efficiency and theoretical guarantees. For smaller games (compared to StarCraft II), there have been extensive methods with strong theoretical convergence guarantee to Nash Equilibrium, e.g., PSRO, DeepNash, CFR, etc.\n"}, {"review_id": "pcCshK6AAa", "reviewer": "Reviewer_zEct", "summary": "This paper describes an AlphaStar-like approach for training a top-human-level StarCraft 2 agent, with three core additions/changes to the approach that was used to train AlphaStar:\n\n1. Training exploiters in the league conditioned on certain goals: exploitative exploiters which are conditioned on the $z$ statistics (i.e., unit composition strategies) that perform best for the main agent, and explorative exploiters which are conditioned on under-explored $z$ statistics.\n2. An auxiliary loss (used in supervised learning on expert replays phase) to predict what strategy (how many of which unit types) the opponent is using.\n3. Reward shaping (used in reinforcement learning phase) that rewards the agent when it improves its own ability (through scouting) to predict what units the opponent has with the part of the model that was trained on the axuiliary loss mentioned above.\n\nThe contributions are primarily intended to make the overall agent more robust / less susceptible to unexpected strategies.", "questions": "Can you provide any clarification in particular on the point I described above about Table 1?", "limitations": "No comments.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Strong empirical results (including against top human experts)\n- The writing is clear (barring just a few minor parts) and correct", "weaknesses": "### Main comments\n- The paper feels seems to barely fit in a conference submission. Results for several experiments (more than just hyperparameter tuning / extensive ablations) are only very briefly mentioned in the main paper (with results not even summarised, just the existence of the experiment being mentioned), with results+discussion only present in supplementary material.\n- The paper mentions AlphaStar's computational costs being prohibitive, and it is probably true that this paper used a bit less.... but, frankly, the hardware used for this paper is **still** extremely much.\n\n### Detailed comments\n- Table 1 does not mention for how long each agent was trained. This was 50 days for ROA-Star, but (I think?) less for AlphaStar. I'm not sure by how much. I imagine that, even if AlphaStar took much less time, the 256 TPUs would still be more expensive than the 64 GPUs... but anyway, this is important information to include here.\n- Line 58: Last but no least --> Last but not least\n- Lines 114-115: this is a bit vague / overly general. \"does not respond [to] the opponent['s] real-time strategy\" --> what does it mean to \"respond\" to a strategy? Surely the agent plays, it must be responding somehow. What is the difference between \"real-time strategy\" and any other form of \"strategy\"?\n- Line 123: More specially --> More specifically\n- Lines 139, 140, 275: z (without mathmode) --> $z$ \n- Line 144: In instead --> Instead\n- Line 174: \"there has been evidence showing that\" --> this needs a reference\n- Line 288: \"Similarly\" --> similar to what?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your review. We respond to your questions below. \n\n**Q1**: Can you provide any clarification in particular on the point I described above about Table 1?\n\n**A1**: Yes. The agents of AlphaStar were trained for 44 days. With the scale of our computational resources, an agent can process about 11,000 environment steps per second. After 50 days of training, each of our agents (MA, ME, and 2 LEs) consumed around $4.42 * 10^{10}$ steps.  By comparison, each agent of AlphaStar can process about 50,000 environment steps per second and consumed around $1.9 * 10^{11}$ steps during the entire training process. We will make this clearer in the revision.\n\n**Q2**: Writing issues.\n\n**A2**: Thanks very much for your suggestion. We will edit the paper according to your advice. \n\n**Q3**: Lines 114-115: this is a bit vague / overly general. \"does not respond [to] the opponent['s] real-time strategy\" --> what does it mean to \"respond\" to a strategy? Surely the agent plays, it must be responding somehow. What is the difference between \"real-time strategy\" and any other form of \"strategy\"?\n\n**A3**:  Responding to a opponent strategy means (in our statement) adjusting one's strategy (by building tech buildings, training military units, and researching new technologies) in order to gain a strategic advantage. For example, if it is discovered that the opponent has produced stealth units (like Dark Templars), one should produce anti-stealth units (like Observers). Opponent real-time strategy refers to the opponent's current strategy, rather than the opponent's historical strategy in early times. We will make this clearer in the revision.\n\n**Q4**: Line 174: \"there has been evidence showing that\" --> this needs a reference\n\n**A4**: We observed this phenomenon in the human evaluation replays of AlphaStar released by DeepMind ( https://www.nature.com/articles/s41586-019-1724-z#Sec32). It is relatively easy to observe that AlphaStar rarely conducts effective scouting and lacks knowledge of the opponent's real-time strategy, making it fragile to uncommon strategies like the Cannon Rush strategy in PvP matches (as shown in our supplementary video). We will add this reference in the revision.\n\n**Q5**: Line 288: \"Similarly\" --> similar to what?\n\n**A5**: Similar to the evaluation method used in section 5.1. We will make this clearer in the revision.\n\n"}, {"review_id": "0gBw0H6hlc", "reviewer": "Reviewer_eEEJ", "summary": "The paper aims to reduce the computational complexity of league training (the AlphaStar system) for Starcraft II by introducing heuristics. Exploiter agents are used in AlphaStar to target weaknesses in the primary policy (main agent) being trained. The paper proposes to condition the exploiter agents on strategies that are either (1) high win rate or (2) divergent from existing strategies. This provides a more guided set of agents that force the main agent to respond to strong opponents or diverse strategies. Another heuristic aims to improve performance by augmenting the base model with a (frozen) opponent prediction model tasked with predicting opponent state. This is used to design a reward that encourages the model to reduce the entropy in opponent model predictions, which creates \"scouting\" behavior that is beneficial in Starcraft II. Ablations show the components added improve model performance compared to the baseline AlphaStar and a human study against expert players demonstrates (modest) success over a large number of repeated matches.", "questions": "- [Q1] How does ROA-Star scale?\n\t- How much can the core work be parallelized across more GPUs or agents (to reduce training time)?\n\t- Are there any particular features of scaling that would differ to AlphaStar? \n\t- Would ROA-Star benefit more or less from maintaining multiple concurrent exploiters of a given type?\n- [Q2] Please provide pseudo-code for the EIE algorithm and ERE algorithm in the main text.\n\t- The details are a bit murky regarding deviation and how it is used to guide sampling. These are the core new methods being introduced, so they should be very clear from the exposition.\n\t- Particularly given no code will be released, this will substantially improve reproducibility.\n- [Q3] Section 5.2\n\t- Consider expanding this analysis more for readers unfamiliar with Starcraft II. What is Figure 4 showing?\n\t\t- It may help to reorganize Figure 4. Show the two models (AlphaStar with and without opponent modeling) compared for each specific building route. So two colors (with vs without opponent modeling), with a separate box for each strategy transition. \n\t\t- Or consider a summary metric that measures the rate of changes in build order in general.\n\t\t- Or perhaps move this mostly to the appendix. It may be sufficient to provide a simple metric showing greater knowledge of opponent state without the specific strategic changes made.\n\t- What are good summary statistics to show tactical changes due to scouting?\n\t- Overall 5.2 was difficult to follow and it's not immediately obvious that the opponent modeling is changing strategy (correctly) more than the baseline.\n- [Q4] Figure 6 is important!\n\t- This is an important phenomenon to document and emphasize.\n\t- I've not seen the degraded exploiter performance reported elsewhere. Creating a metric to track and benchmark this performance decay would help others evaluate other league training methods and assess potential improvements.\n- [Q5] Figure 7\n\t- Mark the 50% win rate line clearly. This will make it more obvious how well/poorly the agent does.\n\t- Please report the final win rates over 20 matches against each human.\n", "limitations": "The paper does not discuss societal impact of the work. This is not that important given the focus on a game that is well discussed - the societal consequences of this work do not alter the conclusions made about the original AlphaStar work.\n\nLimitations are not addressed with specific details. The paper would benefit from pointing out weaknesses in the method or cases that are not currently addressed by the algorithms. Some example questions: When might the scouting reward harm overall performance? What are the limits of a frozen opponent prediction model? Why is performance so close to 50% against human experts? ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "## originality\n\nModest. Augmenting the league training algorithm with more specific strategy conditioning is a welcome addition to techniques for improving the opponents used in league training in a general fashion. \n\nOpponent modeling is not a new task for game playing, but the specific implementation is new (to my knowledge). A scouting reward is an obvious fit to a game like Starcraft II where opponent knowledge is crucial. I look forward to seeing this more widely tested on other hidden information games where scouting may involve stronger trade-offs, like Honor of Kings (meaning: games where scouting comes at a cost compared to other tasks; in Starcraft II most scouting can be done relatively cheaply and in parallel to other activities).\n\n## quality\n\nGood. The human study is very strong evidence of ecological validity and success. Evaluating against 20 matches is a strong criteria and welcome change compared to the previous best of 3 or 5 scenarios.\n\nThe results showing AlphaStar exploiters lose efficacy later in training (figure 6) are another example of solid evaluations of the core algorithms. The main method ablations all reflect a clear attention to evaluating the new methods.\n\n\n## clarity\n\nModest. The methods and experiments are all explained in detail. Some of the details were difficult to follow and would benefit from alternative explanations (see below). Figures were mostly easy to understand, though some of the text assumes familiarity with details of Starcraft II (like Figure 4 and Table 2) that many readers may not possess.\n\n## significance\n\nModest. General league training improvements are of value to RL algorithms applied to large strategic spaces. This will be of interest to a meaningful subset of the RL community investigating game playing. The improvements in saved computational time are modest, but clearly present.\n\nOpponent modeling and the scouting reward are more specific to Starcraft II but will likely apply to certain other games. Overall the methods are promising as a default alternative to the baseline league training methods with no clear downsides aside some code complexity to implement the strategy tracking statistics and selection methods.", "weaknesses": "The results show modest improvements, but it is not clear how the long-run scaling looks. Does ROA-Star asymptote to the same performance as AlphaStar? Reach higher? Figure 2 suggests a meaningful Elo score improvement by 5 days of training, but it was not clear if AlphaStar would eventually catch up, and if so what that gap looks like.\n\nThe win rates against human experts converge to nearly 50%. Is there evidence that the AlphaStar model trained equivalently would do worse?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your review. We respond to all your questions below, and we are happy to provide further details if there is anything still unclear.\n\n**Weaknesses**: It is not clear how the long-run scaling looks? Is there evidence that the AlphaStar model trained equivalently would do worse?\n\n**Answer**: \nWe reimplemented AlphaStar and trained it for 10 days in total, where the comparisons with ROA-Star are reported in Section 5.3. Due to limited resources, we stopped training AlphaStar after a thorough evaluation. First, there is an obvious gap in Elo scores between the AlphaStar’ MA and ours (Figure 5). We found that AlphaStar's MA is very fragile when facing certain models in the ROA-Star league, resulting in the worst win rate of 1% (Table 3). Also, we observed a decrease in the effectiveness of its exploiters (Figure 6), and its league training gradually degraded to self-play. We believe that with an extended period of training, it’s unlikely for AlphaStar to catch up with ROA-Star.\n\nFor a comparison (see Table 1) between DeepMind's AlphaStar and ROA-Star. Our MA trained for 50 days which consumed about $4.42 * 10^{10}$ steps, and DeepMind's MA trained for 44 days and consumed around $1.9 * 10^{11}$ steps. In repeated games against top humans, our agent maintains a win rate of no less than 50%, which means the top humans didn’t find a winning strategy that can repeatedly defeat our agent. Other than the results shown in Table 1, DeepMind's AlphaStar was easily defeated by Grandmaster players in Battle.net with the Cannon Rush strategy (0 wins and 2 losses, PvP), which is one of the uncommon strategies. \n(DeepMind has released the replays of AlphaStar's matches on Battle.net against human players, here is the link https://www.nature.com/articles/s41586-019-1724-z#Sec32.)\n\n**Q1**: How does ROA-Star scale\n\n**A1.1**:  More GPUs can support us to increase the batch size of agent training, thereby accelerating the training speed. Specifically, we have tried increasing the number of GPUs for training a single agent from 64 to 128. By utilizing data parallelism, we were able to double the batch size. Although there was an increase in communication time between GPUs during gradient backpropagation, the overall efficiency improved by nearly 30%.\n\n**A1.2**: Actors in AlphaStar require both CPUs and TPUs to generate samples, while ROA-Star's actors only require CPUs. ROA-Star can accelerate sample generation speed by increasing the number of CPUs.\n\n**A1.3**: We haven't conducted any relevant experiments, but we can offer some conjectures. Under the framework of ROA-Star, maintaining multiple concurrent exploiters of a given type would find more weaknesses of the main agent and the entire league, which is beneficial to improve the robustness of the main agent.\n\n**Q2**: Please provide pseudo-code for the EIE algorithm and ERE algorithm in the main text.\n\n**A2**: Thank you for the suggestion. We will add the relevant pseudocode. Here, we provide some details on calculating deviation. As we mentioned in Section 2.2, the statistic $z$ includes the first 20 build order (buildings or units) of a game. During training, MA conditions on a certain $z$ sampled from human replays and generate the actual executed build order, which might deviate from the build order in $z$. To measure the deviation in execution, we calculate the edit distance between the actual executed build order and the target build order once the game is over. For each $z$ in $D$, we maintain its moving average of the execution deviation. We will make this clearer in the revision.\n\n \n**Q3**: Section 5.2\n\n**A3**: We have reorganized Figure 4 as you suggested, as shown in Figure 2 in the PDF we submitted. \n\nGiven StarCraft II is a highly complex game, it is hard to define a metric to summarize all reasonable strategy transitions. Therefore, we let the agent play against four specific opponents and show its distribution of the first two tech buildings, which is a coarse representation of a strategy. \n\nFor readers who are unfamiliar with Starcraft II, we have demonstrated that opponent modeling increases the agent's flexibility in responding to different opponents, as well as improving win rates in all situations, as shown in Table 2; For Starcraft II players, it is easy to check that the strategy transitions depicted in Figure 4 are straightforward and reasonable.\n\n**Q4**: Figure 6\n\n**A4**: In the original AlphaStar paper, Extended Data Fig. 8 plots a payoff matrix of league training, which shows a similar phenomenon: the effectiveness of exploiters degrades as training progresses. Although AlphaStar did not further investigate this phenomenon, we believe that improving the effectiveness of the exploiter throughout the entire training process would be beneficial for the league training. \nExcept for the payoff matrix, a moving average of the win rate for each exploiter defeating the MA is a meaningful metric that indicates the effectiveness of the exploiter, and this metric can be easily obtained during the training process. We will make it clearer in the revision.\n\n**Q5**: Figure 7\n\n**A5**: Thank you for your suggestion. We have refined Figure 7 and show the revision in Figure 3 in the PDF we submitted. We will add a table to report the final win rates in the final version.\n\n\n**Limitation**: Limitations are not addressed with specific details.\n\n**Answer**：We are willing to discuss the limitations of the frozen opponent prediction model, and we will incorporate this discussion into the revision.\nThe frozen opponent prediction model preserves the prior knowledge of game strategies that human players consider effective. The diversity of strategies in human data may affect the effectiveness of opponent modeling. In other words, if the agents discover new and reasonable strategies during league training, the effectiveness of opponent modeling might decrease. This is something we need to verify in the future work.\n"}]

Author rebuttal:
We have submitted a PDF that includes three figures. In response to reviewer sqN1's question, we demonstrate the agent's 'awareness' of the opponent's strategy in Figure 1 (in the new pdf). Additionally, based on the suggestions from reviewer eEEJ, we have refined the original Figure 4 and Figure 7, and plotted Figure 2 (in the new pdf) and Figure 3 (in the new pdf).

Meta-review: This paper proposes an improved version of AlphaStar for the challenging StarCraft II. The idea is to 1) train goal-conditioned exploiters as opposed to unconditioned exploiters in AlphaStar, 2) train agents to predict the opponent's strategy as an auxiliary task, and 3) give scouting reward to encourage information gathering. The proposed method not only outperformed their replication of AlphaStar in terms of training efficiency but also achieved > 50% win rates against top professional players. All of the reviewers unanimously agreed that the proposed method is interesting and sensible, and the result against top professional players is quite significant, considering how complex and challenging StarCraft II is. Thus, I think this work deserves to be presented at NeurIPS.
Decision: Accept (spotlight)