SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ruozi Huang Xipeng Wu Hongsheng Yu Zhong Fan
Haobo Fu* Qiang Fu Wei Yang
Tencent AI Lab, Shenzhen, China {rosiehuang, haroldwu, yannickyu, zhongfan, haobofu, leonfu, willyang}@tencent.com
It is extremely difficult to train a superhuman Artificial Intelligence (AI) for games of similar size to StarCraft II. AlphaStar is the first AI that beat human professionals in the full game of StarCraft II, using a league training framework that is inspired by a game-theoretic approach. In this paper, we improve AlphaStar's league training in two significant aspects. We train goal-conditioned exploiters, whose abilities of spotting weaknesses in the main agent and the entire league are greatly improved compared to the unconditioned exploiters in AlphaStar. In addition, we endow the agents in the league with the new ability of opponent modeling, which makes the agent more responsive to the opponent's real-time strategy. Based on these improvements, we train a better and superhuman AI with orders of magnitude less resources than AlphaStar (see Table 1 for a full comparison). Considering the iconic role of StarCraft II in game AI research, we believe our method and results on StarCraft II provide valuable design principles on how one would utilize the general league training framework for obtaining a least-exploitable strategy in various, large-scale, real-world games.
As one of the most famous real-time strategy games, StarCraft II poses great challenges, in terms of developing a superhuman Artificial Intelligence (AI), to existing Reinforcement Learning (RL) techniques [Vinyals et al., 2017, 2019, Han et al., 2020, Wang et al., 2021], which mainly come in two-folds. The first challenge is the enormous space of strategies. There are approximately  $10^{26}$  possible choices of actions at each time step, and each game normally consists of tens of thousands of time steps. The second challenge is the non-transitive strategy space with hidden information. Non-transitive strategies mean strategy A winning against B and strategy B winning against C does not necessarily mean strategy A winning against C. This implies simple self-play RL algorithms may exhibit strategic cycles Lanctot et al. [2017], Fu et al. [2021], making little progress in terms of the strength of the trained agent. For the hidden information, opponent entities are observable only when they are currently seen by the agent's own entities, and the information of opponent entities is extremely important in StarCraft II: it reveals the opponent strategy. Yet, it is worth-mentioning that the agent is allowed to actively demystify (known as scouting) the hidden information in StarCraft II, which is in contrast to other games with hidden information, such as poker.
As the first AI that defeated top humans in the full game of StarCraft II, AlphaStar [Vinyals et al., 2019] copes with the challenge of enormous strategy space by using human data. The RL agents in
Table 1: A full comparison between AlphaStar and our method ROA-Star. The computational cost is for a single training agent in the league, and the evaluation results are based on Protoss vs. Protoss matches. ROA-Star uses significantly less computational resources than AlphaStar, and the evaluation against top humans is more comprehensive in ROA-Star than AlphaStar.
AlphaStar	ROA-Star
computational cost	TPU or GPU	256 3rd-generation TPU cores	64 NVIDIA V100 GPUs
CPU	4100 preemptible CPU cores	4600 standard CPU cores
actor	16000 concurrent games	2400 concurrent games
learner	50000 steps per second	11000 steps per second
human evaluation	amateur	11 wins and 3 losses	20 wins and 0 losses
professional	5 wins 0 losses (vs TLO2)	3 wins 2 losses (vs herO3)
5 wins 1 losses (vs MaNa3)	10 wins 10 losses (vs Jieshi3)
12 wins 8 losses (vs Cyan3)
10 wins 10 losses (vs MacSed3)
AlphaStar are all initialized by Supervised Learning (SL) on human replays. Also, a KL-divergence (from human replays) regularization is imposed during RL in AlphaStar. To deal with the nontransitive strategy space with hidden information, AlphaStar employs a league training framework that is inspired by game-theoretic approaches [Heinrich et al., 2015, Lanctot et al., 2017] with convergence guarantees to Nash Equilibrium (NE), which is an optimal solution concept AlphaStar aims for. The AlphaStar league consists of four (yet three types) constantly-learning agents: one main agent, one main exploiter, and two league exploiters. The main agent is the one output for evaluation after training. The main exploiter aims to find weaknesses in the main agent, while the league exploiters aim to find weaknesses in the entire league.
Despite achieving a Grandmaster level on Battle.net and defeating several top humans, AlphaStar is computationally prohibitive (see Table 1), and its main agent was later found fragile to some uncommon counter-strategies  $^{4}$ . This implies the inefficiency of the AlphaStar league training framework in terms of approximating a NE strategy in large-scale games such as StarCraft II. In this paper, we improve the AlphaStar league training framework in two substantial ways. We found that (as shown in Figure 6), in the later training iterations of AlphaStar, the exploiters tend to lose the ability to identify the weaknesses in the main agent and the entire league. To alleviate this problem, we train goal-conditioned (as opposed to unconditioned exploiters in AlphaStar) exploiters that exploit the weaknesses in certain directions. To alleviate the problem that AlphaStar does not respond to the opponent real-time strategy effectively, we introduce a novel opponent modeling auxiliary training task, which conveys an explicit training signal for the feature extraction process to focus on the area of observation that reveals the opponent strategy most. In addition, we construct a novel "scouting" reward based on the prediction of the opponent strategy, which greatly encourages the scouting behaviors and thus helps the agent respond to the opponent real-time strategy more effectively. We term our new method as robust and opponent-aware learning training method for StarCraft II (ROA-Star).
We validate our improvements by comparing ROA-Star to AlphaStar. Extensive experiments demonstrate that the exploiters in ROA-Star are more effective in detecting the weaknesses of the main agent and the entire league; that the main agent in ROA-Star responds to the opponent strategy more effectively; and overall that the main agent in ROA-Star is significantly stronger. We also conducted by far the most comprehensive AI vs. top human evaluations in StarCraft II, where our agent trained by ROA-Star achieved a winning rate above  $50\%$  in repeated games. A detailed comparison, in terms of the computational cost and human evaluation, between AlphaStar and ROA-Star is given in Table 1. In light of the significant improvement of ROA-Star over AlphaStar, we believe ROA-Star provides two insightful design principles, i.e., the goal-conditioned exploiters and the opponent modeling auxiliary task, on how one would utilize a league style training framework for obtaining a least-exploitable strategy in various, large-scale, real-world games.
${}^{2}$  TLO is a professional player majored in Zerg not Protoss.
<sup>3</sup>The rankings of herO, MaNa, Jieshi, Cyan and MacSed among all professional players of Protoss are 2, 13, 19, 25 and 39 according to http://aligulac.com/periods/343/?page=1&race=p&nats=all&sort=vp
4In their released human evaluation replays (https://www.nature.com/articles/s41586-019-1724-z#Sec32), AlphaStar was easily defeated by Grandmaster players using some uncommon strategies, such as Cannon Rush.
2 Preliminary on StarCraft II and AlphaStar
2.1 StarCraft II
In StarCraft II [Vinyals et al., 2017], players act as battlefield commanders and start with a few workers who can gather resources. As resources accumulate, players need to allocate them to build buildings, train military units, and research new technologies. StarCraft II offers players the choice of dozens of unit types, each with different spells and abilities. It requires players to construct their armies strategically and assign different real-time tasks to these military units, which can be roughly divided into scout, defense, and attack. When the offensive units of two players meet, each player must quickly direct their units to engage in combat and effectively control them. To win the game, players require to control units at the micro level while making strategic choices at the macro level. Another fundamental setting in StarCraft II is the "fog of war", which limits a player's vision of the map to only those areas that are within the visual range of their entities (buildings and armies).
2.2 Supervised Learning in AlphaStar
Each agent in AlphaStar is firstly trained through supervised learning on human replays. Formally, an agent's policy is denoted by  $\pi (a_{t}|o_{1:t},a_{1:t - 1},z)$ , where  $\pi$  is represented by a deep neural network. At each time step  $t$ , the agent receives a partial observation  $o_t$  of the complete game state  $s_t$  and selects an action  $a_{t}$ .  $o_t$  consists of its own entities, visible enemy entities, and a mini-map depicting the terrain of the battlefield.  $a_{t}$  includes the choice of action type, the action executors, and the targets. The policy conditions on an extra statistic  $z$ , which is a vectorized description of a strategy.  $z$  encodes the first 20 build order (buildings or units) and some cumulative statistics present in a game. In supervised learning, AlphaStar minimizes the KL divergence between human actions and the conditioned policy  $\pi$  on human data.
2.3 RL and League Training in AlphaStar
To address the game-theoretic challenges, AlphaStar proposes a multi-agent RL algorithm, named league training. It assigns the learning agents three distinct types (main agent, main exploiters, and league exploiters), each corresponding to different training and opponent selection mechanisms. All agents are initialized with the parameters of the supervised learning agents and trained with RL. During the league training, these agents periodically add their checkpoints into the league.
The Main Agent (MA) takes the whole league as opponents, with a Prioritized Fictitious Self-Play (PFSP) mechanism that selects the opponent for RL training with probabilities in proportion to the opponent's win rate against MA. Also, the MA is trained by randomly conditioned on either zero or a statistic  $z$  sampled from  $\mathcal{D}$ , where  $\mathcal{D} = \{z_1, z_2, \dots, z_m\}$  is a set of strategy statistics extracted from human data. Similar to goal-conditioned RL [Schaul et al., 2015, Veeriah et al., 2018], the agent, when conditioned on  $z$ , would receive pseudo-rewards that measure the edit distance between the target build orders in  $z$  and the executed ones.
There are two types of exploiters that are designed to identify the weaknesses of their opponents. The Main Exploiter (ME) trains against the MA. The League Exploiter (LE) trains against the whole league. Both exploiters are unconditionally trained. Exploiters add a checkpoint to the league when they achieve a sufficiently high win rate or reach a timeout threshold. At that point, there is a certain probability that exploiters are reset to the supervised agent.
3 ROA-Star
In this section, we present the technique details of ROA-Star. As we discussed before, the purpose of the exploiters in AlphaStar is to find weaknesses of the MA and the entire league. Yet, as we found in the experiments, exploiters in AlphaStar (we re-implement AlphaStar ourselves) gradually lose the ability to counter the MA or the entire league as the training proceeds. This may be because it is increasingly difficult for the exploiters to counter strong agents in later training iterations via exploring the whole strategy space freely (other than initialized from the SL agents). Hence, we propose to alleviate this problem by training exploiters that exploit in certain directions with goal-conditioned RL. In real-time games like StarCraft II, the opponent's strategy, involving both the
b.
Figure 1: a. The design of goal-conditioned exploiters in ROA-Star. Blue rectangles highlight the original unconditioned exploiters in AlphaStar. Small circles are the agent models. Green histograms display the win rate and the execution deviation of  $z$  in MA during a specific time slice in training. b. Opponent Modeling in ROA-Star. Left is the overview of the architecture of the agent in ROA-Star, which combine the embedding of opponent strategy. Right is the overview of the opponent prediction model.
composition of the entity and the technological development, can change rapidly. AlphaStar exhibits a somewhat slow response to these dynamic changes, resulting in the decrease in the playing strength. Our straightforward solution is introducing an opponent modeling training task. Also, based on the opponent modeling prediction, we construct a novel "scouting" reward to encourage efficient scouting behaviors, which helps respond to the opponent real-time strategy more effectively.
3.1 Goal-conditioned RL for the Exploiters
The exploiters (either the main exploiter or the league exploiter) in AlphaStar are trained unconditionally, i.e., the policy  $\pi(a_{t}|o_{1:t},a_{1:t-1})$  depending on only previous observations and actions. In addition to the unconditioned exploiters in AlphaStar, we introduce another two ways of training the exploiters  $\pi(a_{t}|o_{1:t},a_{1:t-1},z)$  that are extra conditioned on the statistic  $z$ . More specifically, we train Exploitative Exploiters (EIE) that are conditioned on those  $z$ , which are associated with high win-rate in the MA. In other words, EIE try to find the weaknesses of either the main or the league based on "good"  $z$  known so far. Also, we train Explorative Exploiters (ERE) that are conditioned on those  $z$ , which are under-explored so far in the MA. In other words, ERE try to find the weaknesses of either the main or the league based on "under-explored"  $z$  so far. Both the EIE or ERE are trained with goal-conditioned RL, the details of which are described below.
The MA in ROA-Star is trained the same way as that in AlphaStar, where the MA is trained either unconditionally or conditioned on a statistic  $z$  randomly selected from the set  $\mathcal{D}$ . In our case, we maintain a moving average of win rate of the MA for each sampled  $z$  during the training process. This gives us an estimation of the "performance" of different  $z$ . Yet, it is worth mentioning that a MA conditioned on a certain  $z$  is not guaranteed to generate plays that are consistent with the corresponding  $z$ . In other words, a  $z$  with bad performance could be due to the fact that the MA is unable to execute the  $z$  successfully. For this reason, we calculate the edit distance between the actual executed  $z$  and the target  $z$  as the execution deviation, and we maintain a moving average of the execution deviation for each  $z$  in  $\mathcal{D}$  as well.
We only sample from those  $z$  with the Top-N win rate of the MA for EIE to learn. Similar to the training of MA in AlphaStar, EIE are rewarded by the  $z$ -related pseudo-reward measuring the edit distance between the target and executed  $z$ . Different from the exploiters in AlphaStar, EIE are always reset to the latest MA parameters whenever a reset occurs. As a result, EIE inherit the strongest strategies in MA. Also, once a  $z$  is sampled, it is fixed till the next reset of EIE, which is in contrast to the MA training setting that samples  $z$  for each game. Instead of evenly distributing computing
resources among all  $z$  with the Top-N win rate, training with a fixed  $z$  for each reset allows EIE to concentrate all its computational resources to refine the micro-level operations on that  $z$  and uncover potential weaknesses at the micro level.
In comparison to EIE, ERE are conditioned on those  $z$  that are currently under-explored according to the moving average execution deviation statistics of MA. We initialize ERE with the SL agents when reset occurs. We notice that the  $z$ -related pseudo-reward measured by the edit distance would be easily hacked [Ecoffet et al., 2021] on those  $z$  with high execution deviation. For instance, the agent tends to produce the common military entities in target  $z$  to gain reward but skip the critical ones. To reduce the execution deviation on these  $z$ , we design a reward that is similar to the trajectory-conditioned learning algorithm [Guo et al., 2020], which formulates a curriculum on how to imitate the target  $z$ . More specifically, ERE are rewarded for each correctly built entity in the target  $z$  till the first mistaken entity happens, where we ignore the left entities in the target  $z$ . For the first mistaken entity (say  $e_{m}$ ) in the target  $z$ , we additionally apply a strategy-guided loss:
$$
\mathcal {L} _ {s g} = - \sum_ {i = 1} ^ {d} e _ {m} ^ {(i)} \log \pi (a _ {t} ^ {(i)} | o _ {1: t}, a _ {1: t - 1}, z),
$$
where  $e_m \in \{0,1\}^d$ ,  $a_t \in \mathbb{R}^d$ .  $d$  is the dimension of action space, which encompasses all the actions related to entity construction. The strategy-guided loss is disabled for production-irrelevant actions, therefore it won't harm the learning of micro-operation.
We use both the curricular reward and strategy-guided loss in ERE training, which is similar to the idea of Go-Explore [Ecoffet et al., 2021]: the agent can always return to the promising state with the curricular reward and then explore the intended direction guided by the strategy-guided loss. Again, we only sample from those  $z$  with the Top-N execution deviation of the MA. Once a  $z$  is sampled, it is fixed till the next reset of ERE. Also, as ERE are conditioned on the  $z$  with high execution deviation, ERE aim to uncover potential weaknesses at the strategy (macro) level. An illustration of our exploiters in comparison to AlphaStar is given in Figure 1(a).
3.2 Opponent Modeling for the Entire League
Knowing the opponent strategy gives the agent a huge advantage in Starcraft II. Yet, with the basic setting "fog of war" in StarCraft II, it is difficult for agents to predict the opponent strategy. Nonetheless, it is possible to infer the opponent strategy to some extent based on the observed opponent entities. More importantly, because of the "scouting" mechanism in StarCraft II, the agent can actively send units to gather more information about the opponent. AlphaStar does not explicitly predict the opponent strategy, and there has been evidence [5] showing that AlphaStar does not respond to the opponent real-time strategy effectively, which greatly affect the agent's performance as demonstrated in our experiments.
In order to improve an agent's ability of responding to the opponent strategy promptly and effectively, we introduce an opponent modeling auxiliary task to infer the opponent strategy. In particular, during the supervised learning of human data, we train a probabilistic encoder to obtain a latent opponent strategy embedding  $h$ , which serves as input to a probabilistic decoder that explicitly predicts the opponent strategy. The encoder and the decoder are trained by maximizing the Evidence Lower Bound (ELBO) used in  $\beta$ -Variational Autoencoders ( $\beta$ -VAE) [Kingma and Welling, 2014, Higgins et al., 2014]. Afterwards, in our league training, the probabilistic encoder is fixed and serves as a special feature extractor that focuses the parts of observation that mostly reflects the opponent strategy.
To construct the input features for the encoder  $q_{\phi}(h_t|o_{\leq t})$  of our opponent modeling task, we filter out opponent irrelevant information (e.g., own entities) in  $o_t$  and focus on visible opponent information, which includes opponent armies, buildings, and technologies. The encoder  $q_{\phi}(h_t|o_{\leq t})$  predicts a Gaussian distribution of  $h_t$  with mean  $\mu_t$  and variance  $\sigma_t$ . A KL divergence term is optimized between the predicted Gaussian and the standard Normal distribution. The decoder  $p_{\theta}(y_t|h_t)$  predicts (a classification problem) the opponent invisible information at each time step, which includes the
<sup>5</sup>From their released human evaluation replays (https://www.nature.com/articles/s41586-019-1724-z#Sec32), it's relatively easy to observe that AlphaStar rarely conducts effective scouting and lacks knowledge of the opponent's real-time strategy, making it fragile to uncommon strategies like the Cannon Rush strategy in Protoss vs. Protoss matches.
quantity of each entity, the technologies being researched, and the location of important tech buildings (e.g., near the player's base or near the opponent). To summarize, we use the reparametrization trick [Kingma and Welling, 2014] to optimize the following loss:
$$
\begin{array}{l} \mathcal {L} _ {o m} \left(y _ {t}, o _ {0}, \dots , o _ {t}; \theta , \phi\right) \\ = - \mathbb {E} _ {q _ {\phi} \left(h _ {t} \mid o _ {\leq t}\right)} \left[ \log p _ {\theta} \left(y _ {t} \mid h _ {t}\right) \right] + \beta K L \left(q _ {\phi} \left(h _ {t} \mid o _ {\leq t}\right) \| \mathcal {N} (0, 1)\right), \\ \end{array}
$$
where in practice the negative log likelihood in the left term is replaced with the focal loss [Lin et al., 2017] in our case to alleviate the class imbalance problem in StarCraft II. An illustration of the opponent prediction model is given in Figure 1(b).
In addition, we construct a novel "scouting" reward based on the change in the cross-entropy of the opponent strategy prediction. Other than predicting the opponent strategy behind the fog, another way to "defog" is to drive military units to periodically scout the area outside the vision. Given that the full map is too vast to scout entirely, it is crucial for the agent to determine when and where to scout. As a result, good "scouting" behavior should improve the prediction accuracy of the opponent strategy. We thus use the opponent prediction model to obtain such a "scouting" reward. At each step of the RL process, we calculate the cross-entropy between the true opponent strategy and the predicted probabilities of the classifiers in the opponent prediction model:
$$
H (t) = - \sum_ {i = 1} ^ {N} y _ {t} ^ {(i)} \log P _ {t} ^ {(i)}
$$
We reward the agent every 30 seconds with the decrease in cross-entropy to encourage scouting behaviors. Note that the agent will not be punished for the increase in cross-entropy, as there is a sustained growth of cross-entropy as the game progresses (prediction in later stage is much harder).
$$
\begin{array}{l} r _ {s c o u t} ^ {t} = \max  (H (t) - H (t ^ {\min}), 0), \\ w h e r e   t ^ {m i n} = \operatorname * {a r g   m i n} _ {t ^ {\prime}} H (t ^ {\prime}),   t ^ {\prime} \in (t, t + 3 0 s). \\ \end{array}
$$
As demonstrated in our experiments, the "scouting" reward greatly encourages the effective "scouting" behavior and improves the overall performance.
There is enormous literature on either goal-conditioned RL [Andrychowicz et al., 2017, Florensa et al., 2018, Ghosh et al., 2019, Chane-Sane et al., 2021, Liu et al., 2022a] or opponent modeling [He et al., 2016, Albrecht and Stone, 2018, Zheng et al., 2018, Raileanu et al., 2018, Willi et al., 2022, Fu et al., 2022]. As in this paper we are focused on how goal-conditioned RL and opponent modeling would improve the league training efficiency of AlphaStar, we will describe related work only in the realm of AI for StarCraft, with more emphasis on literature that comes after AlphaStar.
Before AlphaStar, early research on StarCraft focus on traditional methods, such as planning methods associated with heuristic rules [Weber, 2010, Ontanón et al., 2013, Ontañón et al., 2008, Weber et al., 2011, Ontanon et al., 2013]. Later, there have been RL methods for micro-management in mini-games [Usunier et al., 2016, Vinyals et al., 2017, Du et al., 2019, Han et al., 2019]. Most recently, RL were applied to the full game with the help of some handcraft rules [Lee et al., 2018, Sun et al., 2018, Pang et al., 2019]. However, none of these works achieves a competitive human level in the full game of StarCraft II.