Learning Diverse Policies in MOBA Games via Macro-Goals
Authors: Yiming Gao
Bei Shi
Xueying Du
Liang Wang
Guangwei Chen
Zhenjie Lian
Fuhao Qiu
GUOAN HAN
Weixuan Wang
Deheng Ye
QIANG FU
Yang Wei
Lanxiao Huang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep Reinforcement Learning
Diverse Policies
Game Playing
Goal-based Learning

Abstract: Recently, many researchers have made successful progress in building the AI systems for MOBA-game-playing with deep reinforcement learning, such as on Dota 2 and Honor of Kings. Even though these AI systems have achieved or even exceeded human-level performance, they still suffer from the lack of policy diversity. In this paper, we propose a novel Macro-Goals Guided framework, called MGG, to learn diverse policies in MOBA games. MGG abstracts strategies as macro-goals from human demonstrations and trains a Meta-Controller to predict these macro-goals. To enhance policy diversity, MGG samples macro-goals from the Meta-Controller prediction and guides the training process towards these goals. Experimental results on the typical MOBA game Honor of Kings demonstrate that MGG can execute diverse policies in different matches and lineups, and also outperform the state-of-the-art methods over 102 heroes.

Learning Diverse Policies in MOBA Games via Macro-Goals
Yiming Gao'
Bei Shi!
Xueying Du' Liang Wang'
Guangwei Chen'
Zhenjie Lian'
Fuhao Qiu
Guoan Han' Weixuan Wang'
Deheng Ye!
Qiang Fu² Wei Yang' Lanxiao Huang?
Tencent AI Lab, Shenzhen, China
Tencent TiMi LI Studio, Chengdu, China
(yatminggao, beishi, sheri enginewang leolian, frankfhqiu, guoanhan, waihinwang, dericye, leonfu, willyang, jacki waw.iehaangnneennceeeooo com
Abstract
Recently, many researchers have made successful progress in building the Al systems for MOBA-game-playine with deep reinforcement learning, such as on Dota 2 and Honor of Kings. Even though these AI systems have achieved or even exceeded human-level performance, they still suffer from the lack of policy diversity. In this paper, we propose novel Macro-Goal Guided framework, called MGG, to learn diverse policies in MOBA games. MGG abstracts strategies as macro-goals from human demonstrations and trains a Meta-Controller to predict these macro-goals. To enhance policy diversity, MGG samples macro-goals from the Meta-Controller prediction and guides the training process towards these goals. Experimental results on the typical MOBA game Honor of Kings demonstrate that MGG can execute diverse policies in different matches and lineups, and also outperform the state-of-the-arn methods over 102 heroes.
Introduction
As a stepping stone of Artificial General Intelligence (AGI). Game AI has achieved significant progress the past decade, including board games Silver et al. 2016], poker games Brown al. [2019] : Moravčík et al. [2017]. Real-time Strategy (RTS) games Vinyals et al. 2019) etc. Especially, Multi-playe Online Battle Arena (MOBA) games have been regarded as challenging problem in Game AI and attracted massive attention from research communities because of the complicated mechanisms such as imperfect information, large state-action space, and long-term sequential decision making. Recently, OpenAl et al. (2019) develop the AI system Dota 2, named OpenAl Five, which defeats the Dota world champion, i.e., Team OG, within a limited size of hero pool. trains with deep reinforcement learning and self-play. Ye et al. [2020a] build the AI system on another popular MOBA game, Honor of Kings, which also defeats top e-sports players. Besides, compared with OpenAl Five, their AI system enables playing full MOBA games, which extends the hero pool size from 17 to 40 heroes, adds ban/pick capabilities, and addresses the scalability issue.
Even though existing Al systems have achieved or even exceeded human-level performance in MOBA games, they still suffer from the lack of policy diversity. Agents during training are controlled by the same hand-crafted reward signal and then present similar strategies even in different matches, while strategies played by humans usually vary according to situations in the game. One of the disadvantages is that we can counter the single learned policy after several attempts effortlessly. For example, one team defcats OpenAI Five nine times with the same strategy in public test:"] The
https://openaiccom/ppppccssssst
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

primary counter-strateyy is to avoid fights as much as possible and instead push outside lanes to pressure the agents into d
Another disadvantage is that agents with a single policy cannot exploit the strength of heroes and lineups. To interest players, MOBA game designers usually make different skill mechanisms for different heroes, which result in heroes play different roles in the game. Therefore, different roles in lineups have different strategies to maximize team's 's utility. For example, the supporter should keep their allies alive and give them opportunities to earn more gold and experience. The assassin can efficiently jungle neutrals rather than farm lanes. Moreover, different combinations of heroes construct lineups with different characteristics. Some lineups have advantages in pushing with a quick pace, while some lineups are good at the development of carry heroes. The agent should find a way to maximize the picked lineup's 's strength instead of playing with single policy in the game.
Although existing methods can learn diverse policies in other games, it remains to be grand challenge in MOBA games. AlphaStar Vinyals et al. (2199) uses a framework hased on the league to improve the diversity of policies in StarCraft II. The historical models are used construct the league and train exploiters to highlight flaws in the league and main agents, The main agents are forced to discover new strategies by training against the exploiters. They also define a latent variable according to building orders in human data as constraints of self-play. However, the league-based methods cannot be applied to MOBA games directly. First, they consume massive computation resources to construct the league. They use 12 copies of models in which there exist 128 TPU cores. Second, the scenarios between MOBA and RTS games are different. They leverage the nature of StarCraft II and use building orders to represent each strategy, which is unavailable in MOBA games.
In this paper, we propose novel learning paradigm named Macro-Goak Guided (MGG) learning to train diverse policies in MOBA games. Motivated by top e-sports players, we define related game information as macro-goals to depict strategic decisions, which constitute macro-strategies in MOBA games. To explore macro-strategies efficiently, we extract macro-goals from human demonstrations and train a supervised model predict macro-goals according to the given lineups and game information. We derive diverse policies by sampling macro- macro-goals from the model, and incorporating the goals and an intrinsic reward into the original actor- cortic framework to guide the policy learning process. We use the famous mobile 5v5 MOBA game, Honor of Kings" as our testbed, which widely used for MOBA-game Al research Ye et al. 2020c a
Wu 2019); Chen el al. [2 Wei er al. [202T] Experimental results demonstrate that the Al agent we obtained can adopt different policies according to the given lineups and improve the Elo score by modelling policy diversities. Our contribution can be concluded as follows:
To the best of our knowledge, we are the first to investigate the problem of policy diversity in MOBA games. We propose a novel framework that can learn diverse policies to adapt to different matches and lineups.
Based on macro macro-goals, our methods can explore diverse macro-strategies efficiently and meanwhile preserve high-level micro-operatioos. Unlike basue methods, our MGG framework uses the same computation resources as the original actor-critic framework.
We conduct performance testing and diversity analysis on the full hero pool, i.e., 102 heroes. Experimental results show that our agents can outperform existing state-of-the-art MOBA-game Al systems and demonstrate diverse policies.
2 Related Works
2.1 Artificial Intelligence in MOBA Games
The progress of Game AI in MOBA mainly focuses on Dota and Honor of Kings. OpenAl et al. [2019) have successfully defeated top e-sports players on playing Ivl and 5v5 games in Dota 2 within limited heroes. Ye al. (2020bc have proposed a series of works that exceed human- level performance and expanded hero pools in Honor of Kings. Besides, TencentHMS Wu 2019 exploits a hierarchical policy structure control macro-strategy decisions and miero-level executions,
http://wwwwerrltttoo//mm/amtttts https://eewwwiiipeediaagggv///// Kings
2

---

respectively. All the mentioned works except the supervised learning adopt single strategy regardless of the heroes and lineups, which cannot take full advantage of heroes and lineups' characteristics.
2.2 Diverse Policies in Other Games
The league-based methods Vinyals al. 2019 Balduzzi et al. [2019] Jaderberg el al. [001 2019
train multiple agents simu linnously to explore strategy space and require massive computation resources. However, it is impractical to use league-based methods in MOBA games. Because the space of strategies is complex in that there are 213, 610, 453, 056 x C1o) lineups for 40 heroes in MOBA games and each lineup has various strategies. Therefore, is difficult to design artificial signal to represent macro-strategies in MOBA games like Starcraft II (e.g., building order).
The aforementioned Game AIs design methods suffer from the following limitations: 1) the need for data on human behaviors; 2) heavy dependence of designer expert knowledge and substantial labor costs in searching a desirable behavior. Shhn el al. (2020) can automatically generate diverse styles with almost no domain knowledge. However, this method is not suitable for complex games that last up hour long and consist of thousands of actions, such MOBA games and RTS games.
2.3 Imitation Learning
In order to reduce the cost of strategy exploration in enormous lineups, imitate macro-strategies from human demonstrations. Actually some works have trained reward functions from human preferences Nair et al. (2018); [Ibarz et al. [2018): Christiano et al. [2017) or learnt policy directly from limited human data Ho and Ermon| [0166] Different from the above works, we abstract strategy as macro-ge goals in MOBA games. Our method not only imitates macro-strategies, but also leaves the agent to explore the optimal micro-operations (e.g., move, attack) by itself through external rewards.
2.4 Goal-based Learning
: Zhu et al. [202T] are originally proposed to solve the problem of training single goal-conditioned policy to accomplish multiple tasks. Some methods refer to the idea of goal-based RL and divide a task into a series of sub-tasks (i.e.. goal) to accelerate the exploration and training of policy, such automatic goal generation FatiiionnFooeeaaa el al. [2018] and hierarchical RL Kulkarni et al. [2016]: Nachum et al. (2018), etc. However, in complex scenarios such as MOBA RTS games, it is difficult to explore the meaningful goals, whether using GAN or training a high-level policy by RL methods. Therefore, we utilize the top players' demonstrations to pre-train macro-level policy generate reasonable and diverse macro-goals.
Goal-based methods peethod|Schaull [Schaul et al. (2015) Andrychowiez et al. [017]:
3 Macro-Goals Guided Reinforcement Learning
The problem we focused on can be formulated as Markov Decision Process (MDP), defined by a tuple (S, G, A, Rt T, 7), where S is the state space, A is the action space, T S x A + S is the transition function, R In a typical MOBA game Silva and Chaimowicz (2017), the agents perform actions a E A (e.g. move, attack) according to the observations s E S (e.g. unit states, in in-game stats) to achieve the final objective (e.g. destroy the crystal). In our setting, G represents the macro-strategic state space in the game. And we define a macro-state extraction function f: f S - G which maps the current state to the multi-dimensional macro-state
A + R is the reward function, and 7 E [0, 1] the discount factor.
(()). The macro-state E G mainly includes the region state (c.g. the area on the map) and resource state (c.g. unit resource, gold, level). The macro- macro-goal g e G represents the macro-state that the agent should achieve in the near future in certain strategy.
Ye et al. [020a] design the general reward function for MOBA games, including farming, KDA, damage, pushing, and win/lose related rewards. As result of the single value system, the agent only learns execute single strategy from self-play training, that is, farming first, then fighting and pushing, no matter what strategy the opponents adopt. Thus, its macro-state set {clc E GRUY is only a part of G, i.e. GRI. C G. On the contrary, human strategies are diverse and their macro-state set Ghuman can be considered to be close to the oracle G, i.e. Ghuman = G U... G.... UN R G, where N is the number of human strategies. We define that strategy i can be expressed as a set of macro-goal {g G. where G2 G. Our objective is to learn diverse strategies, that is, maximize
3

---

the macro-state entropy EesG P(c)logP(c). Therefore, we model the policy as { g) and G the macro-goal of the policy. We also introduce an intrinsic reward function Rgool x R to guide its macro-state set Gr. to be close to the complete G.
utilize g Ghaman
Stage I: Macro-goal Extraction
Stage 2: Supervised Learning
Stage Reinforcement I Learning
Gatel
Game states
parameets copy pretrained parameter
R nooal
Italenol
sample
ponl (5.g)
Human Demonstration
st
St
St
Vental
Taux
0
veri
Meta-Contnolir
Meta Controller
Rr.
na(alss
Macro-goal
Poliag-Vahae Network
Totcet
Human Macro-strateg'
Figure 1: The MGG framework.
In this paper. we propose the Macro-Goals Guided (MGG) framework to learn diverse policies in MOBA games. Intuitively, top players' matches are observed to have high-quality and diverse macro-strategies but noisy micro-operations. Thus, completely imitating human demonstrations, including micro-operations, will worsen the performance of the policy. However, the macro- strateiss from top players' matches can be used to imitate, which can reduce the cost of exploration in training. Our motivation is to define macro-goals to depict human strategies and then learn diverse strategies from these macro-goals only. In general, the MGG consists of three stages, as shown Figure
Stage I: Extract state sf and its corresponding macro- goal gd from top e-sports demonstrations.
Stage 2: Use the extracted state and macro-goal pairs (sP..) as the training data to train the Meta-Controller parameterized by Φ, via supervised learning. During the training process, we use the auxiliary auto-encoding tasks Taur of lineups improve prediction accuracy. Given state s, the Meta-Controller & should predict a macro-goal to depict the current strategy.
Stage 3: Use the Dual-clip PPO algorithm Ye et al. [2020a] to train the policy and value network A conditioned on the macro-goal g. The macro-goals, g of the specified strategy sampled from the pre-trained Meta-Cose Mett--oottolllr. We introduce an intrinsic reward Ri to denote whether the agent achieves the macro-goal g. Besides, we use the multi-head value mechanism Ye et al. [020a] to combine the original value output and the intrinsic value output.
To better illustrate the MGG method, we use Honor of Kings as a case study to describe some common terms associated with MOBA games. But it can be applied to other MOBA games (e.g. Dota 2 and League of Legends), the playing mechanism across MOBA games are similar.
3.1 Generation of Macro-Goals
Region Intent
Resource Intent
Top Lane
Base Area
Turrets
Jangle Area
Noutral
Mid Lane
Monsters
Jumple Area
Neuhral
729
Monsters
Level
Gold
Figure The macro-goal designed for abstracting player's strateystrategy. The red and blue on the map indicate two teams. and the map can be divided into 3 lanes (the top, middle, and bottom lane), 4 jungle areas, and base areas. Circles represent turrets. The brown arrows in the left part indicate region intent (" ("Where to go"). The turrets, monsters, and minions the right represent unit resources. The brown arrows the right indicate resource intent ("What do"), which means that the agent will kill units order level up or gain gold.

---

3.1.1 Macro-goal Definition
The motivation for designing MOBA AI macro-goald gorll is inspired from how human players make strategic decisions. Human players usually make strategic decisions according "Where go" and "What to do". Thus, we define two types of macro-goals for all MOBA games, i.e., region intent ("Where to go") and resource intent ("What to do"), as shown in Figure 2 The region intent the next region where the players move to, as shown in the left part of Figure 2 The resource intent is the quantity of game resources that players aim to obtain, including turrets, monsters, gold assignment, and other resources commonly found in MOBA games, as shown in the right part of Figure 2
3.1.2 Macro-Goal Label Extraction
T,
T
T
T,+
C=+
Tu+C+E
Region
Turrets + Minions
Region
Monsters
Figure 3: example of macro-goal label extraction. The macro-goals at T, is move into the opposite middle lane and gain gold resources from turrets and minions, while T is to move to upper jungle area grab the monsters to level up. C' is the delta of frame numbers hetween the state and its extracted macro-goals.
We extract macro-goals goals from top e-sports players' matches. Given a state 8, in the trajectory, we define
goal and extract its region intents and resource intents in the future frame the
as label, i.e., macro-goa 9: = f((++c). Each dimension denotes an aspect of the macro-goa current state. The multi-dimensional goal can guide our policy learning to achieve multiple goals at the same time. concrete example of macro-gov goal label extraction is shown Figure The trajectory consists of frames from top players' matches. Let T, denotes the current frame. We sample the frame with goals from the interval [T. + E, T. + C' + e). where a hyperparameter representing the delta of frame numbers between the state and its macro-goals, generally 30s. is hyperparameter denoting the range of noise to avoid overfitting. Intuitively, by setting up labels in this way, we expect the
to learn the mapping from the current state 8, its future macro-goal gt-
3.1.3
Training
Game Stats Feature
ML.Ps
Unit Feature
Max Pooling
Units Fibbed
Monster
value
Macro-goal Loss
Turrets
Minions
Shared MLPs
key
Strutegy Eibed
FC
Resource Intent
Heros
Region Intent
Lineup-1
Limeup-2
query
FC
Auxiliary Loss
Attention
Lincup-N
MLPs
Mask
Strategy Flag
Figure 4: Network Architecture of the Meta-Controller. First, the features of cach unit are extracted hy some shared MLPss to the units embedding. Then, the lincup information is encoded by MI.ss and used as query vector calculate the attention between the strategy and units embedding. Finally, the strategy embedding and game stats embedding are used to predict macro-goals ((x))* and strategy flag d(x,)aux
After extraction of macro-goals, we train the prediction model, i.e., Meta-Controller, via supervised learning. The network structure of the Meta-Controller is shown in Figure 4 To predict macro -oals,
5

---

we input current game state features including observable unit's attributes and in in-game statistics information, e.g., health point(hp), location, gold, level, etc. Note that our game state features include information of herves and lineups, where the lineup means combination of five heroes. Therefore, the Meta-Controller can predict macro-goals of different strategies according to the lineup features.
Beside, because the strategies in MOBA games have strong connections with lineups, we regard the auto-encoding task of lineups as an auxiliary task Tour further model this relationship. plays a role of learning latent representations of strategies. Then, we exploit the attention mechanism in which the latent representations of strategies as queries and the hidden encoding of game states as keys and values. In addition, to solve the imbalance of macro-goal labels and improve the
L (+) LF1. (6())), 9²) + (cx((((s)) you
(L)
where L l the focal loss of predicting labels, LCE cross-entropy loss of the auxiliary task, and is the balancing parameter. We use Adam Kingma and Ba (0144 to solve the optimization problem.
3.2 Macro-goals Guided Training Framework
We integrate the Meta-Controller into the original actor-critic framework as shown Figure [ Given the state 8, at frame 4, macro-goal Jr is generated by the pre-trained Meta-Controller. To guide the trained policy towards gt. inspired by goal-based RL Andrychowicz et al. [ er al. (2015). the policy and value function make decisions according to not only st but also gr- For each N frames, the Meta-Controller updates a new goal 9: to follow the strategy. And the same goal Ie is kept during the whole N frames it, t + N) estimate parameters of policy To To(aas 9t) and value functions (s Gr). Besides, we sample macro-goals from the softmax distribution of the Meta- Ceta-Controller prediction to further increase the policy diversity even with the same lineups.
Macro-strategy Level St (s,)
IL+N (S
Miero-operation Level (S.
TTp (5
To No
Policy- Value Netwrok
Policy- Netwrok
Policy-Value Netwrok
Meta-Controller
Meta-Controller
f(s))
5p+1
Setw
f(ceN)
Figure 5: An example of a trajectory with MGG.
We design an intrinsic reward Rgoal to guide the policy towards the macro- macro-goals as follows, which represents the delta of the distance between the current macro-state and the macro-goal.
poout (st, ae, $ g.) - (s(:)) - gi| - f((4))1)) - 9.
(2)
where 인. = (sxs) is the macro-gar -oal generated by the Keta-Controoller at frame f is the macro-state extraction function, which the same with the label extraction process. If the macro-state f((411 more close to ÿ. than fs,), the agent will receive a positive reward, and vice versa.
We combine the intrinsic reward Roeol with the extrinsic environment reward, and the agent is trained to maximize the accumulated environment rewards as well as perform macro-gon goal distribution matching. Therefore, the agent can explore the best way to achieve the goal while learning the human strategy, so to avoid the decline of micro-operation ability caused by completely imitation learning.
Policy updates. In order avoid training instability our large-scale distributed environment, we use the Dual-clip PPO method Ye el al. [2020a]. Differing from the original algorithm, we introduce the macro-goal g into the policy no (a st, ÿc) and advantage estimation Á 9t).

---

When (o( 9.) >> Tosu (orls: (elss:.g)) 9r) and A, < 0, the radio r,(0) = =o (c. is huge, which
we (ael
causes the large and unbounded variance since re(0) < 0. Dual-clip PPO introduces another
clipping parameter e that indicates the lower bound when A, < 0. The new objective is the following:
T.policy (0) = R. [max(cAs, min(clip(r.(0)) - v, + T) n.)
where 7 is the original clip parameter in PPO.
Value updates. We use the multi-head value mechanism Ye et al. 2020a to incorporate the value of macro-goal. Specifically, the value of achieving macro- goals are regarded one head. Therefore, the value loss is defined as follows:
cuiue (0)
B1
Σ
(R.
V())], Viocal
E
WuVe
heads
heads
where wa is the weight of the k-th head and V (8t, 96) is the k-tth value.
Experiments
In this section, we compare the state-of-the-an art methods of supervised learning (SL) Ye et al. [2020b] and reinforcement learning (RL-baseline) Ye al. [2020a] with the MGG method in the game of Honor of Kings, from the aspect of capability and diversity. Then, we compare MGG, RL-baseline and GAIL Ho and Ermon (2016) some special lineup systems to evaluate the capability gains from increasing the policy diversity. Since the GAIL algorithm has a high sample complexity. which makes it impossible to train full hero pool, i.e., 102 heroes. Thus we compare GAIL with other methods only on a small hero pool (20 heroes) in the case study section.
4.1 Experimental Settings
We construct a training dataset by collecting replays from the top %% human players to train the We extract 4060-dimensional state and 64-dimensional macro-goal pairs (sl .g") from the raw data, while the delta C' is 30 seconds and the noise is seconds. We train only a single model for all heroes. We use 8 NVIDIA P40 GPUs for about 26 hours of training, and the batch size of each GPU is set to 512. We set a = 0.75, y = for focal loss LF Lin et al. [2017]. and set X for the weight of auxiliary task. We use Adam with the initial learning rate of 0.0001.
To make fair comparison, except for the additional macro-gen soclo-goalloo features and intrinsic reward, the rest of the MGG network architecture and training settings are exactly the same as the RL-baseline. MGG and other RL methods adopt self-play training and train by randomly selecting heroes over a physical computer cluster with 60,000 CPUs and 830 NVIDIA V100 GPUS. The batch size of each GPU is set to 4096. All methods converged after training for a total of 420 hours. We evaluate methods on 102 heroes in Section 4.2 and 4.3 and about 20 heroes in Section 4.4 For online evaluation, the response time of AI is set to 193ms, including observation delay (133 ms) and response delay (60 ms). Response delay consists of features, models, result processing, and transmission delays. The average APMs of Al and top players are comparable (80.5 and 80.3, respectively).
4.2 AI Performance
We compare MGG with built-in bots, supervised learning agents Ye et al. 20220b (SL., and the original reinforcement learning agents Ye et al. (2020a) (RL-baseeinee. Note that RL- is also the state-of-the-an method in the game of Honor of Kings. We use all methods to play pairwise matches, 400 games for each pair, and then calculate the Elo scores (Coulom (2008). All lineups in each match are randomly selected from the full 102 hero pool.
In addition, during the period from April 1st to April 4th, 2021, we conduct a large-scale human- machine test the game activity to evaluate the average level of human players. We run about 100,000 tests on RL methods, where players are randomly matched against the RL-baseline and MGG agents without being told about the opponent model, just knowing that the opponent is the AL
'see Supplementary for details.
7

---

And we also conduct 50,000 tests on the SL method. The win rates of SL, RL-baseline, and MGG against humans are 93.10%, 97.77%, and 97.81 respectively.
Figure shows that the MGG method performs better than all the baseline methods in real games, demonstrating the benefits from modelling policy diversities and taking advantage of the characteris- tics of lineups.
Al Performance
3500
2851
3048
6016
30.00
905
2183
1788
90%
959
0%%
0%
1.19%
0.01%
1016
0%
No-op
Built-in
Human player
SL
RL-haseline
MGG
Elo Scores
Win Rate Against MGG
Figure 6: The performance comparison between MGG and the haseline methods in the game of Honor of Kings. Blue: Overall comparison of Elo scores. Yellow: Baselines vs. MGG win rate.
4.3 Diversity in Different Lineups and Matches
4.3.1 Diversity in Different Lineups
We visualize macro-state embeddings of MGG, RL-baseline, and human players in Figure 7 and points are colored according to different methods. Figure 7 shows that MGG plays more diverse policies than RL-baseline. The macro-state space of MGG (blue points) covers both the RL-bascline (red points) and human (yellow points), which indicates MGG can play both the human strategies and RL-baseline strategies. Besides, we can observe that the points of MGG and human strategy are clearly separated from each other. Conversely, the points of the RL-baseling strategy are mixed.
We evaluate the Ravies-Bouldis Index Davies and Bouldin 1979 of macro-states embeddings in different lineups, as shown in Table We can see the MGG has a lower Davies-Bouldin Index, which indicates the MGG's macro-strategies have better separation between different lineups.
Table 1: Davies Bidiisn Index MGG and RL-baseline differ- ent lineups. The values closer to zero indicate a better parti- tion, meaning that the strategies (macro-state space) varies more between the different lineups.
Davics-Bouldin Index
RL-baseline
Player
MGG
RL-baseline
10.72
Figure 7: Macro-states t-SNE Embedding of different algorithms in the same space. Left: comparison of MGG (blue points) and RL-baseline (red points). Right: comparison of MGG and human player (yellow points). The strategy space of MGG covers both.
MGG
1.76
human
4.3.2 Diversity in Different Matches
The previous section shows that the MGG agent presents diverse policies in different lineups. To evaluate the diversity when the agents pick the same lincup in different matches, we define the macro- state entropy metric / = Zeeg P(c))ogP(()) where P(c) is the probability mass function of macro-state and с = fs). In this experiment, we run 1000 matches for each comparison, and we let agents play against the same bot with the same lineup and start the game from the same state.
In T we compare MGG with the RI.-baseline, which shows that the MGG agent plays diverse polices even in the same lineup due to sampling the macro-god from the fett--ootrrlller in real-time.

---

Table Comparing macro-state entropy of each method in the different matches while picking the same lineup.
Built-in bots
RI-baseline
MGG
macro-state entropy
0.000
0.014
0.408
4.4 Case Study
In Honor of Kings, the most common strategy for human players is three-lane-stratee in which the marksman, mage, and warrior (or tank) obtain resources from three lanes, respectively, the assassin grabs the neutral jungle resources, and the supporter roams to assist them without gaining resources. Therefore, the general reward function Ye et al. [2000a] is designed following the common strategy, so agents will perform similar strategies in different lineups and thus cannot perform the most suitable strategy in lineup systems that differed significantly from the common strategy to evaluate the diversity and capabilities of MGG and other methods.
special lineups. To evaluate the effectiveness of MGG this issue, select several
We also compare MGG with GAIL Ho and Ermon (016], which shown to be effective behavior imitating tasks. GAIL uses discriminator to distinguish whether state-action pair is from the human player or the learned policy, meanwhile optimizes the policy to confuse the discriminator. However, adversarial training will greaty affect the training efficiency of the policy, separately under certain special lineup systems, not the full hero pool.
only train
4.4.1 Case of Marksman-Con System
Marksman- Core System refers to substituting supporter for the mage so that the powerful marksman can obtain the resources of the two lanes to gain an advantage over the enemy in money and equipment as fast as possible. Therefore, we evaluate the achievement of the resource allocation strategy within during the match against the RL-bascline, and then analyze the advantage over the enemy through the marksman's 's money difference (moneemoorkmman to RL-baseline and GAIL, MGG is closer the distribution of human players in terms of marksman money rate and money difference, indicating that applying MGG is more beneficial for achieving the resource intent strategies, which is difficult for the original reward function to achieve.
the team by analyzing the money rate of the marksman (I mmoneytcem_indee
money
(mmnnyyennny (eneeeeeennnnnez))))e As shown in Figure g| compared
mins
8 mins
mins
(b) Money mitterac tn Mane Nate
0.12
0,52
Table 3: Neutral monster resource metries of Carbb
0.22
0.26
Resource- Grab System. In the early stage, the near side resources are at most 2 and the far side resources are also capped at 2.
0:25
0.23
0.25
exi
144G
1504
2173
2006
1750
2810
4410
Side
Far Side
Total
Resources
Resources
Resources
Human
1,317
0.561
1.878
Rl-baseline
MGG
Player
RL-bascline
1.002
0.000
1.002
GAIL
1.141
0.056
1.197
Figure 8: Statistical indicator of resource allocation in Marksman-Core System.
MGG
1.741
0.658
2,399
4.4.2 Case of Resource-Grab System
Resource-Grat System means using a strong assassin in the early stage (first 2 minutes) to grab the resources of neutral monsters closer to the opponents, which belongs to the long-term regional intent strategy. Therefore, the success of the strategy can be expressed by the number of neutral resources snatched the early stage. As shown in Table 3, RL-baseline maximizes short-term gains by only grabbing the resources closest to it based the common policy. In contrast, MGG effectively implements this long-term strategy and outperforms the human by 0.52 monster resources in total, Besides, GAIL is diffieult to guide agents at the micro-operation level implement the regional intent for grabbing the enemy's 's resources because this strategy requires long-term planning.
see Supplementary for cases of lineup systems and results of their human- machine test.

---

4.4.3 Capability Analysis
To further investigate whether increased diversity can improve the capability of the particular lineup systems, we test the methods against each other, with pairs playing 100 games. In Figure / the Elo scores of built-in bots, SL agents, the RL-baseline agents, GAIL agents, and MGG agents are provided. Elo scores demonstrate that different strategies have significant impact on the upper limit of its capability for the same lineup. Figure | shows that with improved strategy diversity, MGG increases 312 and 560 Elo scores over baseline in the above two lineup systems, respectively. Thus, training MOBA AI with strategies diversity and enhancing specific lineup systems' capabilities benefits significantly from our proposed method. The video of the above case can be found in
Elo Scores
3000
2386 2547
2500
2029
2279
2341
1987
2000
1500
1006
1004
1000
500
Marksman-Corf System
Resource-Grab System
Built-in bots
SL
RL-basclinn
n GAIL
MGG
Figure 9: Comparison of Elo scores between MGG and other methods in two special lineup systems.
Conclusions
In this paper, we propose the MGG framework to learn diverse policies during training. We define macro-goals to depict diverse strategies in MOBA games. We sample macro-goals from softmax distribution of the towards specific strategy, we design an intrinsic reward to indicate whether the agents achieve macro-goale -oorol during training. The trained agents can play diverse policies in different matches and exploit lineups' characteristics to improve capabilities. we will investigate non-transitivity in strategies of MOBA games using game theory. We will also try to find a solution that is closer to Nash equilibrium in MOBA games.
prediction, which enhances policy diversity. To guide the policy
the future, after learning diverse policies,
Broader Impact
To the research community. MOBA (Multiplayer Online Battle Arena) poses a grand challenge to the AI community. Even though the existing MOBA-game Al systems have achieved or even exceeded human-level performance, they still suffer from the lack of policy diversity in such a complex environment. To this end, this paper is introducing a learning paradigm to train diverse policies for the MOBA-game Al systems. We herewith expect that this work can provide inspiration for the diversity of strategies in various game Al research.
To the game industry. Our AI has found several real-world applications in the game, and is changing the way that MOBA game designers' work, elaborated as follows: 1) Game balance testing. In MOBA and many other game types, balancing the ability of each strategy essential. Using similar techniques presented in this paper is an easy way to construct strategy balance testing tool for MOBA games. 2) PVE (player vs environment) game mode. Introducing AI with diverse strategies into the game's PVE mode is low-cost method to increase the interest of players.
To the e-sports community. Like AlphaGo, method can provide high-quality low-cost training environment with diverse strategies for e-sports players. The Al system with diverse macro-strategies and hegh- micro-operations suitable training partner for e-sports athletes.
7 Funding Disclosure
This work is supported by the Tencent Al Department and the Tencent TiMi Studio.
10

---

References
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in neural information processing systems, pages 5048-5058, 2017.
David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech M Czarnecki, Julien Perolat, Max Jaderberg, and Thore Graepel. Open-ended learning symmetric zero-sum games. arXiv preprint arXiv:1901.08100 2019.
Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret mini- mization. In International conference on machine learning, pages 793-802, 2019.
Sheng Chen, Menghui Zhu, Deheng Ye, Weinan Zhang, Qiang Fu, and Wei Yang. Which heroes to pick learning draft in moba games with neural networks and tree search. IEEE Transactions Games, 2021.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances neural information processing systems, pages 4299-4300 2017.
Rémi Coulom. Whole-history rating: A bayesian rating system for players of time-varying strength. In International Conference on Computers and Games, pages 113-124, 2008.
D. L. Davies and D. W. Bouldin. A cluster separation measure. IEEE Transactions on Pattern Analysis and Machine Intelligence, (2):224 227, 1979.
Carlos Florensa, David Held, Xinyang Geng, and Picter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pages 1515-1528. PMLR, 2018
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. arXiv preprint NEXY:16066034766 2016.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, pages 8011 1-8023, 2018
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M Czamnekki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint /.t:::/11.09990 2017.
Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, al. Human- level performance in 3d multiplayer games with population baasedttoo--beed reinforcement learning. Science, 364(6443):859 865, 2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Tejas D Kulkarni, Karthik R Narasimhan, Ardavan Saeedi, and Joshua B Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. arXiv 2016.
preprint arXiv:1604.000
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisy, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337):508-513 2017.
Ofir Nachum, Shixiang Gu, Honglak Lee,, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. arXiv preprint arXiv: 2018

---

Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference on Robolics and Automation (ICRA), pages 6292-6299, 2018.
OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, llya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. 2019.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pages 1312-1320, 2015.
Ruimin Shen, Yan Zheng, Jianye Hao, Zhaopeng Meng, Yingfeng Chen, Changjie Fan, and Yang Liu. Generating rehavior-divers. game ais with evolutionary multi-objective deep reinforcement learn- ing. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAT), pages 3371-3377 2020.
Victor do Nascimento Silva and Luiz Chaimowicz. Moba: new arena for game ai. arXiv preprint arXiv:1705.10444, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche,
Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587)-844-88) 2016.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):3500 350-3344) 2019.
Hua Wei, Deheng Ye. Zhao Liu, Hao Wu, Bo Yuan, Qiang Fu, Wei Yang, et al. Boosting offline reinforcement learning with residual generative modeling. arXiv preprint arXiv:2 106. 10..
10411, 2021.
Bin Wu. Hicrarchical macro strategy model for moba game ai. In Proceedings of the AAAI Conference Artificial Intelligence, pages 1206-1213, 2019.
Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, et al. Towards playing full moba games with deep reinforcement learning. arXiv preprini 201 1.1692, 2020.
Deheng Ye, Guibin Chen, Peilin Zhao, Fuhao Qiu, Bo Yuan, Wen Zhang, Sheng Chen, Mingfei Sun, Xiaoqian Li, Siqin Li, et al. Supervised learning achieves human-level performance in moba games: case study of honor of kings. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Deheng Ye. Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control moba games with deep reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6672 2020.
Menghui Zhu, Minghuan Liu, Jian Shen, Zhicheng Zhang, Sheng Chen, Weinan Zhang, Deheng Ye, Yong Yu, Qiang Fu, and Wei Yang. Mapgo: Model-assisted policy optimization for goal-oriented tasks. arXiv preprint 1rXiv:2105.06: 2021.
12


Reviews:
[{"review_id": "fbu_qADg5Y7", "reviewer": "Reviewer_y4xZ", "summary": "The paper addresses RL agent training in MOBA games, specifically the ability to acquire a diverse set of policies when playing a variety of strategic scenarios (team compositions, aka lineups). Human example data is processes to extract future states reached from initial states, treating these future states as goals. A supervised learning model is trained to predict the future goal state from a given game state using an attention module that accounts for the strategic scenario (lineup) and game state (unit features & aggregate game stats). An existing RL framework is then augmented to condition on this goal state and trained with a reward that encourages reducing the distance to this goal in addition to the baseline reward from the environment. A wide variety of results show the method is better able to adapt to unique strategic scenarios (lineups) and exhibits a broader array of behaviors compared to the baseline agent or an imitation learning agent (GAIL) while retaining strong performance against human players.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has several strengths:\n- Strong empirical results.\n- Variety of experimental comparisons. The experiments test a variety of features of the approach in terms of diversity of strategies, adaptability to new lineups, and overall competitive performance.\n\nThe paper's primary limitations:\n- Limited novelty: Using imitation learning to bootstrap long horizon decisions from human data is not novel per se. The novelty lies in the combination of elements.\n- Lacking discussion of limitations and remaining open problems. As detailed below there are some areas where there is a gap between the performance of the method and the ideal that should be discussed.\n\n\nBelow are comments specific to the review criteria prompts.\n\n# Originality\n\nUsing imitation learning to bootstrap long horizon decisions from human data is not novel, nor is goal-conditioned learning. The primary technical novelties here lie in the specific modeling choices and their integration. I believe this integration is novel and valuable given the complexity of the domain and problem. It is clear this builds on prior efforts while enabling diversification of policies in a different way.\n\nI am not sure if the attention mechanism employed for learning goals to predict is specifically new, but it seems to be.\n\nRelated work is adequately cited for the relevant techniques to compare.\n\n# Quality\n\nThe results partially support the claims. Some specific points:\n1. The new method does not seem to improve over the baseline agent against humans: Figure 6b shows a ~0.04% difference in (cumulative) win rate. The text claims this is superior performance but it does not seem obvious that is true (and there is no statistical test as the evaluation was performed once).\n2. MGG does cover more of behavior space compared to the RL-baseline or players (Figure 7). But MGG also more parts of the space that players do not occupy. It is not clear whether this is a good thing or a problem.\n3. There is no comparison of computational costs of the techniques. While greater efficiency is not claimed per se, this would help complete the picture.\n4. Figure 8 results show a substantial gap remaining between the MGG method and player behavior.\n\nThe methods employed are appropriate for training RL agents in team games, and incorporating goal prediction into the policy training is clear incremental improvement over prior work.\n\nThe paper is not careful to address weaknesses or remaining gaps. Points (1), (2), and (4) above were not addressed. Generally limitations are not discussed. The checklist indicates assumptions were addressed in section 3, but there is no explicit text to this point.\n\n\n# Clarity\n\nThe text can be difficult to read at points, but is generally understandable.\n\nThe organization is adequate and the reader is informed of relevant methods and prior work related to the techniques (like double clipping).\n\nSome clarifications and additions that would improve the text:\n- What were the instructions given to participants?\n- What is the value of the in-game rewards given to participants? \n- How was consent obtained from participants?\n- (describing supervised learning) \"Finally, the macro-goal is predicted via a convergence model of 1 million steps.\" Not sure what this means. Does this indicate the model trained for 1 million steps and obtained convergence at that point?\n- Table 3: How were the human values for different goals determined? For the RL agent we know the conditioned goal but it was not clear how the human goal was decided.\n- Consider briefly explaining the Davies-Bouldin index for readers unfamiliar with clustering.\n\n\n# Significance\n\nThe results are modestly important. Researchers looking to condition agent behavior on human examples of strategic choices could adapt the work to other domains. This will be valuable to those training RL agents starting from imitation learning in domains where strategic or other long-term choices are relevant.\n\nThe paper advances the state of the art in getting diversity (without compromising quality) for MOBA games in specific and likely for RL game AI agents in general. The comparison to GAIL shows this is superior to at least some alternatives for the same problem.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No. Potential negative societal impacts are not discussed. MOBAs as real-time control problems feel like natural models for military automation tasks (training agents to perform various team activities). The paper-specific design to allow specification of a goal that controls agent behavior is easily applied to controlling RL agents toward whatever an actor using the agent may desire, including potentially harmful actions.\n\nNo limitations of what the method can achieve is discussed. These are explained above under \"Quality\". For example, the Figure 8 results show a substantial gap remaining between the MGG method and player behavior - why?. In Figure 7 MGG produces large clusters of behaviors human players do not - what might this indicate? In Figure 6b the differences between MGG and RL-baseline around on the order of 0.04% - is this statistically significant? Why is it so small? None of these are major failings, but they are not framed as limitations to be considered or explained.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Privacy and Security (e.g., consent)", "Responsible Research Practice (e.g., IRB, documentation, research ethics)"], "time_spent_reviewing": "2"}, "extra_text": {"ethical_concerns": "The paper does not address how consent was obtained from players, only that players were informed of being in the experiment. Given the online game setting players may not have had a choice - it's not clear whether/how this fits into standard ethics review protocols (IRB & associated documentation). I'm not confident this crosses any ethical lines, but I would like to flag the topic for expert review.\n\nNote that the checklist indicates \"We declare the test instructions in the game\", but these are not included in the paper / supplement, so I have no way to review them.\nSimilarly the checklist answer about hourly wages indicates \"We give out in-game rewards for testing\", but what these are and how they should be valued is not indicated. Again, a tricky issue as game rewards are hard to convert to real world value/currency, but I wanted to hedge toward being safe.\n\nI would also note that NeurIPS may want to establish some specific details on ethics policy for running experiments in live services (including games) - many companies operate live services where things like IRB documentation, informed consent, and compensation are all different to standard academic procedure. To the best of my knowledge these A/B tests are often conducted without informed consent (often without information). Whether running A/B tests of user interface design or playmate AI, these types of experiments stike me as topics that will appear across many papers appropriate to NeurIPS."}}, {"review_id": "A06GTSET5Kh", "reviewer": "Reviewer_uPYF", "summary": "The authors propose a method for using limited human data to generate macro-goals that serve as an auxiliary imitation loss and reward for deep reinforcement learning agents playing a mobile 5v5 MOBA game (Honor of Kings). The proposed approach provides 'macro' supervision for agent behavior while letting the agent learn the finer details and micro actions to reach goals or maximize game reward.\nExperiments compare the proposed approach to a variety of baselines and indicate higher ELO scores and lower exploitability by human players.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Overview\n\n\"Learning Diverse Policies in MOBA Games via Macro-Goals\" presents a compelling direction for combining offline human feedback with a reinforcement learning policy, and the evaluation criteria investigating exploitability is novel and supports the paper's stated goals. Certain areas of the paper require work to help the reader reproduce the results or understand the significance of the outcomes. \n\n# Originality\n\nThe paper's combination of human data using imitation learning and reinforcement learning has been previously explored in the context of 'human preferences' and summarization work. The approach is novel in the context of MOBA games and demonstrates that in long-action sequences this form of supervision is applicable and can lead to demonstrably more diverse policies.\n\nThe choice of neural network, features, domain, and other aspects are not the focus of the paper, and are also not novel.\n\nThe paper measures the exploitability of the agents using human evaluations (Figure 6) and this evaluation format has not been used previously in the context of learning game AI policies. This test, while difficult to reproduce, serves as an invaluable showcase for the method's impact on realistic play. \n\n# Clarity\n\nThe paper's stated goals and application domain is clearly stated. Similarly, the purpose of the experiments is explained and motivated. Certain results and their applicability hinder the reading and contextualization of the work. As an example, 4.4.3 discusses capability analysis while using ELO scores. The compared approaches are stated to be 'converged', however this term is loaded and hard to transfer across architectures and setups. Similarly the results in Figure 6 and 9 do not provide statistical significance of the results, or potentially a comparison across different points during training (as a way of further confirming that the systems have indeed converged). As such, the conclusions from many of these experiments are hard to measure. A key question that should be answered is: \"does using human games and an additional goal reward in this way provide better results provided with the same resources/time/energy as our reinforcement learning baseline?\". Ideally a good control would allow the reinforcement learning policy to train supervised from human trajectories and then train with self-play, and account for the additional parameters and training time required by the imitation training.\n\nSection 4.4.1 Why is achieving \"resource intent strategies\" important? Does the trained policy *need* to be good at this to improve at the game, or is the purpose to be as similar to human play?\n\nSection 4.4.2 discusses human performance for monster resources: what is this human evaluation? how are they chosen? How many attempts are done by humans? Are these humans professional or amateur players?\n\n# Significance\nThe proposed approach is tested on a challenging benchmark and uses significant resources to train the agents and evaluate them.\nThe significance of this work is limited by two key factors: 1) research in this area is important, but the proposed technique is hard to reproduce owing to the high computational cost, lack of a public interface or system to test and train Honor of Kings agents. 2) the results are provided without statistical significance numbers or selection criteria for human players. The ELO numbers and reward scores should ideally be amended in the final version of the paper with some measure to understand the size of the gains over GAIL or RL-Baseline. The human player performance is hard to asses, and perhaps an ELO score for human players, or a description of the selection process would help. (specifically the exploitability plot in Figure 6 strongly depends on the kind of players that tested either agent, and whether they were allowed to play against either system, or only one kind repeatedly, etc.)\n\n# Areas for improvement\n\n## Statistical significance:\nFigure 6 and Figure 9 provide the most impressive results of the paper, but lack any statistical significance metrics which would help understand the gains between RL-baseline, GAIL, and MGG.\n\n## Wording:\nL249 \"this test effectively proves\" I would reword this as \"this test suggests\" or \"provides evidence for\" as conclusive demonstrations with human testing and a limited number of samples are unlikely, but do provide a great benchmark.\nL259, L265 \"marco states\" -> \"macro states\"\n\n## Citation style:\nOpenAI Five citation should preferably use https: //openai.com/bibtex/openai2019dota.bib (e.g. OpenAI et al.)\n\n## Related Work:\nL78 \"All the mentioned works except the SL model adopt a single strategy regardless of the\n79 heroes and lineups, which cannot take full advantage of heroes and lineups’ characteristics\" -> Slight correction: OpenAI Five has a latent vector encoding lane preferences + a representation of the heroes that affect the employed strategy, so in practice the strategy will change dramatically between games when those aspects are randomized.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "# Limitations\n\nThe authors do not describe specific limitations of their work. In particular use of human data can be seen as a limitation as compared to other MOBA efforts. Furthermore the introduction of additional rewards and losses may complicate or destabilise training, and this should be addressed or discussed.\n\n# Societal Impact\n\nThe authors properly address the societal impact of their work in the checklist and the main text.", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "0pxKt-DPQIG", "reviewer": "Reviewer_S81z", "summary": "The paper proposes a novel method for learning policies to play MOBA (Multi-player Online Battle Arena) games which makes use of expert supervision to extract and learn the prediction of high-level or macro goals (meta-controller) which is subsequently used to train a policy using reinforcement learning to execute each goal with a low-level micro-controller. The proposed method has an emphasis on diversity in strategies which is built-in by sampling from the meta-controller and is also evaluated with quantitative and qualitative experiments. This diversity in strategies is stated to be the reason for improvement over human performance when compared to state of the art baselines. Several qualitative demonstrations of the proposed method (Section 4.4) show how non-conventional strategies are adopted more easily than baselines. A human vs machine study is also presented for measuring the win-rate of the proposed method and baselines and how it changes with more test time i.e. how the win-rate changes when humans are able to learn and improve their own performance against the AI matchups.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper presents a novel method for pushing the state of the art of AI performance in MOBA games. The large-scale human study to evaluate performance against baselines and the performance exceeding the state-of-the-art baselines clearly demonstrate the effectiveness of the proposed MGG method. Specific analyses also support the diversity claims made about the learned strategies. However, the paper in its current state leaves a lot to be desired in terms of clarity, broader motivation and accessibility. Despite the thorough empirical analysis made in this paper, it remains inaccessible to the reader due to these writing issues. \n \nBelow are detailed comments on the strengths and weaknesses that informed my evaluation of the paper.\n \n \n## Strengths\n \n1. The presented MGG method’s main selling point is the diversity in strategies that are aimed to improve performance against human matchups, since humans are capable of overcoming unimodal/repetitive strategies by AI agents given enough time and practice. The claim for diversity in the proposed method is supported by a quantitative analysis with measurements of goal-entropy and Davies-Bouldin Index of the goal embeddings, and a qualitative analysis of specific cases of non-conventional strategies in the chosen game (Honor of Kings) which typically do not align with the default reward models proposed for this game.\n \n2. The large scale human-study with human vs AI matches clearly demonstrates that the proposed method retains performance (w.r.t the baseline) with increasing human test time, which is in line with the hypothesis that diverse strategies are better than unimodal strategies against human matchups.\n \n3. The choice of using expert demonstrations for learning only the high-level macro-goal prediction network is intuitively appealing -- it aims to mimic broader human intentions without worrying about matching the exact execution of each intention.\n \n4. For readers that are familiar with MOBA games, the two case studies in Sections 4.4.1 and 4.4.2 (and the videos linked in the paper) are impressive demonstrations of what the proposed method is capable of in practice. Such strategies seem to be very difficult to learn for AI agents but are easy or obvious for human players.\n \n\n\n## Weaknesses\n \n\n#### Lack of clarity in method description.\n\n1. The second paragraph in Section 3 is confusing due to lack of proper explanation and use of arbitrary new notation. The difference between macro states and macro goals is not clear. The role of f i.e. the macro state extraction function is not made clear until much later in the paper. The “general reward function” is never explained -- what does this mean and what does it consist of? What does the “common strategy” mean? Why is notation introduced for human macro states when this notation is never used again in the entire paper (i.e. served no purpose later on). It could have been explained without any notation. In my opinion, the goal space should be completely defined at this stage of the paper, along with the choice of goal dimension and a description of what the dimensions of the goal specify. Also, notation for intrinsic reward R(f(S), G, A) is introduced without being accompanied by a definition (the definition is much later in the paper).\n\n2. Auxiliary tasks are first mentioned in L135 but what auxiliary tasks are used is not mentioned until L173. Since only a single auxiliary task of auto-encoding is used, this explanation does not deserve to be pushed to such a later stage from it’s first mention.\n\n3. L161 states that C is the number of frames between state and its goal, but it is not made clear how C is specified. Later in the paper, C is then stated to be a hyperparameter which has a fixed value. This description about C being a hyperparam of choice and its value should all be in the same first mention of C in the paper.\n\n4. Equation 2 (and L192 below it) assume that the goal space admits a metric in the form of an L1 norm, but the goal space is not defined at this point in the paper. There is an implicit assumption that the goal space is a multi-dimensional euclidean space (R^d).\n \n\n#### Accessibility. \n\n1. The paper does not introduce key MOBA-specific terminology before using it at several points in the paper. A basic explanation of the mechanics, heros and progression milestones of the game should be explained before terms like “destroying the crystal”, “over 102 heroes…” or “ban/pick capabilities” are used. However, given that this paper specifically focuses solely on MOBA games, this is a minor issue.\nThe paper mentions “league based” methods in the intro (L43 and L66) but does not explain what this category of prior work means until L81, where a very vague explanation is presented -- “league-based methods train multiple agents simultaneously to explore strategy space and require massive computation”.\n\n2. The “general reward function” is mentioned several times in the paper but only introduced on L274 after citing Ye et al. [2020a] without any description. This seems to be an important fact to include in the paper well in advance.\nThe fact that the proposed method is an instance of an imitation learning method where expert data is used is not made clear during comparisons with baselines. It is not mentioned that the “RL-baseline” method does not use any human supervision, which is an important point to note in the comparison.\n \n\n#### Empirical issues.\n\n1. For the main result of AI vs human win rate obtained from the large scale human study, the Y-axis range in Figure 6 (b) seems to be extremely narrow (98.72% - 98.86%). Due to the absence of error bars, it is very difficult to judge the statistical significance of the presented results. It is also very surprising that the RL-baseline, despite lacking diversity, is able to retain such a high win rate after 26 hours of accumulated human test time. This seems to strongly oppose the premise of the paper that humans can easily overcome AI strategies if they lack diversity.\n\n2. Lack of error bars: Figures 6, 9 and Tables 1,2,3 lack error bars, despite the authors having answered “Yes” to the checklist question about whether they have included error bars.\n \n\n## Other minor errors\n\n1. L259 -- A brief explanation of the Davies-Bouldin Index and its relevance should be presented instead of just citing the original paper.\n2. Last line of Conclusions seems very vague, what does “non-transitivity in strategies” mean? The directions for future work can be made more clear.\n3. Table 2 and Section 4.3.2 mention measuring “state entropy”. Shouldn’t this be called “macro state entropy” instead?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have addressed the limitations of the paper adequately. They should also cite Ye et al. [2020a] for the detailed broader impacts that they have mentioned for MOBA game based AI.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "0CBB19n9tkM", "reviewer": "Reviewer_e26o", "summary": "This paper proposes improvements to an existing deep RL agent for the game Honor of Kings, by introducing a meta-controller trained by supervised learning to predict macro goals high-level players would like to pursue next.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The originality of the method is limited - hierarchical control systems are well established and games of similar complexity and with similar mechanics have been learnt previously - but it remains interesting to the sub-community focused on these types of applications as additional evidence of approaches that can be successful utilized in modern games.\n\nI have some concerns about the correctness of statements in the paper that could significantly improve my rating of this paper if adequately addressed in the author's reply:\n\n(1) Figure 6b shows win rate against human without variance. Given the very small differences in percentages on the y-axis, there may not be any significant improvement over the previous RL baseline. Can you provide stronger evidence that the proposed method significantly improves performance in comparison to the chosen baseline?\n\n(2) The contributions on page 2 state the proposed method uses the same computational resources as the baseline, but the proposed method introduces an additional pre-training stage for the macro classifier and augments the policy state space. Both of these modifications will require additional compute - the macro classifier in its own training and the augmented policy space I assume will increase training time for the RL agent. Should this statement instead be that once the agent is trained they use the same resources at the evaluation stage? If so, the claim that this is \"unlike league-based methods\" does not hold.\n\n(3) On line 40 it is claimed that \"the agent should exploit [the opponent] instead of playing with a single policy\" but these two statements are not mutually exclusive. A single policy can learn to exploit any opponent, either by playing a Nash equilibrium strategy or using opponent modelling to recognize online the type of opponent it is playing and condition its policy on this additional observation.\n\n(4) Line 85 claims \"there is no obvious signal ... to represent macro strategies in MOBA games\" as a way to differentiate from prior work but the proposed method defines macro goals (section 3.1.1), uses them throughout training and claims they are applicable to other MOBA games (line 143). I think there are other ways to differentiate the proposed method from the past work cited, but this current statement is contradicted by the work done and so currently weakens the evidence of originality presented in the paper.\n\nOther issues that are lowering my current rating of the quality and clarity of the paper:\n\n(5) Line 232-233 gives statistics about the agents response time, but provides no grounding for how this compares to human players.\n\n(6) In Figure 9 do the built-in bots have an ELO rating of 0? Or is this the no-op agent as in Figure 6?\n\nMinor comments that could be easily addressed if accepted (no need to respond):\n- Due to the complexity of the approach and scale of the study, some details are missing that would be interesting insights to the overall system. Most notably, no results are provided for the meta-controller pre-training and the focal loss is not defined (presumably because it is defined in Lin et al. 2017) - adding these to the appendix would make this a more self contained reference. \n- Line 190: \"even the lineups\" -> \"even WHEN the lineups\"\n- Line 263: \"the marco state\" -> macro", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is limited discussions of limitations in the paper, but plenty that could be discussed to improve the paper:\n\n(1) Elo scores are used throughout the paper to evaluate the agent, without acknowledgement of known issues with this form of evaluation when games (including MOBAs) have non-transitive interaction between policies. See for example Balduzzi et al. Re-evaluating Evaluation [NeurIPS 2018]\n\n(2) Due to the scale of the environment, there are limited ablations and repeats. This is an existence proof of the ability for agents to play at this level, and not a rigorous study or the robustness of this approach. I think this is an acceptable form of publication, but this detail should be acknowledged.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Responsible Research Practice (e.g., IRB, documentation, research ethics)"], "time_spent_reviewing": "4"}, "extra_text": {"ethical_concerns": "This paper presents results from both:\n\n- training the meta-controller on human gameplay data;\n- evaluating the agent in games with human players\n\nBut has limited documentation of how (or if) players provided informed consent to participate. Lines 242-250 imply players in the evaluation phase were made aware they were playing an AI and were rewarded in-game for doing so (noted on the checklist) but I see no evidence that players who's data was used to train the meta-controller gave permission for this use."}}]

Author rebuttal:

Meta-review: After reading the reviews, authors response and discussions, I suggest to accept the paper. The ethical concerns have been answered and the authors took action to conform to what was required by the ethical review. Questions and some concerns on the method have been answered by the authors during the rebuttal and one reviewer upgraded its score from weak reject to a weak accept. The authors try to tackle the problem of learning in a real-time multi-player game which is extremely difficult. Even if the method is not theoretically sound and heavily rely on human knowledge, it is still a technical prowess to achieve good performance on such games. I will also suggest the authors to change their claim concerning the analysis of Fig 6 (b) where the difference between RL and their method is extremely minor. 
Decision: Accept (Poster)