Learning Diverse Policies in MOBA Games via Macro-Goals

1 Introduction . As a stepping stone of Artificial General Intelligence ( AGI ) , Game AI has achieved significant progress in the past decade , including board games Silver et al . [ 2016 ] , poker games Brown et al . [ 2019 ] ; Moravčík et al . [ 2017 ] , Real-time Strategy ( RTS ) games Vinyals et al . [ 2019 ] , etc . Especially , Multi-player Online Battle Arena ( MOBA ) games have been regarded as a challenging problem in Game AI and attracted massive attention from research communities because of the complicated mechanisms such as imperfect information , large state-action space , and long-term sequential decision making . Recently , OpenAI et al . [ 2019 ] develop the AI system on Dota 2 , named OpenAI Five , which defeats the Dota 2 world champion , i.e. , Team OG , within a limited size of hero pool . It trains with deep reinforcement learning and self-play . Ye et al . [ 2020a ] build the AI system on another popular MOBA game , Honor of Kings , which also defeats top e-sports players . Besides , compared with OpenAI Five , their AI system enables playing full MOBA games , which extends the hero pool size from 17 to 40 heroes , adds ban/pick capabilities , and addresses the scalability issue . Even though existing AI systems have achieved or even exceeded human-level performance in MOBA games , they still suffer from the lack of policy diversity . Agents during training are controlled by the same hand-crafted reward signal and then present similar strategies even in different matches , while strategies played by humans usually vary according to situations in the game . One of the disadvantages is that we can counter the single learned policy after several attempts effortlessly . For example , one team defeats OpenAI Five nine times with the same strategy in public tests1 . The 1https : //openai.com/projects/five/ 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . primary counter-strategy is to avoid fights as much as possible and instead push outside lanes to pressure the agents into defending2 . Another disadvantage is that agents with a single policy can not exploit the strength of heroes and lineups . To interest players , MOBA game designers usually make different skill mechanisms for different heroes , which result in heroes play different roles in the game . Therefore , different roles in lineups have different strategies to maximize a team ’ s utility . For example , the supporter should keep their allies alive and give them opportunities to earn more gold and experience . The assassin can efficiently jungle neutrals rather than farm lanes . Moreover , different combinations of heroes construct lineups with different characteristics . Some lineups have advantages in pushing with a quick pace , while some lineups are good at the development of carry heroes . The agent should find a way to maximize the picked lineup ’ s strength instead of playing with a single policy in the game . Although existing methods can learn diverse policies in other games , it remains to be a grand challenge in MOBA games . AlphaStar Vinyals et al . [ 2019 ] uses a framework based on the league to improve the diversity of policies in StarCraft II . The historical models are used to construct the league and train exploiters to highlight flaws in the league and main agents . The main agents are forced to discover new strategies by training against the exploiters . They also define a latent variable according to building orders in human data as constraints of self-play . However , the league-based methods can not be applied to MOBA games directly . First , they consume massive computation resources to construct the league . They use 12 copies of models in which there exist 128 TPU cores . Second , the scenarios between MOBA and RTS games are different . They leverage the nature of StarCraft II and use building orders to represent each strategy , which is unavailable in MOBA games . In this paper , we propose a novel learning paradigm named Macro-Goals Guided ( MGG ) learning to train diverse policies in MOBA games . Motivated by top e-sports players , we define related game information as macro-goals to depict strategic decisions , which constitute macro-strategies in MOBA games . To explore macro-strategies efficiently , we extract macro-goals from human demonstrations and train a supervised model to predict macro-goals according to the given lineups and game information . We derive diverse policies by sampling macro-goals from the model , and incorporating the goals and an intrinsic reward into the original actor-critic framework to guide the policy learning process . We use the famous mobile 5v5 MOBA game , Honor of Kings3 , as our testbed , which is widely used for MOBA-game AI research Ye et al . [ 2020c , a , b ] ; Wu [ 2019 ] ; Chen et al . [ 2021 ] ; Wei et al . [ 2021 ] . Experimental results demonstrate that the AI agent we obtained can adopt different policies according to the given lineups and improve the Elo score by modelling policy diversities . Our contribution can be concluded as follows : • To the best of our knowledge , we are the first to investigate the problem of policy diversity in MOBA games . We propose a novel framework that can learn diverse policies to adapt to different matches and lineups . • Based on macro-goals , our methods can explore diverse macro-strategies efficiently and meanwhile preserve high-level micro-operations . Unlike league-based methods , our MGG framework uses the same computation resources as the original actor-critic framework . • We conduct performance testing and diversity analysis on the full hero pool , i.e. , 102 heroes . Experimental results show that our agents can outperform existing state-of-the-art MOBA-game AI systems and demonstrate diverse policies . 2 Related Works . 2.1 Artificial Intelligence in MOBA Games . The progress of Game AI in MOBA mainly focuses on Dota 2 and Honor of Kings . OpenAI et al . [ 2019 ] have successfully defeated top e-sports players on playing 1v1 and 5v5 games in Dota 2 within limited heroes . Ye et al . [ 2020b , c , a ] have proposed a series of works that exceed humanlevel performance and expanded hero pools in Honor of Kings . Besides , TencentHMS Wu [ 2019 ] exploits a hierarchical policy structure to control macro-strategy decisions and micro-level executions , 2https : //www.reddit.com/r/MachineLearning/comments/bfq8v9/ 3https : //en.wikipedia.org/wiki/Honor_of_Kings respectively . All the mentioned works except the supervised learning adopt a single strategy regardless of the heroes and lineups , which can not take full advantage of heroes and lineups ’ characteristics . 2.2 Diverse Policies in Other Games . The league-based methods Vinyals et al . [ 2019 ] ; Balduzzi et al . [ 2019 ] ; Jaderberg et al . [ 2017 , 2019 ] train multiple agents simultaneously to explore strategy space and require massive computation resources . However , it is impractical to use league-based methods in MOBA games . Because the space of strategies is complex in that there are 213 , 610 , 453 , 056 ( C1040 × C510 ) lineups for 40 heroes in MOBA games and each lineup has various strategies . Therefore , it is difficult to design artificial signal to represent macro-strategies in MOBA games like Starcraft II ( e.g. , building order ) . The aforementioned Game AIs design methods suffer from the following limitations : 1 ) the need for data on human behaviors ; 2 ) heavy dependence of designer expert knowledge and substantial labor costs in searching a desirable behavior . Shen et al . [ 2020 ] can automatically generate diverse styles with almost no domain knowledge . However , this method is not suitable for complex games that last up to an hour long and consist of thousands of actions , such as MOBA games and RTS games . 2.3 Imitation Learning . In order to reduce the cost of strategy exploration in enormous lineups , we imitate macro-strategies from human demonstrations . Actually some works have trained reward functions from human preferences Nair et al . [ 2018 ] ; Ibarz et al . [ 2018 ] ; Christiano et al . [ 2017 ] or learnt policy directly from limited human data Ho and Ermon [ 2016 ] . Different from the above works , we abstract strategy as macro-goals in MOBA games . Our method not only imitates macro-strategies , but also leaves the agent to explore the optimal micro-operations ( e.g. , move , attack ) by itself through external rewards . 2.4 Goal-based Learning . Goal-based methods Schaul et al . [ 2015 ] ; Andrychowicz et al . [ 2017 ] ; Zhu et al . [ 2021 ] are originally proposed to solve the problem of training a single goal-conditioned policy to accomplish multiple tasks . Some methods refer to the idea of goal-based RL and divide a task into a series of sub-tasks ( i.e. , goal ) to accelerate the exploration and training of policy , such as automatic goal generation Florensa et al . [ 2018 ] and hierarchical RL Kulkarni et al . [ 2016 ] ; Nachum et al . [ 2018 ] , etc . However , in complex scenarios such as MOBA or RTS games , it is difficult to explore the meaningful goals , whether using GAN or training a high-level policy by RL methods . Therefore , we utilize the top players ’ demonstrations to pre-train a macro-level policy to generate reasonable and diverse macro-goals . 3 Macro-Goals Guided Reinforcement Learning . The problem we focused on can be formulated as a Markov Decision Process ( MDP ) , defined by a tuple ( S , G , A , R , T , γ ) , where S is the state space , A is the action space , T : S × A → S is the transition function , R : S × A → R is the reward function , and γ ∈ [ 0 , 1 ] is the discount factor . In a typical MOBA game Silva and Chaimowicz [ 2017 ] , the agents perform actions a ∈ A ( e.g . move , attack ) according to the observations s ∈ S ( e.g . unit states , in-game stats ) to achieve the final objective ( e.g . destroy the crystal ) . In our setting , G represents the macro-strategic state space in the game . And we define a macro-state extraction function f : S → G which maps the current state s to the multi-dimensional macro-state c = f ( s ) . The macro-state c ∈ G mainly includes the region state ( e.g . the area on the map ) and resource state ( e.g . unit resource , gold , level ) . The macro-goal g ∈ G represents the macro-state that the agent should achieve in the near future in a certain strategy . Ye et al . [ 2020a ] design the general reward function for MOBA games , including farming , KDA , damage , pushing , and win/lose related rewards . As a result of the single value system , the agent only learns to execute a single strategy from self-play training , that is , farming first , then fighting and pushing , no matter what strategy the opponents adopt . Thus , its macro-state set { c|c ∈ GRL } is only a part of G , i.e . GRL ⊂ G. On the contrary , human strategies are diverse and their macro-state set Ghuman can be considered to be close to the oracle G , i.e . Ghuman = G1 ∪ . . . Gi · · · ∪GN ≈ G , where N is the number of human strategies . We define that strategy i can be expressed as a set of macro-goal { g|g ∈ Gi } , where Gi ⊂ G. Our objective is to learn diverse strategies , that is , maximize the macro-state entropy − ∑ c∈G P ( c ) logP ( c ) . Therefore , we model the policy as π ( a|s , g ) and utilize g ∈ Ghuman ≈ G as the macro-goal of the policy . We also introduce an intrinsic reward function Rgoal : S ×G×A → R to guide its macro-state set GRL to be close to the complete G. In this paper , we propose the Macro-Goals Guided ( MGG ) framework to learn diverse policies in MOBA games . Intuitively , top players ’ matches are observed to have high-quality and diverse macro-strategies but noisy micro-operations . Thus , completely imitating human demonstrations , including micro-operations , will worsen the performance of the policy . However , the macro-strategies from top players ’ matches can be used to imitate , which can reduce the cost of exploration in training . Our motivation is to define macro-goals to depict human strategies and then learn diverse strategies from these macro-goals only . In general , the MGG consists of three stages , as shown in Figure 1 . Stage 1 : Extract state sDt and its corresponding macro-goal g D from top e-sports demonstrations . Stage 2 : Use the extracted state and macro-goal pairs ⟨sDt , gD⟩ as the training data to train the Meta-Controller parameterized by ϕ , via supervised learning . During the training process , we use the auxiliary auto-encoding tasks Taux of lineups to improve prediction accuracy . Given a state s , the Meta-Controller ϕ should predict a macro-goal g to depict the current strategy . Stage 3 : Use the Dual-clip PPO algorithm Ye et al . [ 2020a ] to train the policy and value network θ conditioned on the macro-goal g. The macro-goals g of the specified strategy is sampled from the pre-trained Meta-Controller . We introduce an intrinsic reward Rgoalt to denote whether the agent achieves the macro-goal g. Besides , we use the multi-head value mechanism Ye et al . [ 2020a ] to combine the original value output and the intrinsic value output . To better illustrate the MGG method , we use Honor of Kings as a case study to describe some common terms associated with MOBA games . But it can be applied to other MOBA games ( e.g . Dota 2 and League of Legends ) , as the playing mechanism across MOBA games are similar .

Summary: The authors propose a method for using limited human data to generate macro-goals that serve as an auxiliary imitation loss and reward for deep reinforcement learning agents playing a mobile 5v5 MOBA game (Honor of Kings). The proposed approach provides 'macro' supervision for agent behavior while letting the agent learn the finer details and micro actions to reach goals or maximize game reward. Experiments compare the proposed approach to a variety of baselines and indicate higher ELO scores and lower exploitability by human players.