Learning Diverse Policies in MOBA Games via Macro-Goals

1 Introduction . As a stepping stone of Artificial General Intelligence ( AGI ) , Game AI has achieved significant progress in the past decade , including board games Silver et al . [ 2016 ] , poker games Brown et al . [ 2019 ] ; Moravčík et al . [ 2017 ] , Real-time Strategy ( RTS ) games Vinyals et al . [ 2019 ] , etc . Especially , Multi-player Online Battle Arena ( MOBA ) games have been regarded as a challenging problem in Game AI and attracted massive attention from research communities because of the complicated mechanisms such as imperfect information , large state-action space , and long-term sequential decision making . Recently , OpenAI et al . [ 2019 ] develop the AI system on Dota 2 , named OpenAI Five , which defeats the Dota 2 world champion , i.e. , Team OG , within a limited size of hero pool . It trains with deep reinforcement learning and self-play . Ye et al . [ 2020a ] build the AI system on another popular MOBA game , Honor of Kings , which also defeats top e-sports players . Besides , compared with OpenAI Five , their AI system enables playing full MOBA games , which extends the hero pool size from 17 to 40 heroes , adds ban/pick capabilities , and addresses the scalability issue . Even though existing AI systems have achieved or even exceeded human-level performance in MOBA games , they still suffer from the lack of policy diversity . Agents during training are controlled by the same hand-crafted reward signal and then present similar strategies even in different matches , while strategies played by humans usually vary according to situations in the game . One of the disadvantages is that we can counter the single learned policy after several attempts effortlessly . For example , one team defeats OpenAI Five nine times with the same strategy in public tests1 . The 1https : //openai.com/projects/five/ 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . primary counter-strategy is to avoid fights as much as possible and instead push outside lanes to pressure the agents into defending2 . Another disadvantage is that agents with a single policy can not exploit the strength of heroes and lineups . To interest players , MOBA game designers usually make different skill mechanisms for different heroes , which result in heroes play different roles in the game . Therefore , different roles in lineups have different strategies to maximize a team ’ s utility . For example , the supporter should keep their allies alive and give them opportunities to earn more gold and experience . The assassin can efficiently jungle neutrals rather than farm lanes . Moreover , different combinations of heroes construct lineups with different characteristics . Some lineups have advantages in pushing with a quick pace , while some lineups are good at the development of carry heroes . The agent should find a way to maximize the picked lineup ’ s strength instead of playing with a single policy in the game . Although existing methods can learn diverse policies in other games , it remains to be a grand challenge in MOBA games . AlphaStar Vinyals et al . [ 2019 ] uses a framework based on the league to improve the diversity of policies in StarCraft II . The historical models are used to construct the league and train exploiters to highlight flaws in the league and main agents . The main agents are forced to discover new strategies by training against the exploiters . They also define a latent variable according to building orders in human data as constraints of self-play . However , the league-based methods can not be applied to MOBA games directly . First , they consume massive computation resources to construct the league . They use 12 copies of models in which there exist 128 TPU cores . Second , the scenarios between MOBA and RTS games are different . They leverage the nature of StarCraft II and use building orders to represent each strategy , which is unavailable in MOBA games . In this paper , we propose a novel learning paradigm named Macro-Goals Guided ( MGG ) learning to train diverse policies in MOBA games . Motivated by top e-sports players , we define related game information as macro-goals to depict strategic decisions , which constitute macro-strategies in MOBA games . To explore macro-strategies efficiently , we extract macro-goals from human demonstrations and train a supervised model to predict macro-goals according to the given lineups and game information . We derive diverse policies by sampling macro-goals from the model , and incorporating the goals and an intrinsic reward into the original actor-critic framework to guide the policy learning process . We use the famous mobile 5v5 MOBA game , Honor of Kings3 , as our testbed , which is widely used for MOBA-game AI research Ye et al . [ 2020c , a , b ] ; Wu [ 2019 ] ; Chen et al . [ 2021 ] ; Wei et al . [ 2021 ] . Experimental results demonstrate that the AI agent we obtained can adopt different policies according to the given lineups and improve the Elo score by modelling policy diversities . Our contribution can be concluded as follows : • To the best of our knowledge , we are the first to investigate the problem of policy diversity in MOBA games . We propose a novel framework that can learn diverse policies to adapt to different matches and lineups . • Based on macro-goals , our methods can explore diverse macro-strategies efficiently and meanwhile preserve high-level micro-operations . Unlike league-based methods , our MGG framework uses the same computation resources as the original actor-critic framework . • We conduct performance testing and diversity analysis on the full hero pool , i.e. , 102 heroes . Experimental results show that our agents can outperform existing state-of-the-art MOBA-game AI systems and demonstrate diverse policies . 2 Related Works . 2.1 Artificial Intelligence in MOBA Games . The progress of Game AI in MOBA mainly focuses on Dota 2 and Honor of Kings . OpenAI et al . [ 2019 ] have successfully defeated top e-sports players on playing 1v1 and 5v5 games in Dota 2 within limited heroes . Ye et al . [ 2020b , c , a ] have proposed a series of works that exceed humanlevel performance and expanded hero pools in Honor of Kings . Besides , TencentHMS Wu [ 2019 ] exploits a hierarchical policy structure to control macro-strategy decisions and micro-level executions , 2https : //www.reddit.com/r/MachineLearning/comments/bfq8v9/ 3https : //en.wikipedia.org/wiki/Honor_of_Kings respectively . All the mentioned works except the supervised learning adopt a single strategy regardless of the heroes and lineups , which can not take full advantage of heroes and lineups ’ characteristics . 2.2 Diverse Policies in Other Games . The league-based methods Vinyals et al . [ 2019 ] ; Balduzzi et al . [ 2019 ] ; Jaderberg et al . [ 2017 , 2019 ] train multiple agents simultaneously to explore strategy space and require massive computation resources . However , it is impractical to use league-based methods in MOBA games . Because the space of strategies is complex in that there are 213 , 610 , 453 , 056 ( C1040 × C510 ) lineups for 40 heroes in MOBA games and each lineup has various strategies . Therefore , it is difficult to design artificial signal to represent macro-strategies in MOBA games like Starcraft II ( e.g. , building order ) . The aforementioned Game AIs design methods suffer from the following limitations : 1 ) the need for data on human behaviors ; 2 ) heavy dependence of designer expert knowledge and substantial labor costs in searching a desirable behavior . Shen et al . [ 2020 ] can automatically generate diverse styles with almost no domain knowledge . However , this method is not suitable for complex games that last up to an hour long and consist of thousands of actions , such as MOBA games and RTS games . 2.3 Imitation Learning . In order to reduce the cost of strategy exploration in enormous lineups , we imitate macro-strategies from human demonstrations . Actually some works have trained reward functions from human preferences Nair et al . [ 2018 ] ; Ibarz et al . [ 2018 ] ; Christiano et al . [ 2017 ] or learnt policy directly from limited human data Ho and Ermon [ 2016 ] . Different from the above works , we abstract strategy as macro-goals in MOBA games . Our method not only imitates macro-strategies , but also leaves the agent to explore the optimal micro-operations ( e.g. , move , attack ) by itself through external rewards . 2.4 Goal-based Learning . Goal-based methods Schaul et al . [ 2015 ] ; Andrychowicz et al . [ 2017 ] ; Zhu et al . [ 2021 ] are originally proposed to solve the problem of training a single goal-conditioned policy to accomplish multiple tasks . Some methods refer to the idea of goal-based RL and divide a task into a series of sub-tasks ( i.e. , goal ) to accelerate the exploration and training of policy , such as automatic goal generation Florensa et al . [ 2018 ] and hierarchical RL Kulkarni et al . [ 2016 ] ; Nachum et al . [ 2018 ] , etc . However , in complex scenarios such as MOBA or RTS games , it is difficult to explore the meaningful goals , whether using GAN or training a high-level policy by RL methods . Therefore , we utilize the top players ’ demonstrations to pre-train a macro-level policy to generate reasonable and diverse macro-goals . 3 Macro-Goals Guided Reinforcement Learning . The problem we focused on can be formulated as a Markov Decision Process ( MDP ) , defined by a tuple ( S , G , A , R , T , γ ) , where S is the state space , A is the action space , T : S × A → S is the transition function , R : S × A → R is the reward function , and γ ∈ [ 0 , 1 ] is the discount factor . In a typical MOBA game Silva and Chaimowicz [ 2017 ] , the agents perform actions a ∈ A ( e.g . move , attack ) according to the observations s ∈ S ( e.g . unit states , in-game stats ) to achieve the final objective ( e.g . destroy the crystal ) . In our setting , G represents the macro-strategic state space in the game . And we define a macro-state extraction function f : S → G which maps the current state s to the multi-dimensional macro-state c = f ( s ) . The macro-state c ∈ G mainly includes the region state ( e.g . the area on the map ) and resource state ( e.g . unit resource , gold , level ) . The macro-goal g ∈ G represents the macro-state that the agent should achieve in the near future in a certain strategy . Ye et al . [ 2020a ] design the general reward function for MOBA games , including farming , KDA , damage , pushing , and win/lose related rewards . As a result of the single value system , the agent only learns to execute a single strategy from self-play training , that is , farming first , then fighting and pushing , no matter what strategy the opponents adopt . Thus , its macro-state set { c|c ∈ GRL } is only a part of G , i.e . GRL ⊂ G. On the contrary , human strategies are diverse and their macro-state set Ghuman can be considered to be close to the oracle G , i.e . Ghuman = G1 ∪ . . . Gi · · · ∪GN ≈ G , where N is the number of human strategies . We define that strategy i can be expressed as a set of macro-goal { g|g ∈ Gi } , where Gi ⊂ G. Our objective is to learn diverse strategies , that is , maximize the macro-state entropy − ∑ c∈G P ( c ) logP ( c ) . Therefore , we model the policy as π ( a|s , g ) and utilize g ∈ Ghuman ≈ G as the macro-goal of the policy . We also introduce an intrinsic reward function Rgoal : S ×G×A → R to guide its macro-state set GRL to be close to the complete G. In this paper , we propose the Macro-Goals Guided ( MGG ) framework to learn diverse policies in MOBA games . Intuitively , top players ’ matches are observed to have high-quality and diverse macro-strategies but noisy micro-operations . Thus , completely imitating human demonstrations , including micro-operations , will worsen the performance of the policy . However , the macro-strategies from top players ’ matches can be used to imitate , which can reduce the cost of exploration in training . Our motivation is to define macro-goals to depict human strategies and then learn diverse strategies from these macro-goals only . In general , the MGG consists of three stages , as shown in Figure 1 . Stage 1 : Extract state sDt and its corresponding macro-goal g D from top e-sports demonstrations . Stage 2 : Use the extracted state and macro-goal pairs ⟨sDt , gD⟩ as the training data to train the Meta-Controller parameterized by ϕ , via supervised learning . During the training process , we use the auxiliary auto-encoding tasks Taux of lineups to improve prediction accuracy . Given a state s , the Meta-Controller ϕ should predict a macro-goal g to depict the current strategy . Stage 3 : Use the Dual-clip PPO algorithm Ye et al . [ 2020a ] to train the policy and value network θ conditioned on the macro-goal g. The macro-goals g of the specified strategy is sampled from the pre-trained Meta-Controller . We introduce an intrinsic reward Rgoalt to denote whether the agent achieves the macro-goal g. Besides , we use the multi-head value mechanism Ye et al . [ 2020a ] to combine the original value output and the intrinsic value output . To better illustrate the MGG method , we use Honor of Kings as a case study to describe some common terms associated with MOBA games . But it can be applied to other MOBA games ( e.g . Dota 2 and League of Legends ) , as the playing mechanism across MOBA games are similar .

Summary: The paper addresses RL agent training in MOBA games, specifically the ability to acquire a diverse set of policies when playing a variety of strategic scenarios (team compositions, aka lineups). Human example data is processes to extract future states reached from initial states, treating these future states as goals. A supervised learning model is trained to predict the future goal state from a given game state using an attention module that accounts for the strategic scenario (lineup) and game state (unit features & aggregate game stats). An existing RL framework is then augmented to condition on this goal state and trained with a reward that encourages reducing the distance to this goal in addition to the baseline reward from the environment. A wide variety of results show the method is better able to adapt to unique strategic scenarios (lineups) and exhibits a broader array of behaviors compared to the baseline agent or an imitation learning agent (GAIL) while retaining strong performance against human players.