Hierarchical Meta Reinforcement Learning for Multi-Task Environments

1 INTRODUCTION . With great breakthrough of deep reinforcement learning ( DRL ) methods ( Mnih et al. , 2015 ; Silver et al. , 2016 ; Mnih et al. , 2016 ; Schulman et al. , 2015 ; Lillicrap et al. , 2015 ) , it is an urgent need to use DRL methods to solve more complex decision-making problems . The practical problem in real world is often a subtle combination of multiple sub-tasks , which may happen simultaneously and hard to disentangle by time series . For instance , in StarCraft games ( Pang et al. , 2019 ) , agents need to consider building units and organizing battles , sub-tasks may change rapidly over the whole game process ; sweeping robots tradeoff between navigating and collecting garbage ; shooting agents should move to appropriate positions and launch attacks , etc . The relationship between sub-tasks is complex and subtle . Sometimes they compete with each other and need to focus on one task to gain key advantages ; at other times , they need to cooperate with each other to maintain the possibility of global exploration . It is often time consuming and ineffective to learn simply by collecting experience and rewarding multiple objectives for different sub-tasks . A reasonable idea is to utilize deep hierarchical reinforcement learning ( DHRL ) methods ( Vezhnevets et al. , 2017 ; Igl et al. , 2020 ) , where the whole system is divided into a high-level agent and several low-level agents . Low-level agents learn sub-policies , which select atomic actions for corresponding sub-tasks . The high-level agent is responsible for a meta task in the abstract logic or coarser time granularity , guiding low-level agents by giving a goal , or directly selecting among subpolicies . However , DHRL methods face some inherent problems : due to the complex interaction between multi-level agents , there is no theoretical guarantee of convergence , and it shows unstable experimental performance . Most DHRL methods require heavy manual design , and end-to-end system lacks reasonable semantic interpretation . In addition , these agents are often constrained by specific tasks and are easy to overfit . Even transferring between similar tasks , they perform poorly and need a lot of additional adjustments . We introduce a first-person shooting ( FPS ) environment with random spatial structures . The game contains a 3D scene from human perspective . When the player defeats all enemies , the player wins the game . When the player drops to the ground or losses all health points , the player loses the game . It is very risky for the player to drop to the ground , thus environment contains two key tasks : navigation and combat . The terrain and enemies in the game are randomly generated . This ensures : 1 ) the agent can not learn useful information by memorizing coordinates ; 2 ) the possibility of overfitting is restrained and the generalization ability of learned policy is enhanced . The state information is expressed in the way of raycast . This representation of environment information requires much less computing resources than the raw image representation . It can be trained and tested even with only CPU machines , which makes us pay more attention to the reinforcement learning algorithm itself rather than the computing ablity related to image processing . For this environment , we propose a Meta Soft Hierarchical reinforcement learning framework ( MeSH ) . The high-level policy is a differentiable meta parameter generator , and the low-level policy contains several sub-policies , which are in the same form and differentiated automatically in the training procedure . The high-level policy selects and combines sub-policies through the meta parameter and interacts with the environment . We find that the meta generator can adaptively combines sub-policies with the process of the task , and have strong interpretability in semantics . Compared with a series of baselines , the agent has achieved excellent performance in FPS environment . The main contributions of this work are as follows : • clarifying the complex relationship between multi-task composition . • a novel meta soft hierarchical reinforcement learning framework , MeSH , which uses differentiable meta generator to adaptively select sub-policies and shows strong interpretability . • a series of comparison experiments to show the effectiveness of the framework . • an open-sourced environment and code to encourage further research on multi-task RL 1 . In this paper , we discuss the related work in Section 2 . We introduce the details of the implemented environment in Section 3 . We show our proposed framework in Section 4 . We demonstrate details of our experiments in Section 5 . At last , we conclude in Section 6 . 2 RELATED WORK . In decision-making problems with high-dimensional continuous state space , the agent often needs to complete tasks that contain multiple sub-tasks . To complete taxi agent problem ( Dietterich , 2000 ) , the agent needs to complete sub-tasks such as pickup , navigate , putdown . Menashe & Stone ( 2018 ) proposed Escape Room Domain , which is a testbed for HRL . The agent leaves the room from the starting point and needs to press four buttons of different colors to leave the room . In these environments , the agent needs to optimize several sub-tasks and minimize the mutual negative influence between them . However , sub-tasks in these environments are timing dependent . The proposed methods above are helpless in a multi-task environment that needs to fulfill multiple tasks simultaneously . Architectural solutions use hierarchical structure to decompose tasks into action primitives . Sutton et al . ( 1999 ) models temporal abstraction as an option on top of extended actions , Bacon et al . ( 2017 ) proposes an actor-critic option method based on it . Henderson et al . ( 2017 ) extend the options framework to learn joint reward-policy options . Besides , Jiang et al . ( 2019 ) construct a compositional structure with languages as abstraction or instruction . Due to specific structure design of these methods , high-level agent is unable to execute multiple sub-policies simultaneously in any form . Recent HRL works learn intra-goals to instruct sub-policies . Vezhnevets et al . ( 2017 ) proposes a manager-worker model , manager abstracts goals and instructs worker . This architecture uses directional goal rather than absolute update goal . Oh et al . ( 2017 ) learns a meta controller to instruct implementation and update of sub-tasks . Igl et al . ( 2020 ) presents a new soft hierarchy method based on option , it learns with shared prior and hierarchical posterior policies . Yu et al . ( 2020 ) proposes a method that projects the conflict gradient onto the normal plane to avoid some task gradients dominating others . Compared with the hard hierarchy methods , these methods use the state ’ s natural features to update the upper-level policy , avoiding the timing constraints of handcrafted sub-tasks . Due to the lack of meaningful learning goals of sub-policies , the low-level policies fail to focus on explainable sub-tasks . 1https : //github.com/MeSH-ICLR/MEtaSoftHierarchy.git HRL has been recently combined with meta-learning . Frans et al proposes Meta-Learning Shared Hierarchies ( MLSH ) ( Frans et al. , 2017 ) , which updates master policy and sub-policy by metalearning . Zou et al . ( 2019 ) uses meta-learning to learn reward shaping . Rakelly et al . ( 2019 ) extends the idea of disentangling task inference and control to help agents leverage knowledges between tasks . Although these methods reduced the limitation of policy structures , it is difficult to learn multi-tasks in parallel due to the fixed time steps selected by the master policy for every sub-policy . It is also reasonable to learn sub-tasks based on decomposition of rewards . Saxe et al . ( 2017 ) expresses behaviors as different weighted task blends by the communication between layers so that can maintain a parallel distributed representation of tasks , but the structure is hard to abstract the relationship between different sub-tasks . Grimm & Singh ( 2019 ) and Lin et al . ( 2019 ) allocate each state with one sub-reward function , which is a non-overlapping decomposition of spatial space . In contrast to these approaches , our method can represent complex and subtle relationship between multiple sub-tasks and perform them simultaneously with explainable sub-goals . 3 ENVIRONMENT . To better understand multi-task decision-making problems , we firstly introduce a first-person shooting environment with random terrain , as shown in Figure 1 . The game contains a 3D scene , which analogy to human perspective . This makes the behavior of trained agent similar to human intelligence . The real-time information is shown in Table 1 . The condition of winning the game is to defeat all the enemies in the game . When the player drops to the ground or losses all health points , the game will be judged as failed . This game is very risky as it ’ s easy to drop . Thus the environment contains two key tasks : navigation and combat . The terrain and enemies in the game are randomly generated . This ensures : 1 ) the agent can not learn useful information by memorizing coordinates ; 2 ) the possibility of over-fitting is restrained and the generalization ability of learned policy is enhanced . The state information of the agent is expressed in the way of raycast , as shown in Figure 2 . This representation of environment information requires much less computing resources than raw image representation . It can be trained and tested even with only CPU machines , which makes us pay more attention to the reinforcement learning algorithm itself rather than the computing ability related to image processing . The generation rules of random terrain are as follows . The maximum generated height of random terrain is set to 5 . In the initial state , there is only one parcel , which is also the place where the player is born . We add this parcel to the parcel queue . If the parcel queue is not empty , the parcel at the head of the queue will be taken out . When the maximum height is not reached , we expand the parcel to four directions of the higher level with equal probability . If there is no new parcel in the corresponding position of the higher level , a new parcel is generated . A ramp is established between the new parcel and the current parcel and a random number of enemy is created at the random position of the new parcel , and then the new parcel is added to the parcel queue . If there are new parcels in the corresponding position of the higher level , the new parcels are not generated repeatedly , and only the ramp between the two parcels is added . Repeat adding parcels until the parcel queue is empty , then terrain generation is completed . The FPS environment is typically a combination of two sub-tasks : navigation and combat . The relationship between these two sub-tasks is subtle and complex . Sometimes they compete with each other , but at other times , they cooperate for a common objective . For instance , in navigation missions , in order to explore more unseen terrain , sometimes we need to fight to clear enemies along the way ; but at other times , we have to focus on navigation to pass through narrow terrain . Similarly , in combat missions , sometimes we need to move to get a better position to shoot ; but at other times , we need to focus on shooting to kill the enemy quickly . This environment is very representative of practical problems , since a large number of them can be divided into several parts which are contradictory and unified . In addition , each sub-task in this environment is simple and clear , but the combination of them greatly increases the difficulty of solving the problem . This forces the RL algorithm to focus more on dealing with these complex relationships , rather than the specific techniques for solving a single problem .

Summary: This paper considers a FPS game that can be decomposed into two sub-tasks, navigation and shooting. A hierarchical meta RL method is introduced and the updating rules for sub-policies and meta parameters are provided. Experiments focus on this specific environment and hence the hierarchical structure is also specified as a meta controller over two sub-policies defined for navigation and shooting explicitly.