Efficient meta reinforcement learning via meta goal generation

Meta reinforcement learning ( meta-RL ) is able to accelerate the acquisition of new tasks by learning from past experience . Current meta-RL methods usually learn to adapt to new tasks by directly optimizing the parameters of policies over primitive actions . However , for complex tasks which requires sophisticated control strategies , it would be quite inefficient to directly learn such a meta-policy . Moreover , this problem can become more severe and even fail in spare reward settings , which is quite common in practice . To this end , we propose a new meta-RL algorithm called meta goal-generation for hierarchical RL ( MGHRL ) by leveraging hierarchical actor-critic framework . Instead of directly generate policies over primitive actions for new tasks , MGHRL learns to generate high-level meta strategies over subgoals given past experience and leaves the rest of how to achieve subgoals as independent RL subtasks . Our empirical results on several challenging simulated robotics environments show that our method enables more efficient and effective meta-learning from past experience and outperforms state-of-the-art meta-RL and Hierarchical-RL methods in sparse reward settings . 1 INTRODUCTION . Deep Reinforcement Learning ( DRL ) has recently shown a great success on a wide range of tasks , ranging from games ( Mnih et al. , 2015 ) to robotics control ( Levine et al. , 2016 ; Bengio & LeCun , 2016 ) . However , for more complex problems with larger state and action spaces or sparse reward settings , traditional DRL methods hardly works . Hierarchical reinforcement learning ( HRL ) in which multiple layers of policies are trained to learn to operate on different levels of temporal abstraction , has long held the promise to learn such difficult tasks ( Dayan & Hinton , 1992 ; Parr & Russell , 1997 ; Barto & Mahadevan , 2003 ) . By decomposing a complex problem into subproblems , HRL significantly reduces the difficulty of solving specific task . Learning multiple levels of policies in parallel is challenging due to non-stationary state transition functions . Recent HRL approaches ( Nachum et al. , 2018 ; Levy et al. , 2019 ) use states as goals directly , allowing simple and fast training of the lower layer . Human intelligence is remarkable for their fast adaptation to many new situations using the knowledge learned from past experience . However , agents trained by conventional DRL methods mentioned above can only learn one separate policy per task , failing to generalize to new tasks without additional large amount of training data . Meta reinforcement learning ( meta-RL ) addresses such problems by learning how to learn . Given a number of tasks with similar structures , meta-RL methods enable agents learn such structure from previous experience on many tasks . Thus when encountering a new task , agents can quickly adapt to it with only a small amount of experience . Most current meta-RL methods leverage experience from previous tasks to adapt to new tasks by directly learn the policy parameters over primitive action space . ( Finn et al. , 2017 ; Rakelly et al. , 2019 ) . Such approaches suffer from two problems : ( i ) For complex tasks which requires sophisticated control strategies , it would be quite inefficient to directly learn such policy with one nonlinear function approximator and the adaptation to new tasks is prone to be inaccurate . This problem can become more severe in spare reward settings . ( ii ) When the task distribution is much wider ( riding bicycle as meta-train task and riding motorcycle as meta-test task ) , these methods can hardly be effective since primitive action execution mechanism is entirely different although they may share a similar high-level strategy . Moreover , existing current meta-RL methods perform badly in sparse reward settings , which are quite common in real world . In this paper , we aim at tackling the problems mentioned above by proposing an efficient hierarchical meta-RL method that realizes meta learning high-level goal generation and leaves the learning of low-level policy for independent RL . Intuitively , this is quite similar to how a human being behaves : we usually transfer the overall understanding of similar tasks rather than remember specific actions . Our meta goal-generation framework is built on top of the architecture of PEARL ( Rakelly et al. , 2019 ) and a two level hierarchy inspired by HAC ( Levy et al. , 2019 ) . Our evaluation on several simulated robotics environments ( Plappert et al. , 2018 ) shows the superiority of MGHRL to stateof-the-art meta-RL and hierarchical RL methods in sparse reward settings . Generally , our contributions are as follows : • We propose an algorithm that achieves efficient meta reinforcement learning on challenging robotics environments with sparse reward settings and outperforms other leading methods . • Similar to the way humans leverage past experience to learn new complex tasks , our algorithm focuses on meta learning the overall strategy for different tasks , which provides a much simpler and better way for meta RL comparing with directly learning the detailed solution . Since we focus on meta goal-generation and leave the low level policy for independent learning , we believe our algorithm can still accelerate the acquisition of new tasks sampled from much wider task distributions . For example , to learn tasks such as riding bicycles and riding a motorcycle , the two primitive action execution mechanism are entirely different but the two learning process still share similar high-level structures . Through meta goal-generation learning , we expect our method can still accelerate the acquisition of such tasks . We leave these for future work to explore . 2 RELATED WORK . Our algorithm is based on meta learning framework ( Thrun & Pratt , 1998 ; Schmidhuber , 1987 ; Bengio et al. , 1991 ) , which aims to learn models that can adapt quickly to new tasks . Meta learning algorithms for few-shot supervised learning problems have explored a wide variety of approaches and architectures ( Santoro et al. , 2016 ; Vinyals et al. , 2016 ; Ravi & Larochelle , 2017 ) . In the context of reinforcement learning , recurrent ( Duan et al. , 2016 ; Wang et al. , 2016 ) and recursive ( Mishra et al. , 2018 ) meta-RL methods adapt to new tasks by aggregating experience into a latent representation on which the policy is conditioned . Another set of methods is gradient-based meta reinforcement learning ( Finn et al. , 2017 ; Stadie et al. , 2018 ; Rothfuss et al. , 2019 ; Xu et al. , 2018 ) . Its objective is to learn an initialization such that after one or few steps of policy gradients the agent attains full performance on a new task . These methods focus on on-policy meta learning which are usually sample inefficient . Our algorithm is closely related to probabilistic embeddings for actor-critic RL ( PEARL ) ( Rakelly et al. , 2019 ) , which is an off-policy meta RL algorithm . PEARL leverages posterior sampling to decouple the problems of inferring the task and solving it , which greatly enhances meta-learning efficiency . However , when facing complex tasks that require sophisticated control strategies , PEARL can not effectively learn a proper meta-policy as we will show in Section 5 . Discovering meaningful and effective hierarchical policies is a longstanding research problem in RL ( Dayan & Hinton , 1992 ; Parr & Russell , 1997 ; Sutton et al. , 1999 ; Bacon et al. , 2017 ; Dietterich , 2000 ) . Schmidhuber 1987 proposed a HRL approach that can support multiple levels . Multi-level hierarchies have the potential to accelerate learning in sparse reward tasks because they can divide a problem into a set of short-horizon subproblems . Nachum et al . 2018 proposed HIRO , a 2-level HRL approach that can learn off-policy and outperforms two other popular HRL techniques used in continuous domains : Option-Critic ( Bacon et al. , 2017 ) and FeUdal Networks ( Vezhnevets et al. , 2017 ) . Our algorithm is built on Hierarchical actor-critic ( Levy et al. , 2019 ) , which is a framework that can learn multiple levels of policies in parallel . Most current HRL works focus on the learning problem in a single task and few of them considers to take advantage of HRL for multi-task or metalearning tasks . MLSH ( Frans et al . ) is such a work which also combines meta-RL with Hierarchical RL . It focuses on meta learning on the low level policy and need to retrain its high level policy when facing new tasks . In contrast , with the key insight that humans leverage abstracted prior knowledge obtained from past experience , our method focus on meta learning high level overall strategy using past experience and leave the detailed action execution for independent RL . 3 BACKGROUND . 3.1 META REINFORCEMENT LEARNING . In our meta learning scenario , we assume a distribution of tasks p ( τ ) that we want our model to adapt to . Each task correspond to a different Markov decision process ( MDP ) , Mi = { S , A , Ti , Ri } , with state space S , action space A , transition distribution Ti , and reward function Ri . We assume that the transitions and reward function vary across tasks . Meta RL aims to learn a policy that can adapt to maximize the expected reward for novel tasks from p ( τ ) as efficiently as possible . PEARL ( Rakelly et al. , 2019 ) is an off-policy meta-reinforcement learning method that drastically improves sample efficiency comparing to previous meta-RL algorithms . The meta-training process of PEARL learns a policy that adapts to the task at hand by conditioning the history of past transitions , which we refer to as context c. Specifically , for the ith transition in task τ , cτi = ( si , ai , ri , s ′ i ) . PEARL leverages an inference network qφ ( z|c ) and outputs probabilistic latent variable z . The parameters of q ( z|c ) are optimized jointly with the parameters of the actor πθ ( a|s , z ) and critic Qθ ( s , a , z ) , using the reparametrization trick ( Kingma & Welling , 2014 ) to compute gradients for parameters of qφ ( z|c ) through sampled z ’ s . 3.2 HIERARCHICAL ACTOR-CRITIC . HAC ( Levy et al. , 2019 ) aims to accelerate learning by enabling hierarchical agents to jointly learn a multi-level hierarchy of policies in parallel . HAC is comprised of two components : a particular hierarchical architecture and a method for learning the multiple levels of policies in parallel given sparse rewards . The hierarchies produced by HAC have a specific architecture consisting of a set of nested , goal-conditioned policies that use the state space as the mechanism for breaking down a task into subtasks . HAC extends the idea of Hindsight Experience Replay ( Andrychowicz et al. , 2017 ) by creating two types of hindsight transitions . Hindsight action transition simulates a transition function that uses the optimal low level policy while hindsight goal transition use the final states achieved as the goal state in each step ’ s transition . They enable agents to learn multiple policies in parallel using only sparse reward functions . 4 ALGORITHM . 4.1 TWO-LEVEL HIERARCHY . We set up a hierarchical two-layer RL structure similar to HAC . The high level network uses policy µh to generate goals for temporally extended periods in terms of desired observations . In our task they correspond to the positional features of the gripper . The low level policy µl directly controls the agent and produces actions for moving towards the desired goals . As shown in Figure 1 ( a ) , the high level policy µh observes the state and produces a high level action ( or goal ) gt . Low level policy µl has at most K attempts of primitive action to achieve gt . Here , K which can be viewed as the maximum horizon of a subgoal action is a hyperparameter given by the user . As long as the low level policy µl run out of K attempts or gt is achieved , this high level transition terminates . The high level policy uses agent ’ s current state as the new observation and produced another goal for low level policy to achieve . We use an intrinsic reward function in which a reward of 0 is granted only if the goal produced by high level policy is achieved and a reward of -1 otherwise . Note that the environment ’ s return ( i.e . whether the agent successfully accomplished the task ) will not affect the reward received by the low level policy . In our evaluation on simulated robotics environments , we use the positional features of the observations as the representation for gt . A goal gt is judged to be achieved only if the distance between gt and the gripper ’ s current position sn+1 is less than threshold l .

Summary: In this paper, the authors focus on the problem of meta-reinforcement learning (meta-RL). Specifically, the authors consider the setting of meta-RL for goal reaching tasks where each task corresponds to an unknown goal. Existing meta-RL algorithms directly train for a policy that output low level actions, which might be inefficient in this goal-reaching setting. In this paper, the authors combine the hierarchical RL framework of HAC[1] with the probabilistic task context inference method of PEARL[2], and propose the meta-goal generation for hierarchical RL (MGHRL) algorithm. In this algorithm, a two layer hierarchical policy is used where the high level policy generate goals for the low level goal-reaching policy to reach. In order to adapt to an unknown goal, the high level policy is conditioned on the output of a task inference module to generate goals for the unknown ground truth goal. The goal-reaching policy would then use the generated goal to interact with the environment.