Efficient meta reinforcement learning via meta goal generation

Meta reinforcement learning ( meta-RL ) is able to accelerate the acquisition of new tasks by learning from past experience . Current meta-RL methods usually learn to adapt to new tasks by directly optimizing the parameters of policies over primitive actions . However , for complex tasks which requires sophisticated control strategies , it would be quite inefficient to directly learn such a meta-policy . Moreover , this problem can become more severe and even fail in spare reward settings , which is quite common in practice . To this end , we propose a new meta-RL algorithm called meta goal-generation for hierarchical RL ( MGHRL ) by leveraging hierarchical actor-critic framework . Instead of directly generate policies over primitive actions for new tasks , MGHRL learns to generate high-level meta strategies over subgoals given past experience and leaves the rest of how to achieve subgoals as independent RL subtasks . Our empirical results on several challenging simulated robotics environments show that our method enables more efficient and effective meta-learning from past experience and outperforms state-of-the-art meta-RL and Hierarchical-RL methods in sparse reward settings . 1 INTRODUCTION . Deep Reinforcement Learning ( DRL ) has recently shown a great success on a wide range of tasks , ranging from games ( Mnih et al. , 2015 ) to robotics control ( Levine et al. , 2016 ; Bengio & LeCun , 2016 ) . However , for more complex problems with larger state and action spaces or sparse reward settings , traditional DRL methods hardly works . Hierarchical reinforcement learning ( HRL ) in which multiple layers of policies are trained to learn to operate on different levels of temporal abstraction , has long held the promise to learn such difficult tasks ( Dayan & Hinton , 1992 ; Parr & Russell , 1997 ; Barto & Mahadevan , 2003 ) . By decomposing a complex problem into subproblems , HRL significantly reduces the difficulty of solving specific task . Learning multiple levels of policies in parallel is challenging due to non-stationary state transition functions . Recent HRL approaches ( Nachum et al. , 2018 ; Levy et al. , 2019 ) use states as goals directly , allowing simple and fast training of the lower layer . Human intelligence is remarkable for their fast adaptation to many new situations using the knowledge learned from past experience . However , agents trained by conventional DRL methods mentioned above can only learn one separate policy per task , failing to generalize to new tasks without additional large amount of training data . Meta reinforcement learning ( meta-RL ) addresses such problems by learning how to learn . Given a number of tasks with similar structures , meta-RL methods enable agents learn such structure from previous experience on many tasks . Thus when encountering a new task , agents can quickly adapt to it with only a small amount of experience . Most current meta-RL methods leverage experience from previous tasks to adapt to new tasks by directly learn the policy parameters over primitive action space . ( Finn et al. , 2017 ; Rakelly et al. , 2019 ) . Such approaches suffer from two problems : ( i ) For complex tasks which requires sophisticated control strategies , it would be quite inefficient to directly learn such policy with one nonlinear function approximator and the adaptation to new tasks is prone to be inaccurate . This problem can become more severe in spare reward settings . ( ii ) When the task distribution is much wider ( riding bicycle as meta-train task and riding motorcycle as meta-test task ) , these methods can hardly be effective since primitive action execution mechanism is entirely different although they may share a similar high-level strategy . Moreover , existing current meta-RL methods perform badly in sparse reward settings , which are quite common in real world . In this paper , we aim at tackling the problems mentioned above by proposing an efficient hierarchical meta-RL method that realizes meta learning high-level goal generation and leaves the learning of low-level policy for independent RL . Intuitively , this is quite similar to how a human being behaves : we usually transfer the overall understanding of similar tasks rather than remember specific actions . Our meta goal-generation framework is built on top of the architecture of PEARL ( Rakelly et al. , 2019 ) and a two level hierarchy inspired by HAC ( Levy et al. , 2019 ) . Our evaluation on several simulated robotics environments ( Plappert et al. , 2018 ) shows the superiority of MGHRL to stateof-the-art meta-RL and hierarchical RL methods in sparse reward settings . Generally , our contributions are as follows : • We propose an algorithm that achieves efficient meta reinforcement learning on challenging robotics environments with sparse reward settings and outperforms other leading methods . • Similar to the way humans leverage past experience to learn new complex tasks , our algorithm focuses on meta learning the overall strategy for different tasks , which provides a much simpler and better way for meta RL comparing with directly learning the detailed solution . Since we focus on meta goal-generation and leave the low level policy for independent learning , we believe our algorithm can still accelerate the acquisition of new tasks sampled from much wider task distributions . For example , to learn tasks such as riding bicycles and riding a motorcycle , the two primitive action execution mechanism are entirely different but the two learning process still share similar high-level structures . Through meta goal-generation learning , we expect our method can still accelerate the acquisition of such tasks . We leave these for future work to explore . 2 RELATED WORK . Our algorithm is based on meta learning framework ( Thrun & Pratt , 1998 ; Schmidhuber , 1987 ; Bengio et al. , 1991 ) , which aims to learn models that can adapt quickly to new tasks . Meta learning algorithms for few-shot supervised learning problems have explored a wide variety of approaches and architectures ( Santoro et al. , 2016 ; Vinyals et al. , 2016 ; Ravi & Larochelle , 2017 ) . In the context of reinforcement learning , recurrent ( Duan et al. , 2016 ; Wang et al. , 2016 ) and recursive ( Mishra et al. , 2018 ) meta-RL methods adapt to new tasks by aggregating experience into a latent representation on which the policy is conditioned . Another set of methods is gradient-based meta reinforcement learning ( Finn et al. , 2017 ; Stadie et al. , 2018 ; Rothfuss et al. , 2019 ; Xu et al. , 2018 ) . Its objective is to learn an initialization such that after one or few steps of policy gradients the agent attains full performance on a new task . These methods focus on on-policy meta learning which are usually sample inefficient . Our algorithm is closely related to probabilistic embeddings for actor-critic RL ( PEARL ) ( Rakelly et al. , 2019 ) , which is an off-policy meta RL algorithm . PEARL leverages posterior sampling to decouple the problems of inferring the task and solving it , which greatly enhances meta-learning efficiency . However , when facing complex tasks that require sophisticated control strategies , PEARL can not effectively learn a proper meta-policy as we will show in Section 5 . Discovering meaningful and effective hierarchical policies is a longstanding research problem in RL ( Dayan & Hinton , 1992 ; Parr & Russell , 1997 ; Sutton et al. , 1999 ; Bacon et al. , 2017 ; Dietterich , 2000 ) . Schmidhuber 1987 proposed a HRL approach that can support multiple levels . Multi-level hierarchies have the potential to accelerate learning in sparse reward tasks because they can divide a problem into a set of short-horizon subproblems . Nachum et al . 2018 proposed HIRO , a 2-level HRL approach that can learn off-policy and outperforms two other popular HRL techniques used in continuous domains : Option-Critic ( Bacon et al. , 2017 ) and FeUdal Networks ( Vezhnevets et al. , 2017 ) . Our algorithm is built on Hierarchical actor-critic ( Levy et al. , 2019 ) , which is a framework that can learn multiple levels of policies in parallel . Most current HRL works focus on the learning problem in a single task and few of them considers to take advantage of HRL for multi-task or metalearning tasks . MLSH ( Frans et al . ) is such a work which also combines meta-RL with Hierarchical RL . It focuses on meta learning on the low level policy and need to retrain its high level policy when facing new tasks . In contrast , with the key insight that humans leverage abstracted prior knowledge obtained from past experience , our method focus on meta learning high level overall strategy using past experience and leave the detailed action execution for independent RL . 3 BACKGROUND . 3.1 META REINFORCEMENT LEARNING . In our meta learning scenario , we assume a distribution of tasks p ( τ ) that we want our model to adapt to . Each task correspond to a different Markov decision process ( MDP ) , Mi = { S , A , Ti , Ri } , with state space S , action space A , transition distribution Ti , and reward function Ri . We assume that the transitions and reward function vary across tasks . Meta RL aims to learn a policy that can adapt to maximize the expected reward for novel tasks from p ( τ ) as efficiently as possible . PEARL ( Rakelly et al. , 2019 ) is an off-policy meta-reinforcement learning method that drastically improves sample efficiency comparing to previous meta-RL algorithms . The meta-training process of PEARL learns a policy that adapts to the task at hand by conditioning the history of past transitions , which we refer to as context c. Specifically , for the ith transition in task τ , cτi = ( si , ai , ri , s ′ i ) . PEARL leverages an inference network qφ ( z|c ) and outputs probabilistic latent variable z . The parameters of q ( z|c ) are optimized jointly with the parameters of the actor πθ ( a|s , z ) and critic Qθ ( s , a , z ) , using the reparametrization trick ( Kingma & Welling , 2014 ) to compute gradients for parameters of qφ ( z|c ) through sampled z ’ s . 3.2 HIERARCHICAL ACTOR-CRITIC . HAC ( Levy et al. , 2019 ) aims to accelerate learning by enabling hierarchical agents to jointly learn a multi-level hierarchy of policies in parallel . HAC is comprised of two components : a particular hierarchical architecture and a method for learning the multiple levels of policies in parallel given sparse rewards . The hierarchies produced by HAC have a specific architecture consisting of a set of nested , goal-conditioned policies that use the state space as the mechanism for breaking down a task into subtasks . HAC extends the idea of Hindsight Experience Replay ( Andrychowicz et al. , 2017 ) by creating two types of hindsight transitions . Hindsight action transition simulates a transition function that uses the optimal low level policy while hindsight goal transition use the final states achieved as the goal state in each step ’ s transition . They enable agents to learn multiple policies in parallel using only sparse reward functions . 4 ALGORITHM . 4.1 TWO-LEVEL HIERARCHY . We set up a hierarchical two-layer RL structure similar to HAC . The high level network uses policy µh to generate goals for temporally extended periods in terms of desired observations . In our task they correspond to the positional features of the gripper . The low level policy µl directly controls the agent and produces actions for moving towards the desired goals . As shown in Figure 1 ( a ) , the high level policy µh observes the state and produces a high level action ( or goal ) gt . Low level policy µl has at most K attempts of primitive action to achieve gt . Here , K which can be viewed as the maximum horizon of a subgoal action is a hyperparameter given by the user . As long as the low level policy µl run out of K attempts or gt is achieved , this high level transition terminates . The high level policy uses agent ’ s current state as the new observation and produced another goal for low level policy to achieve . We use an intrinsic reward function in which a reward of 0 is granted only if the goal produced by high level policy is achieved and a reward of -1 otherwise . Note that the environment ’ s return ( i.e . whether the agent successfully accomplished the task ) will not affect the reward received by the low level policy . In our evaluation on simulated robotics environments , we use the positional features of the observations as the representation for gt . A goal gt is judged to be achieved only if the distance between gt and the gripper ’ s current position sn+1 is less than threshold l .

Summary: This paper studies the problem of leveraging past experience to quickly solve new control tasks. The starting point (and perhaps the main contribution) is the observation that some tasks have similar high-level goals, while differing in how those goals are achieved. To that end, the paper introduces an meta-RL algorithm that, given a new task, attempts to solve it by adapting a high-level, goal-setting module, and learn a new, low-level policy to reach each commanded goal. The proposed method might be viewed as a combination of PEARL [Rakelly 19] and HAC [Levy 19]. The proposed method is compared against state-of-the-art hierarchical RL and meta-RL methods on four robotic manipulation tasks. The proposed method outperforms the baselines on each task.