Learning to Execute: Efficient Learning of Universal Plan-Conditioned Policies in Robotics

1 Introduction . A central goal of robotics research is to design intelligent machines that can solve arbitrary and formerly unseen tasks while interacting with the physical world . Reinforcement Learning ( RL ) ( Sutton & Barto , 2018 ) is a generic framework to automatically learn such intelligent behavior with little human engineering . Still , teaching an RL agent to actually exhibit general-purpose problem-solving behavior is , while possible in principle , prohibitive in practice . This is due to practical restrictions including limited computational resources and limited data availability . The latter limitation is particularly dramatic in robotics , where interaction with the physical world is costly . On the other hand , for many robotics scenarios , there is a rough model of the environment available . This can be exploited , e.g. , using model-based planning approaches ( Mordatch et al. , 2012 ; Kuindersma et al. , 2016 ; Toussaint et al. , 2018 ) . Model-based planners potentially offer a more data-efficient way to reason about an agent ’ s interaction with the world . Model-based planners have been used in many areas of robotics , such as for indoor and aerial robots ( Faust et al. , 2018 ) , visual manipulation ( Jeong et al. , 2020 ) , or humanoid walking ( Mordatch et al. , 2015 ) . Still , if the model does not account for stochasticity or contains systematic errors , directly following the resulting plan will not be successful . The present work starts from the observation that both pure RL approaches and pure planning approaches have strengths and limitations that are fairly complementary . RL makes no assumptions about the environment but is data-hungry , and model-based planning generally implies model simplifications but is data-efficient . For robotic manipulation tasks , it seems natural to try and integrate both approaches into one framework that combines the strengths of both . In the present work we seek to add an additional perspective to the open question of how this can be achieved best . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We introduce a novel approach that we call Learning to Execute ( L2E ) . Our approach translates sparsereward goal-conditioned Markov Decision Processes ( MDPs ) ( Bellman , 1957 ) into plan-conditioned MDPs . L2E exploits a simple planning module to create crude plans , which are then used to teach any off-the-shelf off-policy RL agent to execute them . L2E makes use of final-volume-preserving reward shaping ( FV-RS ) ( Schubert et al. , 2021 ) , allowing it to train a universal plan-conditioned policy with high data efficiency . The contributions of this work are : • We introduce L2E , which uses RL to efficiently learn to execute approximate plans from a model-based planner in a plan-conditioned MDP . We describe formally how FV-RS can be used as a tool to construct such plan-conditioned MDPs from goal-conditioned MDPs . • We introduce plan replay strategies to efficiently learn universal plan-conditioned policies . • We demonstrate , using robotic pushing problems , that L2E exhibits increased performance when compared to pure RL methods , pure planning methods , or other methods combining learning and planning . We discuss work related to ours in section 2 , explain background and notation in section 3 , and introduce our method in section 4 . We present our experimental results in section 5 , discuss limitations in section 6 , and conclude with section 7 . 2 Related Work . 2.1 Goal-Conditioned Policies . Goal-conditioned or universal policies ( Kaelbling , 1993 ; Moore et al. , 1999 ; Foster & Dayan , 2002 ; Schaul et al. , 2015 ; Veeriah et al. , 2018 ; Nasiriany et al. , 2019 ) not only act based on the state the agent finds itself in , but also based on the goal it tries to achieve . Hindsight Experience Replay ( HER ) ( Andrychowicz et al. , 2017 ) is a particularly efficient way to learn universal policies . Here , achieved outcomes of the agent ’ s interaction with the environment are interpreted as desired goals in order to improve sample efficiency in sparse-reward settings . L2E draws great inspiration from this work , but in contrast to HER , L2E learns a universal planconditioned policy . This means that the L2E policy in general can execute multiple plans leading to the same goal . Although this presents a more complex learning task , we show in our experiments that by incorporating plan information using plan-based FV-RS , the sample efficiency of L2E is significantly improved over HER . 2.2 Plan- and Trajectory-Conditioned Policies . Plan-conditioned policies create behavior that depends on plans that are input to the decision making . Lynch et al . ( 2020 ) learn plans and how to execute them from data generated by a human “ playing ” with a teleoperated robot . The resulting policy is conditional on a latent space of encoded plans . Our work differs from this paradigm in that human interaction is not needed . Both Lynch et al . ( 2020 ) and Co-Reyes et al . ( 2018 ) directly imitate a planned trajectory by maximizing its likelihood . In contrast , the plans used in the present work are not directly imitated . Using FV-RS guarantees that the fully trained L2E agent will reach its goal after finite time even if the plan provided is wrong . Guo et al . ( 2019 ) learn trajectory-conditioned policies to self-imitate diverse ( optimal and suboptimal ) trajectories from the agent ’ s past experience . We instead assume in this work that the plan is provided by an external model-based planner . This allows the L2E agent to use external information during training that could not be concluded from its own experience yet . 2.3 Learning from Demonstration . L2E learns how to execute plans in order to achieve different tasks . In this sense , it is related to Learning from Demonstration ( LfD ) techniques that exploit demonstrations when learning a task . Existing work ( Argall et al. , 2009 ; Hussein et al. , 2017 ; Ravichandar et al. , 2020 ) differs significantly both in how the demonstration examples are collected and how the policy is then derived . Taylor et al . ( 2011 ) derive an approximate policy from human demonstration , and then use this to bias the exploration in a final RL stage . Hester et al . ( 2017 ) train a policy on both expert data and collected data , combining supervised and temporal difference losses . Salimans & Chen ( 2018 ) use a single demonstration as starting points to which the RL agent is reset at the beginning of each episode . Peng et al . ( 2018 ) use motion capture data to guide exploration by rewarding the RL agent to imitate it . In Cabi et al . ( 2019 ) , demonstrations are combined with reward sketching done by a human . Interactive human feedback during training is another source of information used in Thomaz et al . ( 2006 ) ; Knox & Stone ( 2010 ) . Kinose & Taniguchi ( 2020 ) integrate RL and demonstrations using generative adversarial imitation learning by interpreting the discriminator loss as an additional optimality signal in multi-objective RL . While these LfD approaches are related to L2E in that external information is used to increase RL efficiency , it is in contrast assumed in L2E that this external information is provided by a planner . 2.4 Combining Learning with Planning . Similarly to demonstrations , external plans can be exploited to facilitate learning . Faust et al . ( 2018 ) connect short-range goal-conditioned navigation policies into complex navigation tasks using probabilistic roadmaps . In contrast , L2E learns a single plan-conditioned policy for both short-term and long-term decision making . Sekar et al . ( 2020 ) use planning in a learned model to optimize for expected future novelty . In contrast , L2E encourages the agent to stay close to the planned behavior . Zhang et al . ( 2016 ) use model-predictive control to generate control policies that are then used to regularize the RL agent . In L2E , no such intermediate control policy is created , and a reward signal is computed directly from the plan . In Guided Policy Search ( Levine & Koltun , 2013 ) , differential dynamic programming is used to create informative guiding distributions from a transition model for policy search . These distributions are used to directly regularize the policy in a supervised fashion , while L2E makes use of FV-RS as a mechanism to interface planning and RL . Christiano et al . ( 2016 ) learn an inverse dynamics model to transfer knowledge from a policy in the source domain to a policy in the target domain . The idea of integrating model-based and model-free RL has also been studied independently of planning ( Pong et al. , 2018 ; Janner et al. , 2019 ) . In contrast , in L2E the model is translated by a planner into long-horizon plans . In the experiments section , we compare L2E against two representative examples from the literature mentioned above . The first is using a plan to identify subgoals that are then pursued by an RL agent , as done in Faust et al . ( 2018 ) . The second is executing the plan using an inverse model , similar to the approach in Christiano et al . ( 2016 ) . These two baselines and L2E can be seen as representatives of a continuum : Christiano et al . ( 2016 ) follow the plan very closely , trying to imitate the planner at each time step . Faust et al . ( 2018 ) relax this requirement and only train the agent to reach intermediate goals . Finally , in L2E , the agent is free to deviate arbitrarily from the plan ( although it is biased to stay close ) , as long as it reaches the goal . We find that L2E results in significantly higher success rates when compared against both baselines . 3 Background . 3.1 Goal-Conditioned MDPs and RL . We consider settings that can be described as discrete-time MDPs M = 〈S , A , T , γ , R , PS〉 . S and A denote the set of all possible states and actions , respectively . T : S× A× S→ R+0 is the transition probability ( density ) ; T ( s′|s , a ) is the probability of the next state being s′ if the current state is s and a is chosen as the action . The agent receives a real-valued reward R ( s , a , s′ ) after each transition . Immediate and future rewards are traded off by the discount factor γ ∈ [ 0 , 1 ) . PS : S→ R+0 is the initial state distribution . The goal of RL is to learn an optimal policy π∗ : S×A→ R+0 that maximizes the expected discounted return . In other words , RL algorithms generally try to find π∗ = argmax π ∞∑ t=0 γtEst+1∼T ( ·|st , at ) , at∼π ( ·|st ) , s0∼PS [ R ( st , at , st+1 ) ] ( 1 ) from collected transition and reward data D = { ( si , ai , ri , s′i ) } ni=0 . More specifically for this work , we are interested in applications in robotics , where both S and A are typically continuous . There exists a wide range of algorithms for this case . For the experiments in this paper , soft actor-critic ( SAC ) ( Haarnoja et al. , 2018 ) is used . In a goal-conditioned MDP MG = 〈S , G , A , T , γ , RG , PS , PG〉 , the reward function RG ( s , a , s′ , g ) has an additional input parameter , the goal g ∈ G. Here , PG : G→ R+0 is the distribution of goals . The optimal goal-conditioned policy π∗G acts optimally with respect to any of these goals .

Summary: This paper addresses the problem of sample complexity in reinforcement learning. The approach is to drive down the sample complexity using reward shaping, where the reward shaping function is given by a planner. The paper shows how to use final-volume-preserving reward shaping (FV-RS) to convert a plan into a reward shaping function. The approach is evaluated on a simple pushing domain against planning in two forms, and Hindsight Experience Replay. The L2E approach proposed here outperforms the comparison algorithms.