Learning to Execute: Efficient Learning of Universal Plan-Conditioned Policies in Robotics

1 Introduction . A central goal of robotics research is to design intelligent machines that can solve arbitrary and formerly unseen tasks while interacting with the physical world . Reinforcement Learning ( RL ) ( Sutton & Barto , 2018 ) is a generic framework to automatically learn such intelligent behavior with little human engineering . Still , teaching an RL agent to actually exhibit general-purpose problem-solving behavior is , while possible in principle , prohibitive in practice . This is due to practical restrictions including limited computational resources and limited data availability . The latter limitation is particularly dramatic in robotics , where interaction with the physical world is costly . On the other hand , for many robotics scenarios , there is a rough model of the environment available . This can be exploited , e.g. , using model-based planning approaches ( Mordatch et al. , 2012 ; Kuindersma et al. , 2016 ; Toussaint et al. , 2018 ) . Model-based planners potentially offer a more data-efficient way to reason about an agent ’ s interaction with the world . Model-based planners have been used in many areas of robotics , such as for indoor and aerial robots ( Faust et al. , 2018 ) , visual manipulation ( Jeong et al. , 2020 ) , or humanoid walking ( Mordatch et al. , 2015 ) . Still , if the model does not account for stochasticity or contains systematic errors , directly following the resulting plan will not be successful . The present work starts from the observation that both pure RL approaches and pure planning approaches have strengths and limitations that are fairly complementary . RL makes no assumptions about the environment but is data-hungry , and model-based planning generally implies model simplifications but is data-efficient . For robotic manipulation tasks , it seems natural to try and integrate both approaches into one framework that combines the strengths of both . In the present work we seek to add an additional perspective to the open question of how this can be achieved best . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We introduce a novel approach that we call Learning to Execute ( L2E ) . Our approach translates sparsereward goal-conditioned Markov Decision Processes ( MDPs ) ( Bellman , 1957 ) into plan-conditioned MDPs . L2E exploits a simple planning module to create crude plans , which are then used to teach any off-the-shelf off-policy RL agent to execute them . L2E makes use of final-volume-preserving reward shaping ( FV-RS ) ( Schubert et al. , 2021 ) , allowing it to train a universal plan-conditioned policy with high data efficiency . The contributions of this work are : • We introduce L2E , which uses RL to efficiently learn to execute approximate plans from a model-based planner in a plan-conditioned MDP . We describe formally how FV-RS can be used as a tool to construct such plan-conditioned MDPs from goal-conditioned MDPs . • We introduce plan replay strategies to efficiently learn universal plan-conditioned policies . • We demonstrate , using robotic pushing problems , that L2E exhibits increased performance when compared to pure RL methods , pure planning methods , or other methods combining learning and planning . We discuss work related to ours in section 2 , explain background and notation in section 3 , and introduce our method in section 4 . We present our experimental results in section 5 , discuss limitations in section 6 , and conclude with section 7 . 2 Related Work . 2.1 Goal-Conditioned Policies . Goal-conditioned or universal policies ( Kaelbling , 1993 ; Moore et al. , 1999 ; Foster & Dayan , 2002 ; Schaul et al. , 2015 ; Veeriah et al. , 2018 ; Nasiriany et al. , 2019 ) not only act based on the state the agent finds itself in , but also based on the goal it tries to achieve . Hindsight Experience Replay ( HER ) ( Andrychowicz et al. , 2017 ) is a particularly efficient way to learn universal policies . Here , achieved outcomes of the agent ’ s interaction with the environment are interpreted as desired goals in order to improve sample efficiency in sparse-reward settings . L2E draws great inspiration from this work , but in contrast to HER , L2E learns a universal planconditioned policy . This means that the L2E policy in general can execute multiple plans leading to the same goal . Although this presents a more complex learning task , we show in our experiments that by incorporating plan information using plan-based FV-RS , the sample efficiency of L2E is significantly improved over HER . 2.2 Plan- and Trajectory-Conditioned Policies . Plan-conditioned policies create behavior that depends on plans that are input to the decision making . Lynch et al . ( 2020 ) learn plans and how to execute them from data generated by a human “ playing ” with a teleoperated robot . The resulting policy is conditional on a latent space of encoded plans . Our work differs from this paradigm in that human interaction is not needed . Both Lynch et al . ( 2020 ) and Co-Reyes et al . ( 2018 ) directly imitate a planned trajectory by maximizing its likelihood . In contrast , the plans used in the present work are not directly imitated . Using FV-RS guarantees that the fully trained L2E agent will reach its goal after finite time even if the plan provided is wrong . Guo et al . ( 2019 ) learn trajectory-conditioned policies to self-imitate diverse ( optimal and suboptimal ) trajectories from the agent ’ s past experience . We instead assume in this work that the plan is provided by an external model-based planner . This allows the L2E agent to use external information during training that could not be concluded from its own experience yet . 2.3 Learning from Demonstration . L2E learns how to execute plans in order to achieve different tasks . In this sense , it is related to Learning from Demonstration ( LfD ) techniques that exploit demonstrations when learning a task . Existing work ( Argall et al. , 2009 ; Hussein et al. , 2017 ; Ravichandar et al. , 2020 ) differs significantly both in how the demonstration examples are collected and how the policy is then derived . Taylor et al . ( 2011 ) derive an approximate policy from human demonstration , and then use this to bias the exploration in a final RL stage . Hester et al . ( 2017 ) train a policy on both expert data and collected data , combining supervised and temporal difference losses . Salimans & Chen ( 2018 ) use a single demonstration as starting points to which the RL agent is reset at the beginning of each episode . Peng et al . ( 2018 ) use motion capture data to guide exploration by rewarding the RL agent to imitate it . In Cabi et al . ( 2019 ) , demonstrations are combined with reward sketching done by a human . Interactive human feedback during training is another source of information used in Thomaz et al . ( 2006 ) ; Knox & Stone ( 2010 ) . Kinose & Taniguchi ( 2020 ) integrate RL and demonstrations using generative adversarial imitation learning by interpreting the discriminator loss as an additional optimality signal in multi-objective RL . While these LfD approaches are related to L2E in that external information is used to increase RL efficiency , it is in contrast assumed in L2E that this external information is provided by a planner . 2.4 Combining Learning with Planning . Similarly to demonstrations , external plans can be exploited to facilitate learning . Faust et al . ( 2018 ) connect short-range goal-conditioned navigation policies into complex navigation tasks using probabilistic roadmaps . In contrast , L2E learns a single plan-conditioned policy for both short-term and long-term decision making . Sekar et al . ( 2020 ) use planning in a learned model to optimize for expected future novelty . In contrast , L2E encourages the agent to stay close to the planned behavior . Zhang et al . ( 2016 ) use model-predictive control to generate control policies that are then used to regularize the RL agent . In L2E , no such intermediate control policy is created , and a reward signal is computed directly from the plan . In Guided Policy Search ( Levine & Koltun , 2013 ) , differential dynamic programming is used to create informative guiding distributions from a transition model for policy search . These distributions are used to directly regularize the policy in a supervised fashion , while L2E makes use of FV-RS as a mechanism to interface planning and RL . Christiano et al . ( 2016 ) learn an inverse dynamics model to transfer knowledge from a policy in the source domain to a policy in the target domain . The idea of integrating model-based and model-free RL has also been studied independently of planning ( Pong et al. , 2018 ; Janner et al. , 2019 ) . In contrast , in L2E the model is translated by a planner into long-horizon plans . In the experiments section , we compare L2E against two representative examples from the literature mentioned above . The first is using a plan to identify subgoals that are then pursued by an RL agent , as done in Faust et al . ( 2018 ) . The second is executing the plan using an inverse model , similar to the approach in Christiano et al . ( 2016 ) . These two baselines and L2E can be seen as representatives of a continuum : Christiano et al . ( 2016 ) follow the plan very closely , trying to imitate the planner at each time step . Faust et al . ( 2018 ) relax this requirement and only train the agent to reach intermediate goals . Finally , in L2E , the agent is free to deviate arbitrarily from the plan ( although it is biased to stay close ) , as long as it reaches the goal . We find that L2E results in significantly higher success rates when compared against both baselines . 3 Background . 3.1 Goal-Conditioned MDPs and RL . We consider settings that can be described as discrete-time MDPs M = 〈S , A , T , γ , R , PS〉 . S and A denote the set of all possible states and actions , respectively . T : S× A× S→ R+0 is the transition probability ( density ) ; T ( s′|s , a ) is the probability of the next state being s′ if the current state is s and a is chosen as the action . The agent receives a real-valued reward R ( s , a , s′ ) after each transition . Immediate and future rewards are traded off by the discount factor γ ∈ [ 0 , 1 ) . PS : S→ R+0 is the initial state distribution . The goal of RL is to learn an optimal policy π∗ : S×A→ R+0 that maximizes the expected discounted return . In other words , RL algorithms generally try to find π∗ = argmax π ∞∑ t=0 γtEst+1∼T ( ·|st , at ) , at∼π ( ·|st ) , s0∼PS [ R ( st , at , st+1 ) ] ( 1 ) from collected transition and reward data D = { ( si , ai , ri , s′i ) } ni=0 . More specifically for this work , we are interested in applications in robotics , where both S and A are typically continuous . There exists a wide range of algorithms for this case . For the experiments in this paper , soft actor-critic ( SAC ) ( Haarnoja et al. , 2018 ) is used . In a goal-conditioned MDP MG = 〈S , G , A , T , γ , RG , PS , PG〉 , the reward function RG ( s , a , s′ , g ) has an additional input parameter , the goal g ∈ G. Here , PG : G→ R+0 is the distribution of goals . The optimal goal-conditioned policy π∗G acts optimally with respect to any of these goals .

Summary: The paper proposes Learning to Execute (L2E), which can make use of a pre-designed planner in reinforcement learning (RL) in robotics. The authors extend the idea of final-volume-preserving reward shaping (FV-RS) and formulate plan-conditioned MDPs for designing L2E. The method was evaluated through a simulation experiment.