Flexible and Efficient Long-Range Planning Through Curious Exploration

1 INTRODUCTION . Many complex behaviors such as cleaning a kitchen , organizing a drawer , or cooking a meal require plans that are a combination of low-level geometric manipulation and high-level action sequencing . For example , boiling water requires sequencing high-level actions such as fetching a pot , pouring water into the pot , and turning on the stove . In turn , each of these high-level steps consists of many low-level task-specific geometric action primitives . For instance , grabbing a pot requires intricate motor manipulation and physical considerations such as friction , force , etc . The process of combining these low-level geometric decisions and high-level action sequences is often referred to as multi-step planning . While high-level task planning and low-level geometric planning are difficult problems on their own , integrating them presents unique challenges that add further complexity . Task and Motion Planning ( TAMP ) is a powerful approach to the problem which constructs plans in logical terms that execute a sequence of macro-actions that are composed of geometric motion plans ( Fikes & Nilsson , 1971 ; Dantam et al. , 2016 ) . While TAMP has been successful at generating temporally extended multistep plans that conform to geometric constraints , it requires actions to have defined preconditions and effects that modify the logical description of the world state . This is an unreasonable assumption for complex physical tasks because real-world effects and preconditions are often unknown or difficult to describe with logical predicates . These assumptions limit the flexibility and robustness of the TAMP approach . In addition , TAMP is computationally costly because it requires geometric motion planning for each macro-action sample . In contrast , deep reinforcement learning ( DRL ) methods have shown success at learning flexible policies for a variety of complex tasks in unstructured domains ( Arulkumaran et al. , 2017 ) . However , DRL methods typically require very large sample sizes of successful trajectories to learn useful policies and thus have trouble in sparse reward landscapes ( Choi et al. , 2019 ; Riedmiller et al. , 2018 ) . Multi-step planning , by its very nature , involves sparse reward landscapes , as highly specific action sequences must be executed before any positive rewards are observed . As such , flexible multi-step planning is an open challenge in artificial intelligence and algorithmic robotics . In this paper we combine aspects of TAMP and DRL to achieve progress toward efficient and flexible multi-step planning . We introduce the Curious Sample Planner ( CSP ) , which uses curiosity to bias the search through action space toward novel points in the state space . Our method combines the flexibility and transferability of DRL with the temporally extended multi-step planning of TAMP . We illustrate the power of CSP in a variety of qualitatively distinct problems in multi-step planning for which logical descriptions of action effects would be difficult to construct , including the building of complex structures and the discovery of simple machines for achieving challenging physical goals . We show CSP dramatically improves sample complexity compared to random TAMP exploration and DRL baselines . We compare a variety of distinct curiosity metrics for use with CSP and demonstrate that dynamics-based curiosity performs well on tasks which require many dynamic object interactions while state-based curiosity performs better for structure-building tasks . Finally , we show that CSP can be used to achieve substantial transfer between related but qualitatively distinct tasks , utilizing knowledge in one domain to increase performance in a similar domain . 2 RELATED WORK . Finding policies in sparse-reward environments has been an ever-present challenge for modern deep reinforcement learning . Model-free RL algorithms either require millions of samples in a sparsereward setting ( Mnih et al. , 2015 ; Hessel et al. , 2017 ) or a precise reward curriculum that gives checkpoints or gradients toward the goal ( Heess et al. , 2017 ) . While these algorithms yield convincing performance , humans can learn these tasks without strict reward curricula or massive sample sizes . Model-based RL presents a promising approach for solving these sparse-reward tasks by allowing agents to create plans through virtual experimentation . Rather than learning a direct policy or value function , model-based RL focuses on learning the local or global dynamics of the task to enable planning ( Kaiser et al. , 2019 ; Levine et al. , 2015 ) or to create imagined environment roll-outs for more efficient policy training ( Weber et al. , 2017 ; Ha & Schmidhuber , 2018 ) . Once the structure of the task is known , the question becomes how to best plan under that structure . Our algorithm is a method for multi-step planning after the environmental dynamics and task structure are known . A classical approach to planning is that of sample-based geometric motion planners . Rapidly Exploring Random Trees ( RRT ) and Probabilistic Road Maps ( PRM ) combine goal-directed sampling with off-target sampling to balance exploration of the state space with exploitation of knowledge about the goal configuration ( Lavalle , 1998 ; Kavraki et al. , 1996 ) . While these algorithms can work even for high dimensional configuration spaces , they are computationally intractable for tasks with the complex constraints that often exist in real world settings ( Kingston et al. , 2018 ) . For a robotic manipulator , grasping an object has a necessary condition that the agent ’ s manipulator is in a position to grasp that object . These constraint barriers in the configuration space render the goal-directed component of motion planning ineffective and necessitate a random exploration the entire configuration space . So while sample-based geometric motion planners are effective at simple tasks , they fail for temporally extended multi-step tasks with intricate constraints . We use sample-based geometric motion planners as a part of our solution to the multi-step planning problem . More recently , Task and Motion Planning has shown success in developing temporally extended multi-step plans under both fully known and partially observed environment dynamics . A number of TAMP algorithms ( Kaelbling & Lozano-Pérez , 2013 ; Gravot et al. , 2005 ; Hertle et al. , 2012 ; Srivastava et al. , 2014 ) have solved tasks such as block stacking , object packing , table setting , and much more . TAMP algorithms generally iterate between a motion planning step ( using RRT , PRM , etc ) and a symbolic planning step using task planning algorithms such as Fast Downward Planning ( Helmert , 2006 ) or Fast Forward Planning ( Hoffmann & Nebel , 2001 ) . However , because geometric motion planning is relatively slow , Task and Motion Planning is computationally burdensome . Ad- ditionally , defining logical predicates for the effects and preconditions of macro-actions is a manual undertaking which limits the flexibility of the solution . We take inspiration from Task and Motion planning by using geometric motion planners as procedures used in executing macro-actions , but increase planning efficiency and flexibility by intelligently sampling macro-actions and avoiding logical descriptions of effects and preconditions . Many other methods have been proposed that increase the efficiency and flexibility of Task and Motion Planning . Supervised learning of the preconditions and effects of certain macro-actions removes the need for manual specification , but requires constructing task-specific training datasets ( Kroemer & Sukhatme , 2016 ; Wang et al. , 2018 ) . Other work has focused on factoring the planning problem into submanifolds with analytic constraints in order to reduce the size of the search space . ( Garrett et al. , 2018 ; Vega-Brown & Roy , 2018 ) . TAMP can be accelerated by generating a sampling distribution around the goal trajectory using GANs ( Kim et al. , 2018 ) or by using reinforcement learning to learn search heuristics through expert examples or previously solved problem instances ( Chitnis et al. , 2016 ; Kim et al. , 2019 ) . In this work , we also aim to build an algorithm that can utilize information gained from previous problem instances to speed up planning and generalize to related tasks . However , instead of supervising on expert examples or task-specific training sets , we aim to use a self-supervision signal to increase the efficiency of multi-step planning . Along with model-based RL and planning , intrinsic motivation , also known as curiosity , has been used to turn sparse reward spaces into dense ones by self-supervising on an auxiliary task . Some commonly used auxiliary tasks are forward dynamics ( Burda et al. , 2019 ) , random network distillation ( Choi et al. , 2019 ) , and state estimation ( Mitash et al. , 2017 ) . The error from these predictor networks are considered an intrinsic reward for some reward optimization algorithm . These auxiliary tasks encourage state-space and action-space coverage and increase the probability of encountering sparse rewards over random policies with epsilon-greedy exploration ( Jaderberg et al. , 2016 ) . Our algorithm uses the principle of self-supervision on auxiliary tasks to accelerate goal discovery in multi-step planning . 3 MATHEMATICAL DESCRIPTION . 3.1 PROBLEM DESCRIPTION . In its general formulation , our problem consists of a robot R in an environment with dynamic movable objects D1 , D2 , ... Dk and static objects S1 , S2 , ... Sl . The robot R has an associated configuration space Q containing its possible configurations in its environment , so that dim ( Q ) is the number of degrees of freedom ofR . There is a larger state-space S which , in addition toQ , includes includes the total description of the positions and orientations of the dynamic and static objects , as well as their velocities , masses , and shapes , together with any ( possibly dynamic ) state-space constraints such as links or joints that may ( permanently or temporarily ) exist between objects in the environment . The robot is equipped with a space A of primitive actions that cause reliable ( though possibly stochastic ) changes in the state . The robot is also equipped with a set of macro-actions M that can be translated into sequences of motion primitives in A using geometric motion planning and inverse kinematic procedures . Each macro-action is parameterized by specific continuous and discrete parameters , and has a set of implicitly-defined feasible conditions . In our specific experiments , we use a mounted robot arm with seven bounded revolute joints resulting in a seven dimensional configuration space , with fully-extended length r. Our robot ’ s action primitives A consist of ( 1 ) a 7-dimensional continuous rotation space composed by linearly interpolating between θ1 and θ2 where θ1 , θ2 are 7-vectors corresponding to bounds on each rotational degree of freedom , and ( 2 ) the ability to add a link constraint between any two objects in the environment , as long as the objects are touching and the robot arm is sufficiently close to both . We use Bullet ( Coumans , 2015 ) , a flexible physics simulation library , to execute robot actions in simulation . The robot has access to a PickPlace macro-action which takes a target object , a goal position , and a goal orientation as input , and moves the target block to the goal position and orientation . The robot also has access to AddConstraint macro which takes two target objects as input , moves them into position for linking , and performs the link ; and a corresponding RemoveConstraint that breaks the link if the objects are connected . While this setup is fairly typical of robotics appli- cations , the CSP algorithm we describe below makes no assumptions about the effects of actions on objects in the environment , and so our method could be applied to more complicated environments containing soft-body objects , cloths , and multiple robotic manipulators . The robot must use its actions and macro-actions to achieve a goal , defined by a state-space subset G ⊆ S , by manipulating the state from some specified initial conditions into the G subset . For example , for our robotic arm , the task of moving a dynamic object Di outside the reachable radius r of the robot would be defined by G = { s ∈ S|Dix 2 + Diy 2 > r2 } . Though this type of goal definition is very simple and natural , finding a solution to such a task require complex strategic planning — e.g . perhaps requiring the discovery and construction of a simple machine , such as a ramp that could be used to slide the block out of the robot ’ s reach ( see section 4.1 ) . Finally , the robot has access to a ( deterministic ) dynamics model , f : S × A → S , that maps states and actions to future states i.e . f ( st , a ) = st+τ for some τ > 0 . In this work , f simply gives the robot access directly to the underlying Bullet physics simulator as a black box , but future research could involve having f be a learned ( possibly non-deterministic ) forward dynamic prediction network that must be inferred from agent experience ( such as e.g . ( Mrowca et al. , 2018 ) ) .

Summary: The paper introduces Curious Sample Planner (CSP) a long-horizon motion planning method that combines task and motion planning with deep reinforcement learning in order to solve simulated robotic tasks with sparse rewards. The CSP algorithm considers two different hierarchies of actions: primitive actions, which control the rotation of several joints in a robotic arm, and macro-actions corresponding to complex behaviours  such as moving from one position to another or linking two objects together. Macro-actions are selected using the actor-critic architecture PPO and then turned into primitive actions using geometric motion planning and inverse kinematics. Specifically, RRT-Connect is used for motion planning with the recursive Newton-Euler algorithm for inverse kinematics on a perfect model of the environment to determine the specific sequence of primitive actions necessary to execute the macro-action. As CSP is interacting with the environment, it also builds a tree of states in the environment connected by the macro-actions leading to each of them. Each vertex of the tree is assigned a curiosity score, which is used as an exploration bonus for PPO and to determine the probability with which each vertex is sampled from the tree for future exploration. The whole process is repeated until a feasible path from the initial state to the goal state is found. The paper provides empirical evaluations in four different tasks where it compares the performance of CSP with three different curiosity measures to the performance of PPO and A2C. The results show that CSP accomplishes each task while using significantly less samples. Moreover, a second set of experiments is presented that show the potential for transfer learning across tasks using CSP.