Flexible and Efficient Long-Range Planning Through Curious Exploration

1 INTRODUCTION . Many complex behaviors such as cleaning a kitchen , organizing a drawer , or cooking a meal require plans that are a combination of low-level geometric manipulation and high-level action sequencing . For example , boiling water requires sequencing high-level actions such as fetching a pot , pouring water into the pot , and turning on the stove . In turn , each of these high-level steps consists of many low-level task-specific geometric action primitives . For instance , grabbing a pot requires intricate motor manipulation and physical considerations such as friction , force , etc . The process of combining these low-level geometric decisions and high-level action sequences is often referred to as multi-step planning . While high-level task planning and low-level geometric planning are difficult problems on their own , integrating them presents unique challenges that add further complexity . Task and Motion Planning ( TAMP ) is a powerful approach to the problem which constructs plans in logical terms that execute a sequence of macro-actions that are composed of geometric motion plans ( Fikes & Nilsson , 1971 ; Dantam et al. , 2016 ) . While TAMP has been successful at generating temporally extended multistep plans that conform to geometric constraints , it requires actions to have defined preconditions and effects that modify the logical description of the world state . This is an unreasonable assumption for complex physical tasks because real-world effects and preconditions are often unknown or difficult to describe with logical predicates . These assumptions limit the flexibility and robustness of the TAMP approach . In addition , TAMP is computationally costly because it requires geometric motion planning for each macro-action sample . In contrast , deep reinforcement learning ( DRL ) methods have shown success at learning flexible policies for a variety of complex tasks in unstructured domains ( Arulkumaran et al. , 2017 ) . However , DRL methods typically require very large sample sizes of successful trajectories to learn useful policies and thus have trouble in sparse reward landscapes ( Choi et al. , 2019 ; Riedmiller et al. , 2018 ) . Multi-step planning , by its very nature , involves sparse reward landscapes , as highly specific action sequences must be executed before any positive rewards are observed . As such , flexible multi-step planning is an open challenge in artificial intelligence and algorithmic robotics . In this paper we combine aspects of TAMP and DRL to achieve progress toward efficient and flexible multi-step planning . We introduce the Curious Sample Planner ( CSP ) , which uses curiosity to bias the search through action space toward novel points in the state space . Our method combines the flexibility and transferability of DRL with the temporally extended multi-step planning of TAMP . We illustrate the power of CSP in a variety of qualitatively distinct problems in multi-step planning for which logical descriptions of action effects would be difficult to construct , including the building of complex structures and the discovery of simple machines for achieving challenging physical goals . We show CSP dramatically improves sample complexity compared to random TAMP exploration and DRL baselines . We compare a variety of distinct curiosity metrics for use with CSP and demonstrate that dynamics-based curiosity performs well on tasks which require many dynamic object interactions while state-based curiosity performs better for structure-building tasks . Finally , we show that CSP can be used to achieve substantial transfer between related but qualitatively distinct tasks , utilizing knowledge in one domain to increase performance in a similar domain . 2 RELATED WORK . Finding policies in sparse-reward environments has been an ever-present challenge for modern deep reinforcement learning . Model-free RL algorithms either require millions of samples in a sparsereward setting ( Mnih et al. , 2015 ; Hessel et al. , 2017 ) or a precise reward curriculum that gives checkpoints or gradients toward the goal ( Heess et al. , 2017 ) . While these algorithms yield convincing performance , humans can learn these tasks without strict reward curricula or massive sample sizes . Model-based RL presents a promising approach for solving these sparse-reward tasks by allowing agents to create plans through virtual experimentation . Rather than learning a direct policy or value function , model-based RL focuses on learning the local or global dynamics of the task to enable planning ( Kaiser et al. , 2019 ; Levine et al. , 2015 ) or to create imagined environment roll-outs for more efficient policy training ( Weber et al. , 2017 ; Ha & Schmidhuber , 2018 ) . Once the structure of the task is known , the question becomes how to best plan under that structure . Our algorithm is a method for multi-step planning after the environmental dynamics and task structure are known . A classical approach to planning is that of sample-based geometric motion planners . Rapidly Exploring Random Trees ( RRT ) and Probabilistic Road Maps ( PRM ) combine goal-directed sampling with off-target sampling to balance exploration of the state space with exploitation of knowledge about the goal configuration ( Lavalle , 1998 ; Kavraki et al. , 1996 ) . While these algorithms can work even for high dimensional configuration spaces , they are computationally intractable for tasks with the complex constraints that often exist in real world settings ( Kingston et al. , 2018 ) . For a robotic manipulator , grasping an object has a necessary condition that the agent ’ s manipulator is in a position to grasp that object . These constraint barriers in the configuration space render the goal-directed component of motion planning ineffective and necessitate a random exploration the entire configuration space . So while sample-based geometric motion planners are effective at simple tasks , they fail for temporally extended multi-step tasks with intricate constraints . We use sample-based geometric motion planners as a part of our solution to the multi-step planning problem . More recently , Task and Motion Planning has shown success in developing temporally extended multi-step plans under both fully known and partially observed environment dynamics . A number of TAMP algorithms ( Kaelbling & Lozano-Pérez , 2013 ; Gravot et al. , 2005 ; Hertle et al. , 2012 ; Srivastava et al. , 2014 ) have solved tasks such as block stacking , object packing , table setting , and much more . TAMP algorithms generally iterate between a motion planning step ( using RRT , PRM , etc ) and a symbolic planning step using task planning algorithms such as Fast Downward Planning ( Helmert , 2006 ) or Fast Forward Planning ( Hoffmann & Nebel , 2001 ) . However , because geometric motion planning is relatively slow , Task and Motion Planning is computationally burdensome . Ad- ditionally , defining logical predicates for the effects and preconditions of macro-actions is a manual undertaking which limits the flexibility of the solution . We take inspiration from Task and Motion planning by using geometric motion planners as procedures used in executing macro-actions , but increase planning efficiency and flexibility by intelligently sampling macro-actions and avoiding logical descriptions of effects and preconditions . Many other methods have been proposed that increase the efficiency and flexibility of Task and Motion Planning . Supervised learning of the preconditions and effects of certain macro-actions removes the need for manual specification , but requires constructing task-specific training datasets ( Kroemer & Sukhatme , 2016 ; Wang et al. , 2018 ) . Other work has focused on factoring the planning problem into submanifolds with analytic constraints in order to reduce the size of the search space . ( Garrett et al. , 2018 ; Vega-Brown & Roy , 2018 ) . TAMP can be accelerated by generating a sampling distribution around the goal trajectory using GANs ( Kim et al. , 2018 ) or by using reinforcement learning to learn search heuristics through expert examples or previously solved problem instances ( Chitnis et al. , 2016 ; Kim et al. , 2019 ) . In this work , we also aim to build an algorithm that can utilize information gained from previous problem instances to speed up planning and generalize to related tasks . However , instead of supervising on expert examples or task-specific training sets , we aim to use a self-supervision signal to increase the efficiency of multi-step planning . Along with model-based RL and planning , intrinsic motivation , also known as curiosity , has been used to turn sparse reward spaces into dense ones by self-supervising on an auxiliary task . Some commonly used auxiliary tasks are forward dynamics ( Burda et al. , 2019 ) , random network distillation ( Choi et al. , 2019 ) , and state estimation ( Mitash et al. , 2017 ) . The error from these predictor networks are considered an intrinsic reward for some reward optimization algorithm . These auxiliary tasks encourage state-space and action-space coverage and increase the probability of encountering sparse rewards over random policies with epsilon-greedy exploration ( Jaderberg et al. , 2016 ) . Our algorithm uses the principle of self-supervision on auxiliary tasks to accelerate goal discovery in multi-step planning . 3 MATHEMATICAL DESCRIPTION . 3.1 PROBLEM DESCRIPTION . In its general formulation , our problem consists of a robot R in an environment with dynamic movable objects D1 , D2 , ... Dk and static objects S1 , S2 , ... Sl . The robot R has an associated configuration space Q containing its possible configurations in its environment , so that dim ( Q ) is the number of degrees of freedom ofR . There is a larger state-space S which , in addition toQ , includes includes the total description of the positions and orientations of the dynamic and static objects , as well as their velocities , masses , and shapes , together with any ( possibly dynamic ) state-space constraints such as links or joints that may ( permanently or temporarily ) exist between objects in the environment . The robot is equipped with a space A of primitive actions that cause reliable ( though possibly stochastic ) changes in the state . The robot is also equipped with a set of macro-actions M that can be translated into sequences of motion primitives in A using geometric motion planning and inverse kinematic procedures . Each macro-action is parameterized by specific continuous and discrete parameters , and has a set of implicitly-defined feasible conditions . In our specific experiments , we use a mounted robot arm with seven bounded revolute joints resulting in a seven dimensional configuration space , with fully-extended length r. Our robot ’ s action primitives A consist of ( 1 ) a 7-dimensional continuous rotation space composed by linearly interpolating between θ1 and θ2 where θ1 , θ2 are 7-vectors corresponding to bounds on each rotational degree of freedom , and ( 2 ) the ability to add a link constraint between any two objects in the environment , as long as the objects are touching and the robot arm is sufficiently close to both . We use Bullet ( Coumans , 2015 ) , a flexible physics simulation library , to execute robot actions in simulation . The robot has access to a PickPlace macro-action which takes a target object , a goal position , and a goal orientation as input , and moves the target block to the goal position and orientation . The robot also has access to AddConstraint macro which takes two target objects as input , moves them into position for linking , and performs the link ; and a corresponding RemoveConstraint that breaks the link if the objects are connected . While this setup is fairly typical of robotics appli- cations , the CSP algorithm we describe below makes no assumptions about the effects of actions on objects in the environment , and so our method could be applied to more complicated environments containing soft-body objects , cloths , and multiple robotic manipulators . The robot must use its actions and macro-actions to achieve a goal , defined by a state-space subset G ⊆ S , by manipulating the state from some specified initial conditions into the G subset . For example , for our robotic arm , the task of moving a dynamic object Di outside the reachable radius r of the robot would be defined by G = { s ∈ S|Dix 2 + Diy 2 > r2 } . Though this type of goal definition is very simple and natural , finding a solution to such a task require complex strategic planning — e.g . perhaps requiring the discovery and construction of a simple machine , such as a ramp that could be used to slide the block out of the robot ’ s reach ( see section 4.1 ) . Finally , the robot has access to a ( deterministic ) dynamics model , f : S × A → S , that maps states and actions to future states i.e . f ( st , a ) = st+τ for some τ > 0 . In this work , f simply gives the robot access directly to the underlying Bullet physics simulator as a black box , but future research could involve having f be a learned ( possibly non-deterministic ) forward dynamic prediction network that must be inferred from agent experience ( such as e.g . ( Mrowca et al. , 2018 ) ) .

Summary: This paper tackles the problem of enabling robots to learn long-horizon, sparse-reward tasks. The proposed approach, the Curious Sample Planner (CSP), builds on insights in task and motion planning (TAMP), which is a standard approach for tackling these kinds of tasks. TAMP constructs a plan in the space of macro-actions (e.g., move object 1 to location (x,y)), and uses a motion planner to execute each macro-action. However, TAMP typically requires being able to describe macro-action effects and preconditions with logical predicates, which can be impossible in real-world environments, due to complex dynamics and interactions. CSP overcomes this limitation by planning in the space of macro-actions in a way that is biased toward novelty.