Learning to Plan in High Dimensions via Neural Exploration-Exploitation Trees

1 INTRODUCTION . Path planning is a fundamental problem with many real-world applications , such as robot manipulation and autonomous driving . A simple planning problem within low-dimensional state space can be solved by first discretizing the continuous state space into a grid , and then searching for a path on top of it using graph search algorithms such as A∗ ( Hart et al. , 1968 ) . However , due to the curse of dimensionality , these approaches do not scale well with the number of dimensions of the state space . For high-dimensional planning problems , people often resort to sampling-based approaches to avoid explicit discretization . Sampling-based planning algorithms , such as probabilistic roadmaps ( PRM ) ( Kavraki et al. , 1996 ) , rapidly-exploring random trees ( RRT ) ( LaValle , 1998 ) , and their variants ( Karaman & Frazzoli , 2011 ) incrementally build an implicit representation of the state space using probing samples . These generic algorithms typically employ a uniform sampler which does not make use of the structures of the problem . Therefore they may require lots of samples to obtain a feasible solution for complicated problems . To improve the sample efficiency , heuristic biased samplers , such as Gaussian sampler ( Boor et al. , 1999 ) , bridge test ( Hsu et al. , 2003 ) and reachability-guided sampler ( Shkolnik et al. , 2009 ) have been proposed . All these sampling heuristics are designed manually to address specific structural properties , which may or may not be valid for a new problem , and may lead to even worse performance compared to the uniform proposal . Online adaptation in path planning has also been investigated for improving sample efficiency in current planning problem . Specifically , Hsu et al . ( 2005 ) exploits online algorithms to dynamically adapts the mixture weights of several manually designed biased samplers . Burns & Brock ( 2005a ; b ) fit a model for the planning environment incrementally and use the model for planning . Yee et al . ( 2016 ) mimics the Monte-Carlos tree search ( MCTS ) for problems with continuous state and action spaces . These algorithms treat each planning problem independently , and the collected data from previous experiences and built model will be simply discarded when solving a new problem . However , in practice , similar planning problems may be solved again and again , where the problems are different but sharing common structures . For instance , grabbing a coffee cup on a table at different time are different problems , since the layout of paper and pens , the position and orientation of coffee cups may be different every time ; however , all these problems show common structures of handling similar objects which are placed in similar fashions . Intuitively , if the common characteristics across problems can be learned via some shared latent representation , a planner based on such representation can then be transferred to new problems with improved sample efficiency . ∗indicates equal contribution . Several methods have been proposed recently to learn from past planning experiences to conduct more efficient and generalizable planning for future problems . These works are limited in one way or the other . Zucker et al . ( 2008 ) ; Zhang et al . ( 2018 ) ; Huh & Lee ( 2018 ) treat the sampler in the sampling-based planner as a stochastic policy to be learned and apply policy gradient or TD-algorithm to improve the policy . Finney et al . ( 2007 ) ; Bowen & Alterovitz ( 2014 ) ; Ye & Alterovitz ( 2017 ) ; Ichter et al . ( 2018 ) ; Kim et al . ( 2018 ) ; Kuo et al . ( 2018 ) apply imitation learning based on the collected demonstrations to bias for better sampler via variants of probabilistic models , e.g. , ( mixture of ) Gaussians , conditional VAE , GAN , HMM and RNN . However , many of these approaches either rely on specially designed local features or assume the problems are indexed by special parameters , which limits the generalization ability . Deep representation learning provides a promising direction to extract the common structure among the planning problems , and thus mitigate such limitation on handdesigned features . However , existing work , e.g. , motion planning networks ( Qureshi et al. , 2019 ) , value iteration networks ( VIN ) ( Tamar et al. , 2016 ) , and gated path planning networks ( GPPN ) ( Lee et al. , 2018 ) , either apply off-the-shelf MLP architecture ignoring special structures in planning problems or can only deal with discrete state and action spaces in low-dimensional settings . In this paper , we present Neural EXploration-EXploitation Tree ( NEXT ) , a meta neural path planning algorithm for high-dimensional continuous state space problems . The core contribution is a novel attention-based neural architecture that is capable of learning generalizable problem structures from previous experiences and produce promising search directions with automatic online exploration-exploitation balance adaption . Compared to existing learning-based planners , • NEXT is more generic . We propose an architecture that can embed high dimensional continuous state spaces into low dimensional discrete spaces , on which a neural planning module is used to extract planning representation . These module will be learned end-to-end . • NEXT balances exploration-exploitation trade-off . We integrate the learned neural prior into an upper confidence bound ( UCB ) style algorithm to achieve an online balance between exploration and exploitation when solving a new problem . Empirically , we show that NEXT can exploit past experiences to reduce the number of required samples drastically for solving new planning problems , and significantly outperforms previous state-of-the-arts on several benchmark tasks . 1.1 RELATED WORKS . Designing non-uniform sampling strategies for random search to improve the planning efficiency has been considered as we discussed above . Besides the mentioned algorithms , there are other works along this line , including informed RRT∗ ( Gammell et al. , 2014 ) and batch informed Trees ( BIT * ) ( Gammell et al. , 2015 ) as the representative work . Randomized A∗ ( Diankov & Kuffner , 2007 ) and sampling-based A∗ ( Persson & Sharf , 2014 ) expand the search tree with hand-designed heuristics . These methods incorporate the human prior knowledge via hard-coded rules , which is fixed and unable to adapt to problems , and thus , may not universally applicable . Choudhury et al . ( 2018 ) ; Song et al . ( 2018 ) attempt to learn search heuristics . However , both methods are restricted to planning on discrete domains . Meanwhile , the latter one always employs an unnecessary hierarchical structure for path planning , which leads to inferior sample efficiency and extra computation . The online exploration-exploitation trade-off is also an important issue in planning . For instance , Rickert et al . ( 2009 ) constructs a potential field sampler and tuned the sampler variance based on collision rate for the trade-off heuristically . Paxton et al . ( 2017 ) separates the action space into high-level discrete options and low-level continuous actions , and only considered the trade-off at the discrete option level , ignoring the exploration-exploitation in the fine action space . These existing works address the trade-off in an ad-hoc way , which may be inferior for the balance . There have been sevearl non-learning-based planning methods that can also leverage experiences ( Kavraki et al. , 1996 ; Phillips et al. , 2012 ) by utilizing search graphs created in previous problems . However , they are designed for largely fixed obstacles and can not be generalized to unseen tasks from the same planning problems distribution . 2 SETTINGS FOR LEARNING TO PLAN . Let S ⊆ Rq be the state space of the problem , e.g. , all the configurations of a robot and its base location in the workspace , Sobs ( S be the obstacles set , Sfree : = S \ Sobs be the free space , sinit ∈ Sfree be the initial state and Sgoal ( Sfree be the goal region . Then the space of all collisionfree paths can be defined as a continuous function Ξ : = { ξ ( · ) : [ 0 , 1 ] → Sfree } . Let c ( · ) : Ξ 7→ R be the cost functional over a path . The optimal path planning problem is to find the optimal path in terms of cost c ( · ) from start sinit to goal Sgoal in free space Sfree , i.e. , ξ∗ = argminξ∈Ξ c ( ξ ) , s.t . ξ ( 0 ) = sinit , ξ ( 1 ) ∈ Sgoal . ( 1 ) Traditionally ( Karaman & Frazzoli , 2011 ) , the planner has direct access to ( sinit , Sgoal , c ( · ) ) and the workspace map ( Ichter et al. , 2018 ; Tamar et al. , 2016 ; Lee et al. , 2018 ) , map ( · ) : R2 or R3 7→ { 0 , 1 } , ( 0 : free spaces and 1 : obstacles ) . Since Sfree often has a very irregular geometry ( illustrated in Figure 10 in Appendix A ) , it is usually represented via a collision detection module which is able to detect the obstacles in a path segment . For the same reason , the feasible paths in Ξ are hard to be described in parametric forms , and thus , the nonparametric ξ , such as a sequence of interconnected path segments [ s0 , s1 ] , [ s1 , s2 ] , . . . , [ sT−1 , sT ] ⊂ S with ξ ( 0 ) = s0 = sinit and ξ ( 1 ) = sT , is used with an additive cost ∑T i=1 c ( [ si−1 , si ] ) . Assuming given the planning problems { Ui : = ( sinit , Sgoal , S , Sfree , map , c ( · ) ) } Ni=1 sampled from some distribution U , we are interested in learning an algorithm alg ( · ) , which can produce the ( nearly ) -optimal path efficiently from the observed planning problems . Formally , the learning to plan is defined as alg∗ ( · ) = argminalg∈A EU∈U [ ` ( alg ( U ) ) ] , ( 2 ) where A denotes the planning algorithm family , and ` ( · ) denotes some loss function which evaluates the quality of the generated path and the efficiency of the alg ( · ) , e.g. , size of the search tree . We elaborate each component in Eq ( 2 ) in the following sections . We first introduce the tree-based sampling algorithm template in Section 3 , upon which we instantiate the alg ( · ) via a novel attention-based neural parametrization in Section 4.2 with exploration-exploitation balance mechanism in Section 4.1 . We design the ` -loss function and the meta learning algorithm in Section 4.3 . Composing every component together , we obtain the neural exploration-exploitation trees ( NEXT ) which achieves outstanding performances in Section 5 . Goal Start Goal Goal Start Start l l l l L max V+U max V+U Goal Start Goal Start max V+U π ( |s ) b. Selecta . New Task c. Expand d. Update Repeat l L Goal Start Goal Start Goal Startse lf- im pr ov e se lf- im pr ov e max V+U π ( |s ) ... ... ... ... problem will be extracted based on the tree T . The template of tree-based sampling algorithms is summarized in Algorithm 1 and illustrated in Fig . 1 ( c ) . A key component of the algorithm is the Expand operator , which generates the next exploration point snew and its parent sparent ∈ V . To ensure the feasibility of the solution , the snew must be reachable from T , i.e. , [ sparent , snew ] is collision-free , which is checked by a collision detection function . As we will discuss in Appendix B , by instantiating different Expand operators , we will arrive at many existing algorithms , such as RRT ( LaValle , 1998 ) and EST ( Hsu et al. , 1997 ; Phillips et al. , 2004 ) . One major limitation of existing TSAs is that they solve each problem independently from scratch and ignore past planning experiences in similar environments . We introduce the neural components into TSA template to form the learnable planning algorithm family A , which can explicitly take advantages of the past successful experiences to bias the Expand towards more promising regions .

Summary: Motion-planning in high dimensional spaces is challenging due to the curse of dimensionality. Sampling-based motion planners like PRM, PRM*, RRT, RRT*, BIT* etc have been the go-to solution family. But often these algorithms solve every planning problem tabula rasa. This work combines learning with sampling-based planning such that the parent-sampling and expansion steps instead of being done by common heuristics are learnt in an online manner. Also the resulting exploration-exploitation problem is naturally dealt via using a UCB-style contextual bandit algorithm. Since the number of parents are always varying the common trick of 'describe the action choices with respect to the environment' is adopted so that varying number of actions (states to be sampled from) can be naturally incorporated.