Learning Space Partitions for Path Planning

1 Introduction . Path planning has been used extensively in many applications , ranging from reinforcement learning [ 7 , 13 , 14 ] and robotics [ 27 , 35 , 26 ] to biology [ 24 ] , chemistry [ 40 ] , material design [ 21 ] , and compiler optimization [ 42 ] . The goal is to find the most rewarding trajectory ( i.e. , state-action sequence ) x = ( s0 , a0 , s1 , . . . , sn ) in the search space Ω : x∗ = arg maxx∈Ω f ( x ) , where f ( x ) is the reward . In this work , we focus on deterministic path planning problems with long trajectories x , and discontinuous and/or multimodal reward functions f . Such high-dimensional non-convex optimization problems exist in many real domains , both continuous and discrete . While we could always find nearoptimal x by random sampling given an infinite query budget , in practice we prefer a sample-efficient method that achieves high-reward trajectories with fewer queries of the reward function f . While global methods like Bayesian Optimization ( BO ) [ 3 ] may struggle with limited samples and high-dimensional spaces , classic approaches like CEM [ 37 ] and CMA-ES [ 16 ] learn a local model around promising trajectories . For example , CEM tracks a population of trajectories and repeatedly resamples its population according to the highest-performing trajectories from the previous generation . On the other hand , such a focus can trap CEM in local optima , as confirmed empirically ( Sec . 5 ) . Other recent approaches , such as VOOT [ 22 ] and DOO [ 31 ] , use a ( recursive ) region partitioning scheme : they split the search space Ω into sub-regions Ω = Ω1 ∪ . . . ∪ Ωk , then invest more samples into promising sub-regions while continuing to explore other regions via an upper confidence bound 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( UCB ) . While such exploration-exploitation procedures adaptively focus on promising sub-regions and lead to sub-linear regret and optimality guarantees , their region partition procedure is manually designed by humans and remains non-adaptive . For example , DOO partitions the space with uniform axis-aligned grids and VOOT with Voronoi cells , both independent of the reward f to be optimized . Recently , Wang et al . proposed LaNAS [ 46 ] and LaMCTS [ 45 ] , which adaptively partition the search regions based on sampled function values , and focus on good regions . They achieve strong empirical performance on Neural Architecture Search ( NAS ) and black-box optimization , outperforming many existing methods including evolutionary algorithms and BO . Notably , in recent NeurIPS ’ 20 black-box optimization challenges , two teams that use variants of LaMCTS ranked 3rd [ 38 ] and 8th [ 23 ] . In this paper , we provide a simple theoretical analysis of LaMCTS to reveal the underlying principles of adaptive region partitioning , an analysis missing in the original work . Based on this analysis , we propose Latent Space Partitions for Path Planning ( LaP3 ) , a novel optimization technique for path-planning . Unlike LaMCTS , LaP3 uses a latent representation of the search space . Additionally , we use the maximum ( instead of the mean ) as the node score to improve sample efficiency , verified empirically in Sec . 5.3 . Both changes are motivated by our theoretical analysis . We verify LaP3 on several challenging path-planning tasks , including 2D navigation environments from past work with difficult-to-escape local optima , and real-world planning problems in compiler optimization and molecular design . In all tasks , LaP3 demonstrates substantially stronger exploration ability to escape from local optima compared to several baselines including CEM , CMA-ES and VOOT . On compiler phase ordering , we achieve on average 39 % and 31 % speedup in execution cycles comparing to -O3 optimization and OpenTuner [ 1 ] , two widely used optimization techniques in compilers . On molecular design , LaP3 outperforms all of our baselines in generating molecules with high values of desirable properties , beating the best baseline in average property value by up to 0.4 on properties in a [ 0 , 1 ] range . Additionally , extensive ablation studies show factors that affect the quality of planning and verify the theoretical analysis . LaP3 is a general planning technique and can be readily plugged into existing algorithms with path planning components . For example , we apply LaP3 to PETS [ 7 ] in model-based RL and observe substantially improved performance for high-dimensional continuous control and navigation , compared to CEM as used in the original PETS framework . 2 Latent Space Monte Carlo Tree Search ( LaMCTS ) . LaMCTS [ 45 ] is recently proposed to solve black-box optimization problems x∗ = arg maxx f ( x ) via recursively learning f -dependent region partitions . Fig . 1 and Alg . 1 show the details of LaMCTS as well as our proposed approach LaP3 ( formally introduced in Sec . 4 ) for comparison . Algorithm 1 LaP3 Pseudocode for Path Planning . Improvements over LaMCTS in green . 1 : Input : Number of rounds T , Environment Oracle : f ( x ) , Dataset D , Sampling Latent Model h ( x ) , Partitioning Latent Model s ( x ) . 2 : Parameters : Initial # samples Ninit , Re-partitioning interval Npar , Node partition threshold Nthres , UCB parameter Cp . 3 : Pre-train h ( · ) on D when D 6= ∅ . 4 : Set region partition V0 = { Ω } . 5 : Draw Ninit samples uniformly from S0 = { ( xi , f ( xi ) ) } Niniti=1 ⊂ Ω . 6 : for t = 0 , . . . , T −Ninit − 1 do 7 : if t divides Npar then 8 : Train/fine-tune latent model h ( · ) using samples St ∪ D ( Eqn . ? ? ) . 9 : Re-learn region partition Vt ← Partition ( Ω , St , Nthres , s ( · ) ) in latent space Φs of s ( · ) . 10 : end if 11 : for k : = root , k /∈ Vleaf do 12 : k ← arg max Ωc∈child ( Ωk ) bc , where bc : = 1 n ( Ωc ) ∑ xi∈Ωc f ( xi ) max xi∈Ωc f ( xi ) + Cp √ 2 log n ( Ωk ) n ( Ωc ) . 13 : end for 14 : Initialize CMA-ES using encodings of St ∩ Ωk via h ( · ) . Here Ωk is the chosen leaf sub-region . 15 : St ← St−1 ∪ { ( xt , f ( xt ) ) } , where xt is drawn from CMA-ES and decoded via h−1 ( · ) . 16 : end for Good action Bad action Instance 𝒙 ∈ Ω Space partition High 𝑓 ( 𝒙 ) Low 𝑓 ( 𝒙 ) Search Space Ω Root Good action ( a ) ( b ) 𝜴𝟐 Ω '' 𝜴𝟑 𝜴𝟒 Ω % Ω & Ω' Ω Exploitation Exploration ( c ) 𝜴𝟒 ( 𝒙 , 𝑓 𝒙 ) Sample 𝒙 ∈ Ω ! and get the function value 𝑓 ( 𝒙 ) Figure 1 : LaP3 extends LaMCTS [ 45 ] to path planning . ( a ) Starting from a search space Ω , both LaP3 and LaMCTS first draw a few samples x ∈ Ω , then learn to partition Ω into a sub-region Ω1 with good samples ( high f ( x ) ) and a sub-region Ω2 with bad samples ( low f ( x ) ) . Compared to LaMCTS , LaP3 uses a latent space and reduces the dimensionality of the search space . ( b ) Sampling follows the learned recursive space partition , focusing on good regions while still exploring bad regions using UCB . LaP3 uses the maximum of the sampled value in a region ( maxxi∈Ω f ( xi ) ) as the node value , while LaMCTS uses the mean . ( c ) Upon reaching a leaf , new data points are sampled within the region and the space partition is relearned . LaMCTS starts with Ninit random samples of the entire search space Ω ( line 5 in Alg . 1 ) . For a region Ωk , let n ( Ωk ) be the number of samples within . LaMCTS dictates that , if n ( Ωk ) ≥ Nthres , then Ωk is partitioned into disjoint sub-regions Ωk = Ωgood ∪ Ωbad as its children ( Fig . 1 ( a ) - ( b ) , line 9 in Alg . 1 , the function Partition ) . Intuitively , Ωgood contains promising samples with high f , while Ωbad contains samples with low f . Unlike DOO and VOOT , such a partition is learned using St ∩ Ωk , our samples so far in the region , and is thus dependent on the function f to be optimized . Given tree-structured sub-regions , new samples are mostly drawn from promising regions and occasionally from other regions for exploration . This is achieved by Monte Carlo Tree Search ( MCTS ) [ 4 ] ( line 11-13 ) : at each tree branching , the UCB score b is computed to balance exploration and exploitation ( line 12 ) . Then the subregion with highest UCB score is selected ( e.g. , it may have high f and/or low n ) . This is done recursively until a leaf sub-region Ω′ is reached . Then a new sample x is drawn from Ω′ ( line 15 ) either uniformly , or from a local model constructed by an existing optimizer ( e.g. , TuRBO [ 10 ] , CMA-ES [ 16 ] ) , in which case LaMCTS becomes a meta-algorithm . When more samples are collected , regions are further partitioned and the tree gets deeper . Finally , the function Partition in Alg . 1 is defined as follows : first a 2-class K-means on ( x , f ( x ) ) is used to create positive/negative sample groups . Next , a SVM classifier is used to learn the decision boundary ( hence the partition ) , so that samples with high f ( x ) fall into Ωgood , and samples with low f ( x ) fall into Ωbad ( Fig . 1 ( a ) ) . See Appendix A for the pseudo code . The partition boundary can also be re-learned after more samples are collected ( line 9 ) . 3 A Theoretical Understanding of Space Partitioning . While LaMCTS [ 45 ] shows strong empirical performance , it contains several components with no clear theoretical justification . Here we attempt to give a formal regret analysis when sub-regions { Ωk } are fixed and all at the same tree level , and the function f is deterministic . We leave further analysis of tree node splitting and evolution of hierarchical structure to future work . Despite the drastic simplification , our regret bound still shows why an f -dependent region partition is helpful . By showing that a better regret bound can be achieved by a clever region partition as empirically used in the Partition function in Alg . 1 , we justify the design of LaMCTS . Furthermore , our analysis suggests several empirical improvements over LaMCTS and motivates the design of LaP3 , which outperforms multiple classic approaches on hard path planning problems .

Summary: This work proposes a new path planning method PlaLaM, which is an extension of the LaMCTS algorithm. The authors provide a novel regret analysis of adaptive region partitioning schemes.   The authors demonstrate that the PlaLam method improves performance in toy 2D navigation tasks. The approach is also be applied to compiler phase ordering and molecular design achieving comparable performance to a solution based on proximal policy optimization at 4000 episodes.  The authors show comparisons to a number of baseline methods, but do not include a comparison the the LaMCTS method, which there approach is an extension of. I look forward to their explanation in the rebuttal.  Model-free RL and planning are my domains of expertise. Evolutionary algorithms are not, so please consider my review according.