Learning Space Partitions for Path Planning

1 Introduction . Path planning has been used extensively in many applications , ranging from reinforcement learning [ 7 , 13 , 14 ] and robotics [ 27 , 35 , 26 ] to biology [ 24 ] , chemistry [ 40 ] , material design [ 21 ] , and compiler optimization [ 42 ] . The goal is to find the most rewarding trajectory ( i.e. , state-action sequence ) x = ( s0 , a0 , s1 , . . . , sn ) in the search space Ω : x∗ = arg maxx∈Ω f ( x ) , where f ( x ) is the reward . In this work , we focus on deterministic path planning problems with long trajectories x , and discontinuous and/or multimodal reward functions f . Such high-dimensional non-convex optimization problems exist in many real domains , both continuous and discrete . While we could always find nearoptimal x by random sampling given an infinite query budget , in practice we prefer a sample-efficient method that achieves high-reward trajectories with fewer queries of the reward function f . While global methods like Bayesian Optimization ( BO ) [ 3 ] may struggle with limited samples and high-dimensional spaces , classic approaches like CEM [ 37 ] and CMA-ES [ 16 ] learn a local model around promising trajectories . For example , CEM tracks a population of trajectories and repeatedly resamples its population according to the highest-performing trajectories from the previous generation . On the other hand , such a focus can trap CEM in local optima , as confirmed empirically ( Sec . 5 ) . Other recent approaches , such as VOOT [ 22 ] and DOO [ 31 ] , use a ( recursive ) region partitioning scheme : they split the search space Ω into sub-regions Ω = Ω1 ∪ . . . ∪ Ωk , then invest more samples into promising sub-regions while continuing to explore other regions via an upper confidence bound 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( UCB ) . While such exploration-exploitation procedures adaptively focus on promising sub-regions and lead to sub-linear regret and optimality guarantees , their region partition procedure is manually designed by humans and remains non-adaptive . For example , DOO partitions the space with uniform axis-aligned grids and VOOT with Voronoi cells , both independent of the reward f to be optimized . Recently , Wang et al . proposed LaNAS [ 46 ] and LaMCTS [ 45 ] , which adaptively partition the search regions based on sampled function values , and focus on good regions . They achieve strong empirical performance on Neural Architecture Search ( NAS ) and black-box optimization , outperforming many existing methods including evolutionary algorithms and BO . Notably , in recent NeurIPS ’ 20 black-box optimization challenges , two teams that use variants of LaMCTS ranked 3rd [ 38 ] and 8th [ 23 ] . In this paper , we provide a simple theoretical analysis of LaMCTS to reveal the underlying principles of adaptive region partitioning , an analysis missing in the original work . Based on this analysis , we propose Latent Space Partitions for Path Planning ( LaP3 ) , a novel optimization technique for path-planning . Unlike LaMCTS , LaP3 uses a latent representation of the search space . Additionally , we use the maximum ( instead of the mean ) as the node score to improve sample efficiency , verified empirically in Sec . 5.3 . Both changes are motivated by our theoretical analysis . We verify LaP3 on several challenging path-planning tasks , including 2D navigation environments from past work with difficult-to-escape local optima , and real-world planning problems in compiler optimization and molecular design . In all tasks , LaP3 demonstrates substantially stronger exploration ability to escape from local optima compared to several baselines including CEM , CMA-ES and VOOT . On compiler phase ordering , we achieve on average 39 % and 31 % speedup in execution cycles comparing to -O3 optimization and OpenTuner [ 1 ] , two widely used optimization techniques in compilers . On molecular design , LaP3 outperforms all of our baselines in generating molecules with high values of desirable properties , beating the best baseline in average property value by up to 0.4 on properties in a [ 0 , 1 ] range . Additionally , extensive ablation studies show factors that affect the quality of planning and verify the theoretical analysis . LaP3 is a general planning technique and can be readily plugged into existing algorithms with path planning components . For example , we apply LaP3 to PETS [ 7 ] in model-based RL and observe substantially improved performance for high-dimensional continuous control and navigation , compared to CEM as used in the original PETS framework . 2 Latent Space Monte Carlo Tree Search ( LaMCTS ) . LaMCTS [ 45 ] is recently proposed to solve black-box optimization problems x∗ = arg maxx f ( x ) via recursively learning f -dependent region partitions . Fig . 1 and Alg . 1 show the details of LaMCTS as well as our proposed approach LaP3 ( formally introduced in Sec . 4 ) for comparison . Algorithm 1 LaP3 Pseudocode for Path Planning . Improvements over LaMCTS in green . 1 : Input : Number of rounds T , Environment Oracle : f ( x ) , Dataset D , Sampling Latent Model h ( x ) , Partitioning Latent Model s ( x ) . 2 : Parameters : Initial # samples Ninit , Re-partitioning interval Npar , Node partition threshold Nthres , UCB parameter Cp . 3 : Pre-train h ( · ) on D when D 6= ∅ . 4 : Set region partition V0 = { Ω } . 5 : Draw Ninit samples uniformly from S0 = { ( xi , f ( xi ) ) } Niniti=1 ⊂ Ω . 6 : for t = 0 , . . . , T −Ninit − 1 do 7 : if t divides Npar then 8 : Train/fine-tune latent model h ( · ) using samples St ∪ D ( Eqn . ? ? ) . 9 : Re-learn region partition Vt ← Partition ( Ω , St , Nthres , s ( · ) ) in latent space Φs of s ( · ) . 10 : end if 11 : for k : = root , k /∈ Vleaf do 12 : k ← arg max Ωc∈child ( Ωk ) bc , where bc : = 1 n ( Ωc ) ∑ xi∈Ωc f ( xi ) max xi∈Ωc f ( xi ) + Cp √ 2 log n ( Ωk ) n ( Ωc ) . 13 : end for 14 : Initialize CMA-ES using encodings of St ∩ Ωk via h ( · ) . Here Ωk is the chosen leaf sub-region . 15 : St ← St−1 ∪ { ( xt , f ( xt ) ) } , where xt is drawn from CMA-ES and decoded via h−1 ( · ) . 16 : end for Good action Bad action Instance 𝒙 ∈ Ω Space partition High 𝑓 ( 𝒙 ) Low 𝑓 ( 𝒙 ) Search Space Ω Root Good action ( a ) ( b ) 𝜴𝟐 Ω '' 𝜴𝟑 𝜴𝟒 Ω % Ω & Ω' Ω Exploitation Exploration ( c ) 𝜴𝟒 ( 𝒙 , 𝑓 𝒙 ) Sample 𝒙 ∈ Ω ! and get the function value 𝑓 ( 𝒙 ) Figure 1 : LaP3 extends LaMCTS [ 45 ] to path planning . ( a ) Starting from a search space Ω , both LaP3 and LaMCTS first draw a few samples x ∈ Ω , then learn to partition Ω into a sub-region Ω1 with good samples ( high f ( x ) ) and a sub-region Ω2 with bad samples ( low f ( x ) ) . Compared to LaMCTS , LaP3 uses a latent space and reduces the dimensionality of the search space . ( b ) Sampling follows the learned recursive space partition , focusing on good regions while still exploring bad regions using UCB . LaP3 uses the maximum of the sampled value in a region ( maxxi∈Ω f ( xi ) ) as the node value , while LaMCTS uses the mean . ( c ) Upon reaching a leaf , new data points are sampled within the region and the space partition is relearned . LaMCTS starts with Ninit random samples of the entire search space Ω ( line 5 in Alg . 1 ) . For a region Ωk , let n ( Ωk ) be the number of samples within . LaMCTS dictates that , if n ( Ωk ) ≥ Nthres , then Ωk is partitioned into disjoint sub-regions Ωk = Ωgood ∪ Ωbad as its children ( Fig . 1 ( a ) - ( b ) , line 9 in Alg . 1 , the function Partition ) . Intuitively , Ωgood contains promising samples with high f , while Ωbad contains samples with low f . Unlike DOO and VOOT , such a partition is learned using St ∩ Ωk , our samples so far in the region , and is thus dependent on the function f to be optimized . Given tree-structured sub-regions , new samples are mostly drawn from promising regions and occasionally from other regions for exploration . This is achieved by Monte Carlo Tree Search ( MCTS ) [ 4 ] ( line 11-13 ) : at each tree branching , the UCB score b is computed to balance exploration and exploitation ( line 12 ) . Then the subregion with highest UCB score is selected ( e.g. , it may have high f and/or low n ) . This is done recursively until a leaf sub-region Ω′ is reached . Then a new sample x is drawn from Ω′ ( line 15 ) either uniformly , or from a local model constructed by an existing optimizer ( e.g. , TuRBO [ 10 ] , CMA-ES [ 16 ] ) , in which case LaMCTS becomes a meta-algorithm . When more samples are collected , regions are further partitioned and the tree gets deeper . Finally , the function Partition in Alg . 1 is defined as follows : first a 2-class K-means on ( x , f ( x ) ) is used to create positive/negative sample groups . Next , a SVM classifier is used to learn the decision boundary ( hence the partition ) , so that samples with high f ( x ) fall into Ωgood , and samples with low f ( x ) fall into Ωbad ( Fig . 1 ( a ) ) . See Appendix A for the pseudo code . The partition boundary can also be re-learned after more samples are collected ( line 9 ) . 3 A Theoretical Understanding of Space Partitioning . While LaMCTS [ 45 ] shows strong empirical performance , it contains several components with no clear theoretical justification . Here we attempt to give a formal regret analysis when sub-regions { Ωk } are fixed and all at the same tree level , and the function f is deterministic . We leave further analysis of tree node splitting and evolution of hierarchical structure to future work . Despite the drastic simplification , our regret bound still shows why an f -dependent region partition is helpful . By showing that a better regret bound can be achieved by a clever region partition as empirically used in the Partition function in Alg . 1 , we justify the design of LaMCTS . Furthermore , our analysis suggests several empirical improvements over LaMCTS and motivates the design of LaP3 , which outperforms multiple classic approaches on hard path planning problems .

Summary: The paper presents a black-box optimization method that efficiently searches the parameter by partitioning the parameter space using a latent space. The proposed method is built upon LaMCTS proposed by Wang et al. [2020]. This paper provides theoretical analysis of LaMCTS and proposed the modifications. The proposed algorithm, PlaLaM, was tested in synthetic tasks and real-world tasks, including compiler optimization and molecular design tasks. The proposed algorithm outperformed baseline algorithms in those experiments.