Multi-objective Optimization by Learning Space Partition

1 INTRODUCTION . Multi-objective optimization ( MOO ) has been extensively used in many practical scenarios involving trade-offs between multiple objectives . For example , in automobile design ( Chang , 2015 ) , we must maximize the performance of the engine while simultaneously minimizing emissions and fuel consumption . In finance ( Gunantara , 2018 ) , one prefers a portfolio that maximizes the expected return while minimizing risk . Mathematically , in MOO we optimize M objectives f ( x ) = [ f1 ( x ) , f2 ( x ) , . . . , fM ( x ) ] ∈ RM : min f1 ( x ) , f2 ( x ) , ... , fM ( x ) ( 1 ) s.t . x ∈ Ω While we could set arbitrary weights for each objective to turn it into a single-objective optimization ( SOO ) problem , modern MOO methods aim to find the problem ’ s entire Pareto frontier : the set of solutions that are not dominated by any other feasible solutions1 ( see Fig . 1 for illustration ) . The Pareto frontier yields a global picture of optimal solution structures rather than focusing on one specific weighted combination of objectives . As a result , MOO is fundamentally different from SOO . Instead of focusing on a single optimal solution , a strong MOO optimizer should cover the search space broadly to explore the Pareto frontier . Popular quality indicators in MOO , such as hypervolume ( HV ) , capture this aspect by computing the volume of the currently estimated frontier . Specifically , given a reference point R ∈ RM , as shown in Fig . 1 ( a ) , the hypervolume of a finite approximate Pareto set P is the M-dimensional Lebesgue measure λM of the space dominated by P and bounded from below by R. That is , HV ( P , R ) = λM ( ∪|P|i=1 [ R , yi ] ) , where [ R , yi ] denotes the hyper-rectangle bounded by reference point R and yi . Consequently , the optimizer must consider the diversity of solutions in addition to their optimality . 1Here we define dominance y ≺f x as fi ( x ) ≤ fi ( y ) for all functions fi , and exists at least one i s.t . fi ( x ) < fi ( y ) , 1 ≤ i ≤ M . That is , solution x is always better than solution y , regardless of how the M objectives are weighted . While several previous works have proposed approaches to capture this diversity-optimality trade-off ( Deb et al. , 2002a ; Knowles , 2006 ; Igel et al. , 2007 ; Deb & Jain , 2014 ; Daulton et al. , 2020 ) , in this paper , we take a fundamentally different route by learning promising candidate regions from past explored samples . Ideally , to find the Pareto frontier in as few function evaluations as possible , we want to sample heavily in the Pareto optimal set ΩP , defined as the region of input vectors that corresponds to the Pareto frontier . One way to focus samples on ΩP is to gradually narrow the full search space down to the subregion containing ΩP via partitioning . For example , in the case of quadratic objective functions , ΩP can be separated from the non-optimal set Ω\ΩP via simple linear classifiers ( see Observation 1,2 ) . Motivated by these observations , we thus design LaMOO , a novel MOO meta-optimizer that progressively partitions regions into sub-regions and then focuses on sub-regions that are likely to contain Pareto-optimal regions , where existing solvers can help . Therefore , LaMOO is a meta-algorithm . Unlike cutting-plane methods ( Loganathan & Sherali , 1987 ; Hinder , 2018 ; Vieira & Lisboa , 2019 ) that leverage the ( sub ) -gradient of convex objectives as the cutting plane , with global optimality guarantees , LaMOO is data-driven : it leverages previous samples to build classifiers to learn the partition and focuses future samples in these promising regions . No analytical formula of objectives or their sub-gradients is needed . LaMOO is a multi-objective extension of recent works ( Wang et al. , 2020 ; Yang et al. , 2021 ) that also learn space partitions but for a single black-box objective . Empirically , LaMOO outperforms existing approaches on many benchmarks , including standard benchmarks in multi-objective black-box optimization , and real-world multi-objective problems like neural architecture search ( NAS ) ( Cai et al. , 2019 ; 2020 ) and molecule design . For example , as a meta-algorithm , LaMOO combined with CMA-ES as an inner routine requires only 62.5 % , 8 % , and 29 % as many samples to reach the same hypervolume as the original CMA-ES ( Igel et al. , 2007a ) in BraninCurrin ( Belakaria et al. , 2019 ) , VehicleSafety ( Liao et al. , 2008 ) and Nasbench201 ( Dong & Yang , 2020 ) , respectively . On average , compared to qEHVI , LaMOO uses 50 % samples to achieve the same performance in these problems . In addition , LaMOO with qEHVI ( Daulton et al. , 2020 ) and CMA-ES require 71 % and 31 % fewer samples on average , compared to naive qEHVI and CMA-ES , to achieve the same performance in molecule discovery . 2 RELATED WORK . Bayesian Optimization ( BO ) ( Zitzler et al. , 2003 ; Knowles , 2006 ; Ponweiser et al. , 2008 ; Couckuyt et al. , 2014 ; Paria et al. , 2018 ; Yang et al. , 2019 ; Daulton et al. , 2020 ) is a popular family of methods to optimize black-box single and multi-objectives . Using observed samples , BO learns a surrogate model f̂ ( x ) , search for new promising candidates based on acquisition function built on f̂ ( x ) , and query the quality of these candidates with the ground truth black-box objective ( s ) . In multi-objective Bayesian optimization ( MOBO ) , most approaches leverage Expected Hypervolume Improvement ( EHVI ) as their acquisition function ( Zitzler et al. , 2003 ; Couckuyt et al. , 2014 ; Yang et al. , 2019 ) , since finding the Pareto frontier is equivalent to maximizing the hypervolume given a finite search space ( Fleischer , 2003 ) . There are methods ( Knowles , 2006 ; Ponweiser et al. , 2008 ; Paria et al. , 2018 ) that use different acquisition functions like expected improvement ( Jones et al. , 1998 ) and Thompson sampling ( Thompson , 1933 ) . EVHI is computationally expensive : its cost increases exponentially with the number of objectives . To address this problem , qEHVI ( Daulton et al. , 2020 ) accelerates optimization by computing EHVI in parallel , and has become the state-of-the-art MOBO algorithm . In this paper , we leverage qEHVI as a candidate inner solver in our proposed LaMOO algorithm . Evolutionary algorithms ( EAs ) ( Deb et al. , 2002a ; Igel et al. , 2007a ; Zhang & Li , 2007 ; Beume et al. , 2007 ; Fang et al. , 2018 ) are also popular methods for MOO tasks . One category of MOOEAs ( Srinivas & Deb , 1994 ; Deb et al. , 2002a ; Deb & Jain , 2014 ) leverages Pareto dominance to simultaneously optimize all objectives . A second category ( e.g. , ( Zhang & Li , 2007 ) ) decomposes a multi-objective optimization problem into a number of single-objective sub-problems , converting a difficult MOO into several SOOs . Another category is quality indicator-based methods , such as ( Beume et al. , 2007 ) and ( Igel et al. , 2007a ) . They scalarize the current Pareto frontier using quality indicators ( e.g. , HV ) and transfer a MOO to a SOO . New samples are generated by crossover and mutation operations from existing ones . However the drawbacks of non-quality indicator-based methods ( i.e. , the first two categories ) can be not overlooked . Specifically , for MOO with many objectives , NSGA-II ( Deb et al. , 2002a ) easily gets stuck in a dominance resistant solution ( Pang et al. , 2020 ) which is far from the true Pareto frontier . while MOEA/D perform better in many-objective problems but specifying the weight vector for problems with unknown Pareto front is the main obstacle ( Deb & Jain , 2014 ) . In addition , A * search based algorithms are also considered to be extended to MOO ( Stewart & White , 1991 ; Tung Tung & Lin Chew , 1992 ; De la Cruz et al. , 2005 ) . Quality Indicators . Besides hypervolume , there are several other quality indicators ( Van Veldhuizen & Lamont , 1998 ; Zitzler et al. , 2000 ; Bosman & Thierens , 2003 ) for evaluating sample quality , which can be used to scalarize the MOO to SOO . The performance of a quality indicator can be evaluated by three metrics ( Deng et al. , 2007 ; Li et al. , 2014 ) , including convergence ( closeness to the Pareto frontier ) , uniformity ( the extent of the samples satisfy the uniform distribution ) , and spread ( the extent of the obtained approximate Pareto frontier ) . Sec . B specifically illustrates the merits of each quality indicator . HyperVolume is the only metric we explored that can simultaneously satisfy the evaluation of convergence , uniformity , and spread without the knowledge of the true Pareto frontier while it may suffer from expensive calculation in many-objective problems . Therefore , throughout this work , we use HV to evaluate the optimization performance of different algorithms . 3 LEARNING SPACE PARTITIONS : A THEORETICAL UNDERSTANDING . Searching in high-dimensional space to find the optimal solution to a function is in general a challenging problem , especially when the function ’ s properties are unknown to the search algorithm . The difficulty is mainly due to the curse of dimensionality : to adequately cover a d-dimensional space , in general , an exponential number of samples are needed . For this , many works use a “ coarse-to-fine ” approach : partition the search space and then focusing on promising regions . Traditionally , manually defined criteria are used , e.g. , axis-aligned partitions ( Munos , 2011b ) , Voronoi diagrams ( Kim et al. , 2020 ) , etc . Recently , ( Wang et al. , 2019 ; 2020 ; Yang et al. , 2021 ) learn space partitions based on the data collected thus far , and show strong performance in NeurIPS black box optimization challenges ( Sazanovich et al . ; Kim et al. ) . On the other hand , there is little quantitative understanding of space partition . In this paper , we first give a formal theoretical analysis on why learning plays an important role in space-partition approaches for SOO . Leveraging our understanding of how space partitioning works , we propose LaMOO which empirically outperforms existing SoTA methods on multiple MOO benchmarks . 3.1 PROBLEM SETTING . Intuitively , learning space partitions will yield strong performance if the classifier can determine which regions are promising given few data points . We formalize this intuition below and show why it is better than fixed and manually defined criteria for space partitioning . Consider the following sequential decision task . We have N samples in a discrete subset S0 and there exists one sample x∗ that achieves a minimal value of a scalar function f . Note that f can be any property we want , e.g. , in the Pareto optimal set . The goal is to construct a subset ST ⊆ S0 after T steps , so that ( 1 ) x∗ ∈ ST and ( 2 ) |ST | is as small as possible . More formally , we define the reward function r as the probability that we get x∗ by randomly sampling from the resulting subset ST : r : = 1 |ST | P ( x∗ ∈ ST ) ( 2 ) It is clear that 0 ≤ r ≤ 1. r = 1 means that we already found the optimal sample x∗ . Here we use discrete case for simplicity and leave continuous case ( i.e. , partitioning a region Ω0 instead of a discrete set S0 ) to future work . Note N could be large , so here we consider it infeasible to enumerate S0 to find x∗ . However , sampling from S0 , as well as comparing the quality of sampled solutions are allowed . An obvious baseline is to simply set ST : = S0 , then rb = N−1 . Now the question is : can we do better ? Here we seek help from the following oracle : Definition 1 ( ( α , η ) -Oracle ) . Given a subset S that contains x∗ , after taking k samples from S , the oracle can find a good subset Sgood with |Sgood| ≤ |S|/2 and P ( x∗ ∈ Sgood|x∗ ∈ S ) ≥ 1− exp ( − k η|S|α ) ( 3 ) Lemma 1 . The algorithm to uniformly draw k samples in S , pick the best and return is a ( 1 , 1 ) -oracle . See Appendix for proof . Note that a ( 1 , 1 ) -oracle is very weak , and is of little use in obtaining higher reward r. We typically hope for an oracle with smaller α and η ( i.e. , both smaller than 1 ) . Intuitively , such oracles are more sample-efficient : with few samples , they can narrow down the region containing the optimal solution x∗ with high probability . Note that α < 1 corresponds to semi-parametric models . In these cases , the oracle has generalization property : with substantially fewer samples than N ( i.e. , on the order of Nα ) , the oracle is able to put the optimal solution x∗ on the right side . In its extreme case when α = 0 ( or parametric models ) , whether we classify the optimal solution x∗ on the correct side only depends on the absolute number of samples collected in S , and is independent of its size . For example , if the function to be optimized is linear , then with d+ 1 samples , we can completely characterize the property of all |S| samples . Relation with cutting plane . Our setting can be regarded as a data-driven extension of cutting plane methods ( Loganathan & Sherali , 1987 ; Vieira & Lisboa , 2019 ; Hinder , 2018 ) in optimization , in which a cutting plane is found at the current solution to reduce the search space . For example , if f is convex and its gradient∇f ( x ) is available , then we can set Sgood : = { x : ∇f ( x0 ) > ( x− x0 ) ≤ 0 , x ∈ S0 } , since for any x ∈ S0 \ Sgood , convexity gives f ( x ) ≥ f ( x0 ) + ∇f ( x0 ) > ( x − x0 ) > f ( x0 ) and thus x is not better than current x0 . However , the cutting plane method relies on certain function properties like convexity . In contrast , learning space partition can leverage knowledge about the function forms , combined with observed samples so far , to better partition the space .

Summary: This paper presents a learning space partitions-based multi-objective optimization framework by using Monte Carlo tree search and an innovatively proposed metric, i.e., dominance number. Solid theoretical analysis on single-objective optimization (SOO) and some observation on multi-objective optimization (MOO) for the space partitions are given. Extensive experiments are conducted based on synthetic functions, vehicle safety problem, Nasbench201 neural architecture search problem, and molecular design. Comparative results with other baselines show the superiority and effectiveness of the proposed LaMOO.