A Two-Stage Neural-Filter Pareto Front Extractor and the need for Benchmarking

1 INTRODUCTION . Multi-Objective Optimization ( MOO ) problems arise frequently across diverse fields such as engineering ( Marler & Arora , 2004 ) , finance ( Tapia & Coello , 2007 ) , and supply chain management ( Trisna et al. , 2016 ) . Such problems share a common requirement to satisfy multiple competing objectives under a set of constraints imposed by physical or economic limits . A Pareto optimal solution Pareto ( 1906 ) for an MOO problem is defined as the solution point away from which no single objective can be improved without diminishing at least one other objective . A Pareto front is then defined as the set of all such optimal points that satisfy this definition . Since practical MOO problems contain competing objectives and additional domain specific constraints , a Pareto solver should be able to handle both functions and constraints . Moreover , problems in classification and recommendation are non-convex ( Hsieh et al. , 2015 ) , necessitating solvers that are robust for non-convex problems . Although many saddle point solvers have been proposed in the literature ( Benzi et al. , 2005 ; Benzi & Wathen , 2008 ) , they can not be generalized to handle constraints without specialized pre-conditioners designed to handle non-linear constraints . These existing solvers are based upon fixed-point iterations , returning one solution per run with specialized local initialization to generate an even spread of Pareto points across the feasible set of saddle points . Recently proposed Multi-Task Learning ( MTL ) solvers ( Sener & Koltun , 2018 ; Lin et al. , 2019a ; Mahapatra & Rajan , 2020 ; Ma et al. , 2020 ; Navon et al. , 2021 ) criticize Linear Scalarization ( LS ) since it retrieves a subset of the Pareto optimal set . MTL solvers claim to resolve this issue by decomposing the functional/variable domain ( into cones or rays ) while using LS in each of the subdomains . However , the stationary points of the LS problem ( now decomposed ) remain unchanged ( see Appendix B ) , and therefore any fixed-point iterative strategy will still retrieve only a subset of the Pareto optimal set . Additionally , the theorems presented in MTL works rely upon convexity assumption that are never justified through numerical experiments . Furthermore , a notable absence of benchmarks against known analytical forms makes it difficult to assess reported results , verify optimality , and A/B test alternative methods . This is in contrast to studies in Operations Research ( OR ) ( Das & Dennis , 1998 ; Ghane-Kanafi & Khorram , 2015 ; Pirouz & Khorram , 2016 ) in which such comparative benchmarking and verification is well established . Although accurate and verifiable , existing OR methods tend to generate Pareto points with low density and limited scalability , with compute times ranging from hours to days as variable dimensionality increases . While lack of formal guarantees for non-convex cases certainly does not preclude use of MTL methods for such problems , rigorous numerical evaluation becomes particularly important . However , the true Pareto front ( closed-form analytical solution ) is often unknown for real MOO problems , hence it is challenging to compare the accuracy of a Pareto solver on such problems . We follow the OR literature in advocating that the correctness of any proposed Pareto solver should first be tested on constructed benchmark problems with known analytic solutions . This is also consistent with broader ML community practice of first evaluating proposed methods in controlled synthetic or simulated conditions to verify correctness , before applying these systems on real datasets . In general , rigorously evaluating a new method on a diverse set of complicated benchmarking scenarios can provide valuable insights prior to venturing into the unknown and builds user trust in the model . We propose a two-stage framework : Hybrid Neural Pareto Front ( HNPF ) for extracting Pareto optimal solution sets . Stage 1 consists of an interpretable neural network that extracts a weak Pareto solution manifold as the output , given a dataset as input . Following this , Stage 2 is a low-cost Pareto filter to remove dominated points from the weak Pareto set . The network loss function uses a discriminator based on Fritz-John Conditions ( FJC ) to account for multiple objectives and constraints . An approximate weak Pareto manifold is extracted as a weighted output of the softmax function from the last layer of the network . The softmax activation classifies weak Pareto vs. non-Pareto data points . HNPF extracts this weak Pareto front as a continuous manifold approximated by the Stage-1 neural network . Numerical experiments show the computational efficiency of HNPF vs. OR approaches and accuracy comparisons against MTL methods . HNPF produces only Pareto points ( no false positives ) with an even spread and high density . Furthermore , HNPF is scalable ( compared to OR methods ) with both increasing dimensions of the variable domain , and the number of functions and constraints . Table 1 summarizes key properties of HNPF vs. existing methods . Our key contributions are : 1 . A neural manifold extraction strategy for weak Pareto front identification based on Fritz-John conditions as the discriminator , for both convex and non-convex scenarios , supporting constraints . 2 . The final neural net layer is interpretable as a continuous approximation of the weak Pareto mani- fold . The extracted manifold error is bounded below w.r.t . the true manifold upon convergence . 3 . The necessity of Pareto filter to remove dominated points from the weak Pareto set . Design of a computationally efficient Pareto filter to extract the non-dominated Pareto optimal set . 4 . Raising community awareness about existence of benchmarks from literature in OR field . 2 PROBLEM STATEMENT . A general MOO problem can formulated as : optimize F ( x ) = ( f1 ( x ) , f2 ( x ) , . . . , fk ( x ) ) ( 1 ) s.t . x 2 H = { x 2 Rn|G ( x ) = ( g1 ( x ) , g2 ( x ) , . . . , gm ( x ) 0 } in n variables ( x1 , . . . , xn ) , k objective functions ( f1 , . . . , fk ) , and m constraint functions ( g1 , . . . , gm ) . Here , H is the feasible set i.e . the set of input values x that satisfy the constraints G ( x ) . For a MOO problem there is typically no single global solution , and it is often necessary to determine a set of points that all fit a predetermined definition for an optimum . See Appendix A for definitions . 2.1 FRITZ JOHN CONDITIONS ( FJC ) . Let the objective and constraint function in Eq . 1 be differentiable once at a decision vector x̃⇤ 2 H. The Fritz-John necessary conditions ( FJC ) for x̃⇤ to be weak Pareto optimal is that vectors must exists for 0 2 Rk , 0 µ 2 Rm and ( , µ ) 6= ( 0 , 0 ) ( not identically zero ) s.t . the following holds : kX i=1 irfi ( x̃ ⇤ ) + mX j=1 µjrgj ( x̃ ⇤ ) = 0 , µjgj ( x̃ ⇤ ) = 0 , 8j = 1 , . . . , m ( 2 ) Gobbi et al . ( 2015 ) presented an L matrix form of FJC as follows : L = rF rG 0 G [ ( n + m ) ⇥ ( k + m ) ] ( 3 ) rFn⇥k = [ rf1 , . . . , rfk ] , rGn⇥m = [ rg1 , . . . , rgm ] , Gm⇥m = diag ( g1 , . . . , gm ) In L matrix , for x⇤ to be Pareto optimal , is to show the existence of 2 Rk+m in Eq . 2 such that L · = 0 s.t . L = L ( x̃⇤ ) , 0 , 6= 0 ( 4 ) The non-trivial solution ( not identically zero ) for Eq . 4 is : det ( LT L ) = 0 ( 5 ) Remark . Note that det ( L ) = 0 is equivalent to det ( LT L ) = 0 ( see Appendix G for derivation ) . The weak Pareto front is characterized by the set of points such that matrix L is low rank . This ensures that points identified are either inside the feasible set or at boundaries dictated by the constraints . For e.g . if µ1 = 0 for any i , then P i ifi = 0 must be satisfied for the corresponding internal point x ⇤ to be Pareto . Similarly if µ1 6= 0 , µj 6=1 = 0 in the aforementioned case , then g1 = 0 holds true for the corresponding boundary point x⇤ to be Pareto . All Pareto points satisfy rfi = 0 for at least one i whether they lie inside the feasible set or on the boundary , i.e . all points x⇤ need to be local optimizers ( stationary points ) 1 for at least one fi . The rank of the matrix LT L determines the dimension of the Pareto manifold . FJC written as det ( LT L ) is independent of the preference parameters i , µj . Thus , Eq . 5 is an oracle serving as a discriminator in HNPF to identify a weak Pareto front . Remark . For a multi-objective optimization ( MOO ) problem scalarization results in a singleobjective optimization ( SOO ) problem . However , the stationary points of this resulting SOO problem can only be a small subset of the weak Pareto solution set . The reader is referred to Appendix B for a visual explanation of stationarity w.r.t . an SOO problem arising from an MOO problem . 3 RELATED WORKS . Generic and Enhanced Scalarization : One common approach is to convert an MOO problem into a Single Objective Optimization ( SOO ) problem via Linear Scalarization ( LS ) . These include : Balashankar et al . ( 2019 ) ; Lin et al . ( 2019a ) ; Martinez et al . ( 2020 ) ; Valdivia et al . ( 2021 ) ; Wei & Niethammer ( 2020 ) . Enhanced scalarization approaches ( Das & Dennis , 1998 ; Ghane-Kanafi & Khorram , 2015 ; Pirouz & Khorram , 2016 ) rely upon localization of the objective space to handle nonconvex functions and constraints . Although accurate and complete , these approaches suffer from low computational scalability and low density of Pareto points on the solution manifold . For example , the 30 dimensional benchmark ( Case V in Section 5 ) shows enhanced scalarization methods ( mCHIM and PK ) generating a Pareto set in approximately 18 hours . Multi Task Learning : MTL methods rely upon a utopia/ideal point ( similar to Das & Dennis ( 1998 ) ) but are shown to be scalable for high-dimensional MOO problems . MOOMTL ( Sener & Koltun , 2018 ) uses a multi-gradient descent approach , but does not guarantee an even spread of solution points along the Pareto front . PMTL ( Lin et al. , 2019a ) attempts to address this issue by dividing the functional domain into equal spaced cones , with increasing computational cost as the number of cones increase . EPO ( Mahapatra & Rajan , 2020 ) uses preference rays along specified weights to find Pareto points in the vicinity of the rays . EPSE ( Ma et al. , 2020 ) uses a combination Hessian of the functions and Krylov subspace to find Pareto solutions . PHN ( Navon et al. , 2021 ) is initialized either by LS or EPO and is the first neural framework that can approximate the front as a neural manifold . 1A stationary point w.r.t . gradient descent fixed point iteration is the set of points corresponding to the local or global minima , maxima or saddle points of an objective function . The reader is further referred to Appendix O for a detailed review of these methods including other from the Bayesian , Genetic and Fairness literature . Current Limitations : Firstly , these aforementioned point-based approaches extract one solution point at a time , given the optimization problem converges . However , for practical applications , with non-convex objectives and constraints , ensuring optimality is non-trivial . Secondly , multiple runs with different trade-off parameters must be performed in order to extract the weak Pareto solution set , resulting in substantial computational overhead . Finally , the Pareto solution set can still form a non-convex manifold even when the objectives are convex due to the presence of non-convex constraints ( Case III in Section 5 ) . These challenges prove to be major obstacles in the deployment of MOO solution methods as practical tools for Pareto set extraction .

Summary: This paper has two contributions. First it argues for the need to benchmark multi-objective problems  using analytic functions for convex and non-convex conditions. Second, it presents a two-stage method for finding the Pareto front.