A Two-Stage Neural-Filter Pareto Front Extractor and the need for Benchmarking

1 INTRODUCTION . Multi-Objective Optimization ( MOO ) problems arise frequently across diverse fields such as engineering ( Marler & Arora , 2004 ) , finance ( Tapia & Coello , 2007 ) , and supply chain management ( Trisna et al. , 2016 ) . Such problems share a common requirement to satisfy multiple competing objectives under a set of constraints imposed by physical or economic limits . A Pareto optimal solution Pareto ( 1906 ) for an MOO problem is defined as the solution point away from which no single objective can be improved without diminishing at least one other objective . A Pareto front is then defined as the set of all such optimal points that satisfy this definition . Since practical MOO problems contain competing objectives and additional domain specific constraints , a Pareto solver should be able to handle both functions and constraints . Moreover , problems in classification and recommendation are non-convex ( Hsieh et al. , 2015 ) , necessitating solvers that are robust for non-convex problems . Although many saddle point solvers have been proposed in the literature ( Benzi et al. , 2005 ; Benzi & Wathen , 2008 ) , they can not be generalized to handle constraints without specialized pre-conditioners designed to handle non-linear constraints . These existing solvers are based upon fixed-point iterations , returning one solution per run with specialized local initialization to generate an even spread of Pareto points across the feasible set of saddle points . Recently proposed Multi-Task Learning ( MTL ) solvers ( Sener & Koltun , 2018 ; Lin et al. , 2019a ; Mahapatra & Rajan , 2020 ; Ma et al. , 2020 ; Navon et al. , 2021 ) criticize Linear Scalarization ( LS ) since it retrieves a subset of the Pareto optimal set . MTL solvers claim to resolve this issue by decomposing the functional/variable domain ( into cones or rays ) while using LS in each of the subdomains . However , the stationary points of the LS problem ( now decomposed ) remain unchanged ( see Appendix B ) , and therefore any fixed-point iterative strategy will still retrieve only a subset of the Pareto optimal set . Additionally , the theorems presented in MTL works rely upon convexity assumption that are never justified through numerical experiments . Furthermore , a notable absence of benchmarks against known analytical forms makes it difficult to assess reported results , verify optimality , and A/B test alternative methods . This is in contrast to studies in Operations Research ( OR ) ( Das & Dennis , 1998 ; Ghane-Kanafi & Khorram , 2015 ; Pirouz & Khorram , 2016 ) in which such comparative benchmarking and verification is well established . Although accurate and verifiable , existing OR methods tend to generate Pareto points with low density and limited scalability , with compute times ranging from hours to days as variable dimensionality increases . While lack of formal guarantees for non-convex cases certainly does not preclude use of MTL methods for such problems , rigorous numerical evaluation becomes particularly important . However , the true Pareto front ( closed-form analytical solution ) is often unknown for real MOO problems , hence it is challenging to compare the accuracy of a Pareto solver on such problems . We follow the OR literature in advocating that the correctness of any proposed Pareto solver should first be tested on constructed benchmark problems with known analytic solutions . This is also consistent with broader ML community practice of first evaluating proposed methods in controlled synthetic or simulated conditions to verify correctness , before applying these systems on real datasets . In general , rigorously evaluating a new method on a diverse set of complicated benchmarking scenarios can provide valuable insights prior to venturing into the unknown and builds user trust in the model . We propose a two-stage framework : Hybrid Neural Pareto Front ( HNPF ) for extracting Pareto optimal solution sets . Stage 1 consists of an interpretable neural network that extracts a weak Pareto solution manifold as the output , given a dataset as input . Following this , Stage 2 is a low-cost Pareto filter to remove dominated points from the weak Pareto set . The network loss function uses a discriminator based on Fritz-John Conditions ( FJC ) to account for multiple objectives and constraints . An approximate weak Pareto manifold is extracted as a weighted output of the softmax function from the last layer of the network . The softmax activation classifies weak Pareto vs. non-Pareto data points . HNPF extracts this weak Pareto front as a continuous manifold approximated by the Stage-1 neural network . Numerical experiments show the computational efficiency of HNPF vs. OR approaches and accuracy comparisons against MTL methods . HNPF produces only Pareto points ( no false positives ) with an even spread and high density . Furthermore , HNPF is scalable ( compared to OR methods ) with both increasing dimensions of the variable domain , and the number of functions and constraints . Table 1 summarizes key properties of HNPF vs. existing methods . Our key contributions are : 1 . A neural manifold extraction strategy for weak Pareto front identification based on Fritz-John conditions as the discriminator , for both convex and non-convex scenarios , supporting constraints . 2 . The final neural net layer is interpretable as a continuous approximation of the weak Pareto mani- fold . The extracted manifold error is bounded below w.r.t . the true manifold upon convergence . 3 . The necessity of Pareto filter to remove dominated points from the weak Pareto set . Design of a computationally efficient Pareto filter to extract the non-dominated Pareto optimal set . 4 . Raising community awareness about existence of benchmarks from literature in OR field . 2 PROBLEM STATEMENT . A general MOO problem can formulated as : optimize F ( x ) = ( f1 ( x ) , f2 ( x ) , . . . , fk ( x ) ) ( 1 ) s.t . x 2 H = { x 2 Rn|G ( x ) = ( g1 ( x ) , g2 ( x ) , . . . , gm ( x ) 0 } in n variables ( x1 , . . . , xn ) , k objective functions ( f1 , . . . , fk ) , and m constraint functions ( g1 , . . . , gm ) . Here , H is the feasible set i.e . the set of input values x that satisfy the constraints G ( x ) . For a MOO problem there is typically no single global solution , and it is often necessary to determine a set of points that all fit a predetermined definition for an optimum . See Appendix A for definitions . 2.1 FRITZ JOHN CONDITIONS ( FJC ) . Let the objective and constraint function in Eq . 1 be differentiable once at a decision vector x̃⇤ 2 H. The Fritz-John necessary conditions ( FJC ) for x̃⇤ to be weak Pareto optimal is that vectors must exists for 0 2 Rk , 0 µ 2 Rm and ( , µ ) 6= ( 0 , 0 ) ( not identically zero ) s.t . the following holds : kX i=1 irfi ( x̃ ⇤ ) + mX j=1 µjrgj ( x̃ ⇤ ) = 0 , µjgj ( x̃ ⇤ ) = 0 , 8j = 1 , . . . , m ( 2 ) Gobbi et al . ( 2015 ) presented an L matrix form of FJC as follows : L = rF rG 0 G [ ( n + m ) ⇥ ( k + m ) ] ( 3 ) rFn⇥k = [ rf1 , . . . , rfk ] , rGn⇥m = [ rg1 , . . . , rgm ] , Gm⇥m = diag ( g1 , . . . , gm ) In L matrix , for x⇤ to be Pareto optimal , is to show the existence of 2 Rk+m in Eq . 2 such that L · = 0 s.t . L = L ( x̃⇤ ) , 0 , 6= 0 ( 4 ) The non-trivial solution ( not identically zero ) for Eq . 4 is : det ( LT L ) = 0 ( 5 ) Remark . Note that det ( L ) = 0 is equivalent to det ( LT L ) = 0 ( see Appendix G for derivation ) . The weak Pareto front is characterized by the set of points such that matrix L is low rank . This ensures that points identified are either inside the feasible set or at boundaries dictated by the constraints . For e.g . if µ1 = 0 for any i , then P i ifi = 0 must be satisfied for the corresponding internal point x ⇤ to be Pareto . Similarly if µ1 6= 0 , µj 6=1 = 0 in the aforementioned case , then g1 = 0 holds true for the corresponding boundary point x⇤ to be Pareto . All Pareto points satisfy rfi = 0 for at least one i whether they lie inside the feasible set or on the boundary , i.e . all points x⇤ need to be local optimizers ( stationary points ) 1 for at least one fi . The rank of the matrix LT L determines the dimension of the Pareto manifold . FJC written as det ( LT L ) is independent of the preference parameters i , µj . Thus , Eq . 5 is an oracle serving as a discriminator in HNPF to identify a weak Pareto front . Remark . For a multi-objective optimization ( MOO ) problem scalarization results in a singleobjective optimization ( SOO ) problem . However , the stationary points of this resulting SOO problem can only be a small subset of the weak Pareto solution set . The reader is referred to Appendix B for a visual explanation of stationarity w.r.t . an SOO problem arising from an MOO problem . 3 RELATED WORKS . Generic and Enhanced Scalarization : One common approach is to convert an MOO problem into a Single Objective Optimization ( SOO ) problem via Linear Scalarization ( LS ) . These include : Balashankar et al . ( 2019 ) ; Lin et al . ( 2019a ) ; Martinez et al . ( 2020 ) ; Valdivia et al . ( 2021 ) ; Wei & Niethammer ( 2020 ) . Enhanced scalarization approaches ( Das & Dennis , 1998 ; Ghane-Kanafi & Khorram , 2015 ; Pirouz & Khorram , 2016 ) rely upon localization of the objective space to handle nonconvex functions and constraints . Although accurate and complete , these approaches suffer from low computational scalability and low density of Pareto points on the solution manifold . For example , the 30 dimensional benchmark ( Case V in Section 5 ) shows enhanced scalarization methods ( mCHIM and PK ) generating a Pareto set in approximately 18 hours . Multi Task Learning : MTL methods rely upon a utopia/ideal point ( similar to Das & Dennis ( 1998 ) ) but are shown to be scalable for high-dimensional MOO problems . MOOMTL ( Sener & Koltun , 2018 ) uses a multi-gradient descent approach , but does not guarantee an even spread of solution points along the Pareto front . PMTL ( Lin et al. , 2019a ) attempts to address this issue by dividing the functional domain into equal spaced cones , with increasing computational cost as the number of cones increase . EPO ( Mahapatra & Rajan , 2020 ) uses preference rays along specified weights to find Pareto points in the vicinity of the rays . EPSE ( Ma et al. , 2020 ) uses a combination Hessian of the functions and Krylov subspace to find Pareto solutions . PHN ( Navon et al. , 2021 ) is initialized either by LS or EPO and is the first neural framework that can approximate the front as a neural manifold . 1A stationary point w.r.t . gradient descent fixed point iteration is the set of points corresponding to the local or global minima , maxima or saddle points of an objective function . The reader is further referred to Appendix O for a detailed review of these methods including other from the Bayesian , Genetic and Fairness literature . Current Limitations : Firstly , these aforementioned point-based approaches extract one solution point at a time , given the optimization problem converges . However , for practical applications , with non-convex objectives and constraints , ensuring optimality is non-trivial . Secondly , multiple runs with different trade-off parameters must be performed in order to extract the weak Pareto solution set , resulting in substantial computational overhead . Finally , the Pareto solution set can still form a non-convex manifold even when the objectives are convex due to the presence of non-convex constraints ( Case III in Section 5 ) . These challenges prove to be major obstacles in the deployment of MOO solution methods as practical tools for Pareto set extraction .

Summary: This work proposes a Hybrid Neural Pareto Front (HNPF) framework to solve multi-objective optimization problems. The proposed method needs to first generate a huge number of feasible solutions to cover the whole decision space (e.g., random sampling in this work), and then use a two-stage approach to select the Pareto optimal solutions. In the first stage, it builds a Fritz-John Condition (FJC) based neural network to verify the weak Pareto optimality for given solutions. In the second stage, it further filters the truly strong Pareto optimal subset from the obtained solutions. Experiments have been conducted on several low-dimensional problems to validate the proposed method's performance.