Preference Conditioned Neural Multi-objective Combinatorial Optimization

1 INTRODUCTION . Many real-world applications can be modeled as different multiobjective combinatorial optimization ( MOCO ) problems ( Ehrgott & Gandibleux , 2000 ) . The multiobjective traveling salesman problem ( MOTSP ) ( Lust & Teghem , 2010a ) , multiobjective vehicle routing problem ( MOVRP ) ( Jozefowiez et al. , 2008 ) and multiobjective knapsack problem ( MOKP ) ( Bazgan et al. , 2009 ) are three widely studied problems in the research community . These problems have multiple conflicting objectives to optimize , and no single solution can optimize all the objectives at the same time . Instead , there is a set of Pareto solutions with different optimal trade-offs among the objectives . It is very challenging to find all exact Pareto optimal solutions for a MOCO problem . Finding only one Pareto solution would be already NP-hard for many problems ( Ehrgott & Gandibleux , 2000 ) , and the number of Pareto solutions could be exponentially large to the problem size ( Ehrgott , 2005 ; Herzel et al. , 2021 ) . The decision-maker ’ s preference among different objectives is usually unknown in advance , making it hard to reduce the problem into a single-objective one . Over the past several decades , many methods have been developed to find an approximate Pareto set for different MOCO problems within a reasonable computational time . These methods often need carefully handcrafted and specialized heuristics for each problem , which can be very labor-intensive in practice . In many real-world applications , the practitioners need to repeatedly solve different problems with similar characteristics , of which the problem instances can be easily obtained or generated ( Bengio et al. , 2020 ) . It is desirable to learn the patterns behind these problem instances explicitly or implicitly to design efficient algorithms ( Cappart et al. , 2021a ) . Machine learning techniques can be naturally served for this purpose . Some learning-based methods have been recently proposed for solving single-objective combinatorial optimization problems ( Bengio et al. , 2020 ; Vesselinova et al. , 2020 ; Mazyavkina et al. , 2021 ; Cappart et al. , 2021a ) . In this work , we extend the learning-based method to solve MOCO problems in a principled way as shown in Figure 1 . Our main contributions include : • We propose a novel neural combinatorial optimization method to approximate the whole Pareto set with infinite trade-offs via a single model , which allows decision-makers to directly generate any preferred trade-off solution without extra search . This novel and powerful property is not supported by any other learning-based or heuristics-based methods . • We develop an efficient end-to-end reinforcement learning algorithm to train the single model for all different preferences simultaneously , and a simple yet powerful active adaption method to handle out-of-distribution problem instances . • We conduct comprehensive experiments on MOTSP , MOVR and MOKP problems with different settings . The results show that our proposed method can successfully approximate the Pareto sets for different problems in an efficient way . It also significantly outperforms other methods in solution quality , speed , and/or model efficiency . 2 BACKGROUND AND RELATED WORK . Multiobjective Combinatorial Optimization ( MOCO ) . MOCO has been attracting much research effort from different communities over the past several decades ( Sawaragi et al. , 1985 ; Wallenius et al. , 2008 ; Herzel et al. , 2021 ) . There are two main approaches to tackle the MOCO problems : the exact method and the approximation method ( Ehrgott , 2005 ) . Exact methods could be impractically costly when , as it often happens , the MOCO problem is NP-hard and the number of Pareto optimal solutions is huge ( Florios & Mavrotas , 2014 ) . For this reason , many heuristics ( Jaszkiewicz , 2002 ; Ehrgott & Gandibleux , 2008 ) and approximation methods ( Papadimitriou & Yannakakis , 2000 ; Herzel et al. , 2021 ) have been developed to find a moderately sized set of approximated Pareto solutions with a reasonable amount of computational budget . However , these methods usually depend on carefully handcrafted designs for each specific problem ( Ehrgott & Gandibleux , 2000 ) , of which the effort would be nontrivial for a new problem in real-world applications . Machine Learning for Combinatorial Optimization . According to Bengio et al . ( 2020 ) , there are three main learning-based approaches for CO : learning to configure algorithms ( Kruber et al. , 2017 ; Bonami et al. , 2018 ) , learning alongside the algorithms ( Lodi & Zarpellon , 2017 ; Gasse et al. , 2019 ; Chen & Tian , 2019 ; Kool et al. , 2021 ) , and learning to directly predict the solutions ( Nowak et al. , 2018 ; Emami & Ranka , 2018 ; Larsen et al. , 2018 ) . Neural combinatorial optimization ( NCO ) belongs to the last category where the training model directly predicts a good solution for a given problem instance . Vinyals et al . ( 2015 ) propose the pointer network to sequentially construct a solution for TSP problem . Bello et al . ( 2017 ) make a critical improvement to use reinforcement learning to train the model , eliminating the impractical optimal solutions collection for NP-hard problems . Further improvements on model structure and training procedure have been proposed in the past few years ( Nazari et al. , 2018 ; Deudon et al. , 2018 ; Kool et al. , 2019 ; Joshi et al. , 2020 ) , especially with graph neural networks ( GNNs ) ( Dai et al. , 2017 ; Li et al. , 2018 ; Joshi et al. , 2019 ; Dwivedi et al. , 2020 ; Drori et al. , 2020 ) . Some works have recently been proposed to design more powerful training strategies , such as optimization with multiple optimum ( Kwon et al. , 2020 ) , unsupervised learning ( Karalias & Loukas , 2020 ) and curriculum learning schemes ( Lisicki et al. , 2020 ) . Neural MOCO . The current learning-based methods are mostly for solving single-objective combinatorial problems . Recently , a few attempts have been made to solve MOCO problems ( Li et al. , 2020 ; Wu et al. , 2020 ; Zhang et al. , 2021b ; a ) . They first decompose a MOCO problem into multiple single-objective scalarized subproblems , and then build a set of models for each subproblem . However , since the number of Pareto solutions would be exponentially large ( Ehrgott , 2005 ) , the required number of models would be huge for finding the whole Pareto set . This work proposes a single preference-based model for solving MOCO problems , where the decision-makers can obtain arbitrary trade-off solutions . The proposed single neural MOCO solver could be much easier to use in a real-world system ( Veličković & Blundell , 2021 ) , rather than a large set of separate models . Our method ’ s model efficiency is crucial to truly extend the NCO approach to solve MOCO problems . 3 PROBLEM FORMULATION . 3.1 MULTIOBJECTIVE COMBINATORIAL OPTIMIZATION . A multiobjective combinatorial optimization ( MOCO ) problem can be defined as follows : min x∈X F ( x ) = ( f1 ( x ) , f2 ( x ) , . . . , fm ( x ) ) , ( 1 ) where X is a discrete search space , and F ( x ) = ( f1 ( x ) , . . . , fm ( x ) ) is an m-objective vector . Since the individual objectives often conflict with each other , no single solution can optimize all of them at the same time . Thus , practitioners are interested in Pareto optimal solutions , defined as follows . Definition 1 ( Pareto Dominance ) . Let xa , xb ∈ X , xa is said to dominate xb ( xa ≺ xb ) if and only if fi ( xa ) ≤ fi ( xb ) , ∀i ∈ { 1 , ... , m } and fj ( xa ) < fj ( xb ) , ∃j ∈ { 1 , ... , m } . Definition 2 ( Pareto Optimality ) . A solution x∗ ∈ X is a Pareto optimal solution if there does not exist x̂ ∈ X such that x̂ ≺ x∗ . The set of all the Pareto solutions is called the Pareto set , and the image of the Pareto set in the objective space is called the Pareto front . Each Pareto solution represents an optimal trade-off among the objectives , and it is impossible to further improve one of the objectives without deteriorating any other objectives . For many MOCO problems , the size of Pareto set would be exponentially large with respect to the input size ( e.g. , nodes in MOTSP ) , which makes finding the whole Pareto set intractable ( Herzel et al. , 2021 ) . Therefore , the current heuristic-based and learning-based methods all focus on finding a small subset of approximate Pareto solutions { x1 , x2 , · · · , xp } ( e.g. , p = 101 ) . The decision-makers can only select solutions from this fixed small set , which is unlikely to contain their preferred solutions precisely . 3.2 PREFERENCE-BASED SCALARIZATION . Preference-based scalarization is a widely-used approach to solve MOCO problems ( Ehrgott , 2005 ) . It decomposes a given multiobjective problem into several subproblems with different preferences . By solving all subproblems , we can obtain a set of approximated Pareto solutions . For an mobjective problems , we can define a preference vector λ ∈ Rm satisfies λi ≥ 0 and ∑m i=1 λi = 1 such that λi is the preference for the i-th objective . Weight-Sum Scalarization is the most straightforward approach : min x∈X gws ( x|λ ) = min x∈X ∑m i=1 λifi ( x ) , ( 2 ) where gws ( x|λ ) is a single-objective subproblem with preference λ . The decision-makers can assign their preferences among the objectives , and obtain different subproblems . However , the weightedsum method can only find solutions on the convex hull of the Pareto set ( Ehrgott , 2005 ) . Weighted-Tchebycheff ( Weighted-TCH ) Scalarization is an alternative : min x∈X gtch ( x|λ ) = min x∈X max 1≤i≤m { λi|fi ( x ) − ( z∗i − ε ) | } , ( 3 ) where z∗i is the ideal value for objective fi ( x ) ( e.g. , the lower bound ) , and u ∗ i = z ∗ i − ε is a utopia value with small positive component ε . According to Choo & Atkins ( 1983 ) , any Pareto optimal solution could be an optimal solution of problem ( 3 ) with a specific ( but unknown ) preference λ . To find the whole Pareto set , it still needs to solve an exponentially huge number of subproblems . Drawbacks and Remarks . Although the scalarization methods are straightforward , some critical drawbacks can not be ignored . First of all , not all scalarized multiobjective problem is the same as its single-objective version ( Ehrgott , 2005 ) . For MOTSP , each weight-sum scalarized subproblem is a single objective TSP so that a powerful TSP solver can solve it . However , it is not the case for MOCVRP . Secondly , although the Tchebycheff scalarization has good properties , it involves min-max optimization , which is typically different from the original problem . For example , the Tchebycheff scalarized MOTSP is not a single-objective TSP and can not be solved by the state-ofthe-art single-objective solver developed in the current years . In our reinforcement learning based framework , we use the Tchebycheff scalarization to combine the rewards of different objectives . It is much more flexible to use , and does not have to consider the problem-specific condition .

Summary: This submission treats multi-objective combinatorial optimization problems and aims to approximate the pareto set. The idea is to build a single ML model that represents the pareto set by providing a pareto set solution for any desired trade-off. The model is build using reinforcement learning and may either be used to find singular solutions with fixed trade-off, or to approximate the pareto set with uniform samples. The authors prove that the pareto set is approximated well if individual trade-offs are approximated well.