Pareto Frontier Approximation Network (PA-Net) Applied to Multi-objective TSP

1 INTRODUCTION . Travelling Salesman Problem ( TSP ) is a popular sequencing problem . TSP aims at finding a sequence or a tour of visiting each node in a given graph and finally returning to the starting node such that the overall cost is minimized . TSP and its variants are widely used in robotics for applications like path planning for UAVs ( Xu & Che , 2019 ) , multi-robot path planning ( Yu et al. , 2002 ) , task allocation for robotic manipulators ( Zacharia & Aspragathos , 2005 ) , coverage planning ( Bormann et al. , 2018 ) etc . We intend to use multi-objective travelling salesman problem ( MOTSP ) for application of coverage planning . Algorithms for coverage planning generates trajectories for robots that can cover a given area ( Bormann et al. , 2018 ) . Area coverage is used in robotic applications like cleaning robots , surveillance etc . Grid based TSP planners ( Bormann et al. , 2018 ) , segment a given map into multiple cells , and generate a coverage pattern for each cell.The optimal visiting order for these cells is generated by solving TSP that minimizes the length of the tour . Now imagine a scenario where a robot has to visit these cells and the order is dependent on multiple objectives such as tour length , priority order of the cells etc . So MOTSP would be a more appropriate choice in such a scenario . There is a wide variety of algorithms ranging from exact methods to evolution based methods to solve MOTSP ( Lust & Teghem , 2010 ) . Evolutionary algorithms like non-dominated sorting genetic algorithm-II ( NSGA-II ) ( Beirigo & dos Santos , 2016 ) and multi-objective evolutionary algorithm ( MOEA/D ) ( Peng et al. , 2009 ) are a popular choice of methods to tackle MOTSP and other multi objective optimization problems . Many algorithms also use evolutionary algorithms coupled with local search heuristics ( Jaszkiewicz , 2002 ; Ke et al. , 2014 ; Cai et al. , 2014 ) . In practice , these evolutionary based methods suffer in performance and computation time with an increase in the scale of the problem ( Zhang et al. , 2016 ) . We intend to address these issues using Deep Reinforcement Learning ( DRL ) . Contribution : In this work , we present Pareto frontier approximation network ( PA-Net ) that generates an approximation of a set of Pareto optimal tours for MOTSP . We use PA-Net to generate MOTSP tours for coverage planning . Our main contributions are : ( 1 ) The drawback of existing ap- proaches ( Li et al. , 2020 ) is costly training of separate networks for different preferences in objective space . Our method avoids this costly training through the use of a single network , while providing generalizability over different preferences in objective space . ( 2 ) PA-Net can generate solutions for large number of preferences , which contributes to a dense approximation of Pareto front . Our network has competitive results on various metrics evaluating the quality of Pareto front and faster inference times . ( 3 ) Our design can be easily extended to generate a set of Pareto optimal solutions for other multi-objective reinforcement learning and Multi-Objective Optimization ( MOO ) tasks . Related Work : Sequencing problems are a subset of Combinatorial Optimization ( CO ) where the decision variables are discrete . Most CO problems are NP-Hard , as a result state-of-the-art algorithms rely on handcrafted heuristics for making decisions that are otherwise too expensive to compute or mathematically not well-defined . Recently , researchers are addressing these issues using deep learning and machine learning ( ML ) ( Bengio et al. , 2020 ; Vesselinova et al. , 2020 ; Mazyavkina et al. , 2020 ) . Many recent works have been dedicated to solving TSP and other sequencing problems using deep networks . Google Brain ’ s Pointer Network ( Ptr-Net ) ( Vinyals et al. , 2015 ) learns the conditional probability of an output sequence of elements that are discrete tokens corresponding to positions in an input sequence . They used Ptr-Net to solve Euclidean TSP ( and other CO problems ) in an end to end fashion , where the solutions from classical methods are used as baselines for training . Similarly , TSP was solved using 2D graph conv-nets followed by beam search ( Joshi et al. , 2019 ) . RL was used to solve various combinatorial optimization problems in ( Bello et al. , 2016 ) . Their network uses an RNN based encoder and a Ptr-Net . They trained their network using policy gradient . Similar methodology was adopted in ( Deudon et al. , 2018 ) . They used transformers as the encoder . Kool et al . ( 2018 ) proposed an attention based network which used a baseline based on greedy rollouts . Their network outperformed other deep-learning based solvers . While these methods generated competitive results , however , they still lag in performance in comparison to TSP solvers like OR-Tools and Concorde . Finding Pareto optimal solutions has been studied in deep learning literature for various multiobjective tasks . In supervised learning tasks , many works focussed on multi-objective classification tasks ( Sener & Koltun , 2018 ; Lin et al. , 2019 ; Mahapatra & Rajan , 2020 ; Ruchte & Grabocka , 2021 ; Navon et al. , 2021 ) . Similarly , many multi-objective RL methods have been to solve multi-objective MDPs ( Roijers et al. , 2013 ; Parisi et al. , 2014 ) . Some works in RL train many single policy networks to approximate Pareto front ( Vamplew et al. , 2017 ; Li et al. , 2020 ) . While others have trained a single network to generate a set of Pareto optimal solutions ( Yang et al. , 2019 ; Parisi et al. , 2016 ) . There are various methodologies to tackle MOO problems . In ϵ-constrained methods optimize one of the objectives at a time while using the other objectives as constraints ( Mavrotas , 2009 ; Chinchuluun & Pardalos , 2007 ) . One of the most common ways is to use preference vectors ( or weights ) . Some methods use the weighting vectors to scalarize the objective function and Pareto front can be obtained by solving the optimization for multiple preference vectors ( Coello et al. , 2009 ; Boyd et al. , 2004 ) . On the other hand , some methods ( including ours ) use these weighting vectors in constraints ( Das & Dennis , 1998 ) . Li et al . ( 2020 ) solved MOTSP by training multiple single policy networks . They converted the MOO problem into a single objective using linear scalarization with the help of preference vectors . They train K different networks , each with different preference vector , to approximate the Pareto front . Their network , called DRL-MOA , generated competitive results in comparison to classical methods . However , the downside of their method is that it is redundant to train multiple networks and requires a lot of resources . Furthermore , solutions on concave regions of Pareto fronts can not be uncovered using the linear scalarization technique , as proved in ( Boyd et al. , 2004 ) . In our work , we train a single network that can predict solutions for any preference vector . This enables us to produce a much denser Pareto front . Instead of using linear scalarization , our network learns to solve a constrained optimization problem where the constraints are dependent on the preference vectors . In this work , we present PA-Net , a framework of deep network trained using policy gradient that can approximate Pareto front for MOO problems . Our choice of using RL is motivated by success of Deep CO methods and the fact that it ’ s hard to generate training data for complex problems like MOTSP . We use PA-Net to find a set of Pareto optimal tours for MOTSP . An augmented version of the network presented in ( Deudon et al. , 2018 ) is adopted for PA-Net to solve MOTSP . The novelty of our algorithm is that we pose the problem of finding a set of Pareto optimal solutions as a con- strained optimization problem , rather than using linear scalarization . We use preference vectors as constraints , which indicates the desired location of the solution in objective space . Finally , we train our network using the reward constrained policy optimization ( Tessler et al. , 2018 ) , a constrained RL method . Our network performs better than other methods in computation time and generates competitive results in terms of quality of the Pareto front . 2 BACKGROUND . Here we review the definition of Pareto optimality , with a brief primer on solving TSP using DRL . 2.1 PROBLEM SETUP . A MOO is defined as : minx F⃗ ( x ) = ( f1 ( x ) , f2 ( x ) , ... fm ( x ) ) ( 1 ) where F⃗ ( x ) is a vector of m-objective functions and x ∈ X is the decision variable in Rn . In such problems , often different objectives are conflicting in nature , i.e . no single solution can simultaneously optimize all the objectives . Instead , a set of Pareto optimal solutions provide the best solutions with different trade-offs between various objectives . Pareto optimality is defined as follows : • Dominance : A solution xa is said to dominate xb ( xa ≺ xb ) if and only if fi ( xa ) ≤ fi ( xb ) , ∀i ∈ { 1 , ... m } and fj ( xa ) < fj ( xb ) such that ∃j ∈ { 1 , ... m } . • Pareto Optimality : A solution x∗ is said to be Pareto optimal if there does not exist any solution x ′ such that x ′ ≺ x∗ . A set of all such points form a Pareto frontier denoted by Υ . The Euclidean TSP is defined over a graph of n cities , where each city has coordinates a ∈ R2 . A TSP tour π provides a sequence of visiting cities exactly once and then returning to the starting city . MOTSP is a MOO problem that aims to find a set of Pareto optimal TSP tours Π ( Π ⊂ Zn ) on a complete graph s , while optimizing for m objectives . Each city in s can have p a set of features . In case of bi-objective TSP , the input graph s is a sequence of n nodes in a four dimensional space s = { a1i , a2i } i=1 : n , where ami ∈ R2 for each m ∈ { 1 , 2 } ( Li et al. , 2020 ) . The goal is to find a tour π ∈ Π that visits each city once and can simultaneously optimize the objectives for m ∈ { 1 , 2 } : fm ( π|s ) = ∥∥amπ ( n ) − amπ ( 1 ) ∥∥2 + n−1∑ i=1 ∥∥amπ ( i ) − amπ ( i+1 ) ∥∥2 ( 2 ) 2.2 TSP USING DRL . An attention based network to solve TSP was proposed by Deudon et al . ( 2018 ) . We will refer to this network as TSP-Net . We use a modified version of TSP-Net for our network . TSP-Net uses an actor-critic network which is trained using REINFORCE ( Williams , 1992 ) . The input to the network is a graph s. Each city coordinates is embedded into a higher dimension . These embedded representations are then passed on to a Multi-Headed attention encoder that generates an encoded representation of the complete graph . Both actor and critic networks share the same encoder architecture . The decoder network of the actor uses a Pointer-Network ( Ptr-Net ) to generate the TSP tours . The tour is sequentially constructed , where at each step an appropriate city is selected based on the current state and the previous actions by the Ptr-Net . The actor network θ , is trained to minimize the total tour length given by equation 2 . The network is trained on a batch of TSP problem instances of size B . The training objective for the actor is given by : D ( θ ) = Es∼S [ Eπ∼pθ ( .|S ) [ Q ( π|S ) ] ] ( 3 ) Here , S is the distribution from which training graphs are drawn and pθ ( π|S ) is the probability of a tour generated by the Ptr-Network . The decoder of the critic with parameters ϕ , is a feed forward network that predicts the baseline for the objective function .

Summary:  This paper studied an NN-based method to approximate the Pareto front (PF) for multi-objective optimization (MOO) problems. As a practical example of the MOO problem, the authors focused on the MO traveling salesman problem (MOTSP) and developed a new approximation method.  A critical problem discussed in the paper is how to efficiently and effectively approximate the PF. In contrast to an existing work (Li et al. 2020) that separately training networks for different preferences, meaning that the training process is time-consuming, the proposed approach using PA-Net uses a single network (i.e., an augmented version of the TSP-Net).  The targeting MOO problem is converted to constrained optimization problems (key: divide problems using constraints and preference vectors). The Lagrangians of constraints are used to train a policy in an actor-critic manner using solutions and preference vectors.  From the experimental results, the advantage of the proposed PA-Net is shown in terms of (1) training time, (2) HV, and (3) computational time (i.e., inference time). Some graphical illustrations are helpful to compare the PFs estimated by different methods. Roughly speaking, the performance of the PA-Net outperformed existing methods.