Pareto Frontier Approximation Network (PA-Net) Applied to Multi-objective TSP

1 INTRODUCTION . Travelling Salesman Problem ( TSP ) is a popular sequencing problem . TSP aims at finding a sequence or a tour of visiting each node in a given graph and finally returning to the starting node such that the overall cost is minimized . TSP and its variants are widely used in robotics for applications like path planning for UAVs ( Xu & Che , 2019 ) , multi-robot path planning ( Yu et al. , 2002 ) , task allocation for robotic manipulators ( Zacharia & Aspragathos , 2005 ) , coverage planning ( Bormann et al. , 2018 ) etc . We intend to use multi-objective travelling salesman problem ( MOTSP ) for application of coverage planning . Algorithms for coverage planning generates trajectories for robots that can cover a given area ( Bormann et al. , 2018 ) . Area coverage is used in robotic applications like cleaning robots , surveillance etc . Grid based TSP planners ( Bormann et al. , 2018 ) , segment a given map into multiple cells , and generate a coverage pattern for each cell.The optimal visiting order for these cells is generated by solving TSP that minimizes the length of the tour . Now imagine a scenario where a robot has to visit these cells and the order is dependent on multiple objectives such as tour length , priority order of the cells etc . So MOTSP would be a more appropriate choice in such a scenario . There is a wide variety of algorithms ranging from exact methods to evolution based methods to solve MOTSP ( Lust & Teghem , 2010 ) . Evolutionary algorithms like non-dominated sorting genetic algorithm-II ( NSGA-II ) ( Beirigo & dos Santos , 2016 ) and multi-objective evolutionary algorithm ( MOEA/D ) ( Peng et al. , 2009 ) are a popular choice of methods to tackle MOTSP and other multi objective optimization problems . Many algorithms also use evolutionary algorithms coupled with local search heuristics ( Jaszkiewicz , 2002 ; Ke et al. , 2014 ; Cai et al. , 2014 ) . In practice , these evolutionary based methods suffer in performance and computation time with an increase in the scale of the problem ( Zhang et al. , 2016 ) . We intend to address these issues using Deep Reinforcement Learning ( DRL ) . Contribution : In this work , we present Pareto frontier approximation network ( PA-Net ) that generates an approximation of a set of Pareto optimal tours for MOTSP . We use PA-Net to generate MOTSP tours for coverage planning . Our main contributions are : ( 1 ) The drawback of existing ap- proaches ( Li et al. , 2020 ) is costly training of separate networks for different preferences in objective space . Our method avoids this costly training through the use of a single network , while providing generalizability over different preferences in objective space . ( 2 ) PA-Net can generate solutions for large number of preferences , which contributes to a dense approximation of Pareto front . Our network has competitive results on various metrics evaluating the quality of Pareto front and faster inference times . ( 3 ) Our design can be easily extended to generate a set of Pareto optimal solutions for other multi-objective reinforcement learning and Multi-Objective Optimization ( MOO ) tasks . Related Work : Sequencing problems are a subset of Combinatorial Optimization ( CO ) where the decision variables are discrete . Most CO problems are NP-Hard , as a result state-of-the-art algorithms rely on handcrafted heuristics for making decisions that are otherwise too expensive to compute or mathematically not well-defined . Recently , researchers are addressing these issues using deep learning and machine learning ( ML ) ( Bengio et al. , 2020 ; Vesselinova et al. , 2020 ; Mazyavkina et al. , 2020 ) . Many recent works have been dedicated to solving TSP and other sequencing problems using deep networks . Google Brain ’ s Pointer Network ( Ptr-Net ) ( Vinyals et al. , 2015 ) learns the conditional probability of an output sequence of elements that are discrete tokens corresponding to positions in an input sequence . They used Ptr-Net to solve Euclidean TSP ( and other CO problems ) in an end to end fashion , where the solutions from classical methods are used as baselines for training . Similarly , TSP was solved using 2D graph conv-nets followed by beam search ( Joshi et al. , 2019 ) . RL was used to solve various combinatorial optimization problems in ( Bello et al. , 2016 ) . Their network uses an RNN based encoder and a Ptr-Net . They trained their network using policy gradient . Similar methodology was adopted in ( Deudon et al. , 2018 ) . They used transformers as the encoder . Kool et al . ( 2018 ) proposed an attention based network which used a baseline based on greedy rollouts . Their network outperformed other deep-learning based solvers . While these methods generated competitive results , however , they still lag in performance in comparison to TSP solvers like OR-Tools and Concorde . Finding Pareto optimal solutions has been studied in deep learning literature for various multiobjective tasks . In supervised learning tasks , many works focussed on multi-objective classification tasks ( Sener & Koltun , 2018 ; Lin et al. , 2019 ; Mahapatra & Rajan , 2020 ; Ruchte & Grabocka , 2021 ; Navon et al. , 2021 ) . Similarly , many multi-objective RL methods have been to solve multi-objective MDPs ( Roijers et al. , 2013 ; Parisi et al. , 2014 ) . Some works in RL train many single policy networks to approximate Pareto front ( Vamplew et al. , 2017 ; Li et al. , 2020 ) . While others have trained a single network to generate a set of Pareto optimal solutions ( Yang et al. , 2019 ; Parisi et al. , 2016 ) . There are various methodologies to tackle MOO problems . In ϵ-constrained methods optimize one of the objectives at a time while using the other objectives as constraints ( Mavrotas , 2009 ; Chinchuluun & Pardalos , 2007 ) . One of the most common ways is to use preference vectors ( or weights ) . Some methods use the weighting vectors to scalarize the objective function and Pareto front can be obtained by solving the optimization for multiple preference vectors ( Coello et al. , 2009 ; Boyd et al. , 2004 ) . On the other hand , some methods ( including ours ) use these weighting vectors in constraints ( Das & Dennis , 1998 ) . Li et al . ( 2020 ) solved MOTSP by training multiple single policy networks . They converted the MOO problem into a single objective using linear scalarization with the help of preference vectors . They train K different networks , each with different preference vector , to approximate the Pareto front . Their network , called DRL-MOA , generated competitive results in comparison to classical methods . However , the downside of their method is that it is redundant to train multiple networks and requires a lot of resources . Furthermore , solutions on concave regions of Pareto fronts can not be uncovered using the linear scalarization technique , as proved in ( Boyd et al. , 2004 ) . In our work , we train a single network that can predict solutions for any preference vector . This enables us to produce a much denser Pareto front . Instead of using linear scalarization , our network learns to solve a constrained optimization problem where the constraints are dependent on the preference vectors . In this work , we present PA-Net , a framework of deep network trained using policy gradient that can approximate Pareto front for MOO problems . Our choice of using RL is motivated by success of Deep CO methods and the fact that it ’ s hard to generate training data for complex problems like MOTSP . We use PA-Net to find a set of Pareto optimal tours for MOTSP . An augmented version of the network presented in ( Deudon et al. , 2018 ) is adopted for PA-Net to solve MOTSP . The novelty of our algorithm is that we pose the problem of finding a set of Pareto optimal solutions as a con- strained optimization problem , rather than using linear scalarization . We use preference vectors as constraints , which indicates the desired location of the solution in objective space . Finally , we train our network using the reward constrained policy optimization ( Tessler et al. , 2018 ) , a constrained RL method . Our network performs better than other methods in computation time and generates competitive results in terms of quality of the Pareto front . 2 BACKGROUND . Here we review the definition of Pareto optimality , with a brief primer on solving TSP using DRL . 2.1 PROBLEM SETUP . A MOO is defined as : minx F⃗ ( x ) = ( f1 ( x ) , f2 ( x ) , ... fm ( x ) ) ( 1 ) where F⃗ ( x ) is a vector of m-objective functions and x ∈ X is the decision variable in Rn . In such problems , often different objectives are conflicting in nature , i.e . no single solution can simultaneously optimize all the objectives . Instead , a set of Pareto optimal solutions provide the best solutions with different trade-offs between various objectives . Pareto optimality is defined as follows : • Dominance : A solution xa is said to dominate xb ( xa ≺ xb ) if and only if fi ( xa ) ≤ fi ( xb ) , ∀i ∈ { 1 , ... m } and fj ( xa ) < fj ( xb ) such that ∃j ∈ { 1 , ... m } . • Pareto Optimality : A solution x∗ is said to be Pareto optimal if there does not exist any solution x ′ such that x ′ ≺ x∗ . A set of all such points form a Pareto frontier denoted by Υ . The Euclidean TSP is defined over a graph of n cities , where each city has coordinates a ∈ R2 . A TSP tour π provides a sequence of visiting cities exactly once and then returning to the starting city . MOTSP is a MOO problem that aims to find a set of Pareto optimal TSP tours Π ( Π ⊂ Zn ) on a complete graph s , while optimizing for m objectives . Each city in s can have p a set of features . In case of bi-objective TSP , the input graph s is a sequence of n nodes in a four dimensional space s = { a1i , a2i } i=1 : n , where ami ∈ R2 for each m ∈ { 1 , 2 } ( Li et al. , 2020 ) . The goal is to find a tour π ∈ Π that visits each city once and can simultaneously optimize the objectives for m ∈ { 1 , 2 } : fm ( π|s ) = ∥∥amπ ( n ) − amπ ( 1 ) ∥∥2 + n−1∑ i=1 ∥∥amπ ( i ) − amπ ( i+1 ) ∥∥2 ( 2 ) 2.2 TSP USING DRL . An attention based network to solve TSP was proposed by Deudon et al . ( 2018 ) . We will refer to this network as TSP-Net . We use a modified version of TSP-Net for our network . TSP-Net uses an actor-critic network which is trained using REINFORCE ( Williams , 1992 ) . The input to the network is a graph s. Each city coordinates is embedded into a higher dimension . These embedded representations are then passed on to a Multi-Headed attention encoder that generates an encoded representation of the complete graph . Both actor and critic networks share the same encoder architecture . The decoder network of the actor uses a Pointer-Network ( Ptr-Net ) to generate the TSP tours . The tour is sequentially constructed , where at each step an appropriate city is selected based on the current state and the previous actions by the Ptr-Net . The actor network θ , is trained to minimize the total tour length given by equation 2 . The network is trained on a batch of TSP problem instances of size B . The training objective for the actor is given by : D ( θ ) = Es∼S [ Eπ∼pθ ( .|S ) [ Q ( π|S ) ] ] ( 3 ) Here , S is the distribution from which training graphs are drawn and pθ ( π|S ) is the probability of a tour generated by the Ptr-Network . The decoder of the critic with parameters ϕ , is a feed forward network that predicts the baseline for the objective function .

Summary: The paper introduces a method to approximate a Pareto front for multi-objective TSP from neural combinatorial optimization using deep reinforcement learning. The PA-Net method uses a constrained optimization problem formalization with a preference vector to control the solution generation. An experimental evaluation is performed and PA-Net is compared to multi-objective genetic algorithms and one DRL-based method from the literature. PA-Net performs competitive and approximates the Pareto front well.