Generalization in Deep RL for TSP Problems via Equivariance and Local Search

1 INTRODUCTION . The traveling salesman problem ( TSP ) has numerous applications from supply chain management ( Snyder & Shen , 2019 ) to electronic design automation ( Gerez , 1999 ) or bioinformatics ( Jones & Pevzner , 2004 ) . As an NP-hard problem , solving large-sized problem instances is generally intractable . Deep reinforcement learning ( RL ) -based heuristic solvers have been demonstrated ( Bello et al. , 2016 ; Dai et al. , 2017 ; Kool et al. , 2019 ; Ma et al. , 2019 ) as being able to provide competitive solutions while being fast , which is crucial in many domains such as logistics for real-time operations . However , this ability has only been demonstrated on small-sized problems where RLbased solvers are usually trained and tested on small instances . When evaluated instead on larger instances , such solvers perform poorly . Since the computational cost of training on large instances is prohibitive , increasing the size of the training instances is not a practical option for obtaining efficient solvers for larger instances . To overcome this limitation , this paper investigates techniques to increase the generalization capability of deep RL solvers , which would allow to train on small instances and then solve larger ones . Generalization in deep RL ( François-Lavet et al. , 2018 ) can be improved by adjusting the following three components : input/representation space , objective function , and learning algorithm . We propose to achieve this by exploiting equivariance , local search , and stochastic curriculum learning , which we explain next . Although our approach is demonstrated on TSP problems , we believe it can be naturally extended to other combinatorial optimization problems , but also to more classic RL domains . We leave this for future work . Problems with spatial information , such as TSP problems , enjoy many spatial symmetries which can be exploited for RL training and generalization . An RL solver is invariant with respect to some symmetry , if its outputs remain unchanged for symmetric inputs . For instance , translating all the city positions leave optimal solutions intact . More generally , an RL solver is equivariant , if its outputs are also transformed with a corresponding symmetry for symmetric inputs . For instance , if cities are permuted , a corresponding permutation is required on the outputs of the RL solver . Invariance and equivariance of the solver with respect to some symmetries allows a smaller input space to be considered during training and more abstract representation insensitive to those symmetries to be learned while the trained solver still covers the whole input space . Previous work has considered using local search as a simple additional step to improve the feasible solution output by a RL-based solver . In contrast to most previous work , we further employ local search as a tool to smooth the objective function optimized during RL training . To that aim , we interleave RL training with local search using the improved solution provided by the latter to train the RL solver via a modified policy gradient . Moreover , we propose a novel simple baseline called the policy rollout baseline to reduce the variance of its estimation . Curriculum learning ( Soviany et al. , 2021 ) can accelerate training , but also improve generalization ( Weinshall et al. , 2018 ) . By stochastically adjusting the difficulty of the training instances , as described by instance sizes , the RL-based solver can learn more easily and faster . Since the solver sees various sizes of instances , this may prevent it to overfit to one particular instance size . The architecture of our model includes a graph neural network ( GNN ) , a multi-layer perceptron ( MLP ) , and an attention mechanism . Due to all the used techniques , we name our model as eMAGIC ( e for equivariance , M for MLP , A for Attention , G for Graph neural network , I for Interleaved local search , and C for Stochastic Curriculum learning ) . We demonstrate that it can be trained on small random instances ( up to 50 cities ) and perform competitively on larger random or realistic instances ( up to 1000 cities ) . The only learning methods that can perform better than ours require learning on the instance to be solved ( see Section 2 for a discussion ) . 2 RELATED WORK . Research on exploiting deep learning ( Vinyals et al. , 2015 ; Li et al. , 2018 ; Joshi et al. , 2019a ; Prates et al. , 2019 ; Xing & Tu , 2020 ) or RL ( Bello et al. , 2016 ; Dai et al. , 2017 ; Kool et al. , 2019 ; Deudon et al. , 2018 ; Ma et al. , 2019 ; Da Costa et al. , 2020 ; Zheng et al. , 2021 ; Kwon et al. , 2021 ; Wu et al. , 2021b ) to design heuristics for solving TSP problems has become very active . The current achievements demonstrate the potential of machine-learning-based approaches , but also reveal their usual limitation on solving larger-size instances , which has prompted much research on generalization ( Joshi et al. , 2019b ; Fu et al. , 2021 ; Wu et al. , 2021b ) . Some experimental work ( Joshi et al. , 2019b ) suggests that deep RL may provide more generalizable solvers than supervised learning , which is a further motivation for our own work . Another advantage of deep RL is that it does not require optimal solvers to train . We discuss next the related work that exploits equivariance or local search like our method . To the best of our knowledge , no other work uses stochastic curriculum learning for designing an RL-based solver for TSP , although Lisicki et al . ( 2020 ) evaluated deterministic curriculum learning strategies on small TSP instances . For space reasons , our discussion below emphasizes the approaches using deep RL as a constructive heuristic ( which builds a solution iteratively ) . Invariance and equivariance have been important properties to exploit for designing powerful deep learning architectures ( Bengio & Lecun , 1997 ; Gens & Domingos , 2014 ; Cohen & Welling , 2016 ) . They have also inspired some of the previous work on solving TSP problems , although they were often not explicitly discussed like in our work . For instance , Deudon et al . ( 2018 ) use principal component analysis to exploit rotation invariance as a single preprocessing step . In contrast to their work , we compose various preprocessing transformations to be applied at every solving step . To the best of our knowledge , we are the first to propose such technique . In addition , permutation invariance is the motivation for using attention models ( Kool et al. , 2019 ; Deudon et al. , 2018 ) or GNNs ( Ma et al. , 2019 ) in RL solvers . Contrary to Deudon et al . ( 2018 ) , Kool et al . ( 2019 ) select the next city to visit based on the first and last visited cities while Ma et al . ( 2019 ) propose to use relative positions for translation invariance . Our proposition combines those two ideas , but compared to Kool et al . ( 2019 ) , our model is based on GNN , while compared to Ma et al . ( 2019 ) , it has a simpler architecture that does not require an LSTM . For vehicle routing problems , Peng et al . ( 2020 ) propose to remove visited cities from the RL agent ’ s input , a technique we also apply to TSP . In contrast to their work , our GNN model is simpler and we investigate generalization from small instances to very large instances . To summarize , compared to all previous work , our paper investigates in a more systematic fashion the exploitation of equivariance to help training and improve generalization . Several previous studies combine deep learning or RL with various local search techniques to find better solutions : 2-opt ( Deudon et al. , 2018 ; Ma et al. , 2019 ; Da Costa et al. , 2020 ; Wu et al. , 2021b ) , k-opt ( Zheng et al. , 2021 ) , or Monte-Carlo tree search ( Fu et al. , 2021 ) . In contrast to these methods , we use a combined local search technique , which applies several heuristics in an efficient way . Moreover , to the best of our knowledge , no previous work considers interleaving local search with policy gradient updates to obtain a more synergetic final method for solving TSP . For the bin packing problem , Cai et al . ( 2019 ) use RL to learn a perturbative heuristics to provide a good initial solution to a heuristic optimizer . Our interleaved training process is based on a similar idea , but our RL agent learns a constructive heuristics and we provide an intuitive motivation why this approach may work via our smoothed policy gradient . All these machine-learning-based methods can be categorized as learning from a batch of instances or as directly learning on the instance to be solved . Like our work , most methods are part of the first category . However , some recent work belongs to the second category ( Zheng et al. , 2021 ) or are hybrid ( Fu et al. , 2021 ) . For instance , Zheng et al . ( 2021 ) apply RL to make the LKH algorithm ( Helsgaun , 2017 ) , a classic efficient heuristic for TSP , adaptive to the instance to be solved . In Fu et al . ( 2021 ) ’ s method , evaluations learned in a supervised way from a batch of instances are used in Monte-Carlo tree search so that solutions can adaptively be found for the current instance . Being adaptive to the current instance can undoubtedly boost the performance of the solver . We leave for future work the improvement of our solver to become more adaptive to the instance to be solved . 3 PRELIMINARIES . Following previous work , we focus on the symmetric 2D Euclidean TSP , which we recall below . We then explain how a TSP instance can be tackled with RL . We first provide some notations : For any positive integerN ∈ N , [ N ] denotes the set { 1 , 2 , . . . , N } . Vectors and matrices are denoted in bold ( e.g. , x or X ) . For a set of subscripts I , XI denotes the matrix formed by the rows of X whose indices are in I . Traveling Salesman Problem A symmetric 2D Euclidean TSP instance is described by a set of N cities ( identified to the set [ N ] ) and their coordinates in R2 . The goal in this problem is to find the shortest tour that visits each city exactly once . The coordinates of city i are denoted xi ∈ R2 . The matrix whose rows correspond to city coordinates is denoted X ∈ RN×2 . A feasible solution ( i.e. , tour ) of a TSP instance is a permutation σ over [ N ] with length equal to : Lσ ( X ) = N∑ t=1 ‖xσ ( t ) − xσ ( t+1 ) ||2 ( 1 ) where ‖·‖2 denotes the ` 2-norm , σ ( t ) ∈ [ N ] is the t-th visited city in tour σ , and by abuse of notation , σ ( N + 1 ) denotes σ ( 1 ) . Therefore , solving a TSP instance consists in finding the permutation σ that minimizes the tour length Lσ ( X ) defined in Equation ( 1 ) . Since TSP solutions are invariant to scaling , we assume that the city coordinates are in the square [ 0 , 1 ] 2 . In Appendix A , we recall some heuristics ( e.g. , insertion , k-opt ) that have been proposed for solving TSP . RL as a Constructive Heuristic RL can be used to construct a TSP tour σ sequentially . Intuitively , at iteration t ∈ [ N ] , an RL solver ( i.e. , policy ) selects the next unvisited city σ ( t ) to visit based on the current partial tour and the description of the TSP instance ( i.e. , coordinates of cities ) . Therefore , this RL problem corresponds to a repeated N -horizon sequential decision-making problem . As noticed by Kool et al . ( 2019 ) , the decision for the next city to visit only depends on the description of the TSP instance and the first and last visited cities . In addition , we update the description of the TSP instance by removing the coordinates of visited cities . Surprisingly , to the best of our knowledge , no previous work exploits this simplified RL model . Formally , in the RL language , at time step t ∈ [ N ] , an action at represents the next city to visit , i.e. , at = σ ( t ) . Let I1 = [ N ] and It+1 = [ N ] \ { σ ( 1 ) , . . . , σ ( t ) } be the set of remaining cities after t cities have been already visited . Moreover , let J1 = [ N ] and Jt = It ∪ { σ ( 1 ) , σ ( t ) } be the set of unvisited cities in addition of the first and last visited cities ( σ ( 1 ) and σ ( t ) ) . Therefore , at ∈ It for all t ∈ [ N ] . A state st can be represented by a matrix XJt with flags indicating the first and last visited cities . This matrix includes the coordinates of unvisited cities ( XIt ) in addition to the coordinates of the first and last visited cities ( xσ ( 1 ) and xσ ( t ) ) . Note that at t = 1 , no city has been chosen yet , so the initial state s1 only contains the list of city coordinates . A state sN contains the coordinates of the unvisited cities and those of the first and last visited cities . After choosing the last city to visit in state sN , the tour σ is completely generated . The immediate reward r ( st , at ) for an action at in a state st can be defined as the negative length between the last visited city and the next chosen city , since we want the tour length to be small : r ( st , at ) = 0 for t = 1 −||xσ ( t ) − xσ ( t−1 ) ||2 for t = 2 , . . . , N − 1 −||xσ ( N ) − xσ ( N−1 ) ||2 − ||xσ ( 1 ) − xσ ( N ) ||2 for t = N ( 2 ) After choosing the first city a1 = σ ( 1 ) , the reward is zero since no length can be computed . After the final action aN , an additional reward −||xσ ( 1 ) − xσ ( N ) ||2 is added to complete the tour length . In deep RL , a policy πθ is represented as a neural network parametrized by θ . The goal is then to find θ∗ that maximizes the objective function J ( θ ) : θ ? = argmax θ J ( θ ) = argmax θ Eτ∼pθ ( τ ) [ N∑ t=1 rt ] , ( 3 ) where for all t ∈ [ N ] , rt = r ( st , at ) , τ = ( s1 , a1 , s2 , a2 , . . . , sN , aN ) is a complete trajectory , and pθ is the probability over trajectories induced by policy πθ . This objective function J ( θ ) can be optimized by a policy gradient ( Williams , 1992 ) or actor-critic method ( Sutton & Barto , 1998 ) .

Summary: This paper proposes a novel combination of policy gradient and local search algorithms for the traveling salesman problems (TSP). The main idea is to apply local search to tours generated by policy rollout and compute policy gradients using the (potentially) improved tour from local search. In addition to the algorithmic contribution, this paper also presents a collection of preprocessing steps to ensure problem instance features are equivalent. Empirical studies are provided on random TSP instances as well as those in TSPLIB. Finally, ablation studies show the importance for each component of the proposed algorithm.