Generalization in Deep RL for TSP Problems via Equivariance and Local Search

1 INTRODUCTION . The traveling salesman problem ( TSP ) has numerous applications from supply chain management ( Snyder & Shen , 2019 ) to electronic design automation ( Gerez , 1999 ) or bioinformatics ( Jones & Pevzner , 2004 ) . As an NP-hard problem , solving large-sized problem instances is generally intractable . Deep reinforcement learning ( RL ) -based heuristic solvers have been demonstrated ( Bello et al. , 2016 ; Dai et al. , 2017 ; Kool et al. , 2019 ; Ma et al. , 2019 ) as being able to provide competitive solutions while being fast , which is crucial in many domains such as logistics for real-time operations . However , this ability has only been demonstrated on small-sized problems where RLbased solvers are usually trained and tested on small instances . When evaluated instead on larger instances , such solvers perform poorly . Since the computational cost of training on large instances is prohibitive , increasing the size of the training instances is not a practical option for obtaining efficient solvers for larger instances . To overcome this limitation , this paper investigates techniques to increase the generalization capability of deep RL solvers , which would allow to train on small instances and then solve larger ones . Generalization in deep RL ( François-Lavet et al. , 2018 ) can be improved by adjusting the following three components : input/representation space , objective function , and learning algorithm . We propose to achieve this by exploiting equivariance , local search , and stochastic curriculum learning , which we explain next . Although our approach is demonstrated on TSP problems , we believe it can be naturally extended to other combinatorial optimization problems , but also to more classic RL domains . We leave this for future work . Problems with spatial information , such as TSP problems , enjoy many spatial symmetries which can be exploited for RL training and generalization . An RL solver is invariant with respect to some symmetry , if its outputs remain unchanged for symmetric inputs . For instance , translating all the city positions leave optimal solutions intact . More generally , an RL solver is equivariant , if its outputs are also transformed with a corresponding symmetry for symmetric inputs . For instance , if cities are permuted , a corresponding permutation is required on the outputs of the RL solver . Invariance and equivariance of the solver with respect to some symmetries allows a smaller input space to be considered during training and more abstract representation insensitive to those symmetries to be learned while the trained solver still covers the whole input space . Previous work has considered using local search as a simple additional step to improve the feasible solution output by a RL-based solver . In contrast to most previous work , we further employ local search as a tool to smooth the objective function optimized during RL training . To that aim , we interleave RL training with local search using the improved solution provided by the latter to train the RL solver via a modified policy gradient . Moreover , we propose a novel simple baseline called the policy rollout baseline to reduce the variance of its estimation . Curriculum learning ( Soviany et al. , 2021 ) can accelerate training , but also improve generalization ( Weinshall et al. , 2018 ) . By stochastically adjusting the difficulty of the training instances , as described by instance sizes , the RL-based solver can learn more easily and faster . Since the solver sees various sizes of instances , this may prevent it to overfit to one particular instance size . The architecture of our model includes a graph neural network ( GNN ) , a multi-layer perceptron ( MLP ) , and an attention mechanism . Due to all the used techniques , we name our model as eMAGIC ( e for equivariance , M for MLP , A for Attention , G for Graph neural network , I for Interleaved local search , and C for Stochastic Curriculum learning ) . We demonstrate that it can be trained on small random instances ( up to 50 cities ) and perform competitively on larger random or realistic instances ( up to 1000 cities ) . The only learning methods that can perform better than ours require learning on the instance to be solved ( see Section 2 for a discussion ) . 2 RELATED WORK . Research on exploiting deep learning ( Vinyals et al. , 2015 ; Li et al. , 2018 ; Joshi et al. , 2019a ; Prates et al. , 2019 ; Xing & Tu , 2020 ) or RL ( Bello et al. , 2016 ; Dai et al. , 2017 ; Kool et al. , 2019 ; Deudon et al. , 2018 ; Ma et al. , 2019 ; Da Costa et al. , 2020 ; Zheng et al. , 2021 ; Kwon et al. , 2021 ; Wu et al. , 2021b ) to design heuristics for solving TSP problems has become very active . The current achievements demonstrate the potential of machine-learning-based approaches , but also reveal their usual limitation on solving larger-size instances , which has prompted much research on generalization ( Joshi et al. , 2019b ; Fu et al. , 2021 ; Wu et al. , 2021b ) . Some experimental work ( Joshi et al. , 2019b ) suggests that deep RL may provide more generalizable solvers than supervised learning , which is a further motivation for our own work . Another advantage of deep RL is that it does not require optimal solvers to train . We discuss next the related work that exploits equivariance or local search like our method . To the best of our knowledge , no other work uses stochastic curriculum learning for designing an RL-based solver for TSP , although Lisicki et al . ( 2020 ) evaluated deterministic curriculum learning strategies on small TSP instances . For space reasons , our discussion below emphasizes the approaches using deep RL as a constructive heuristic ( which builds a solution iteratively ) . Invariance and equivariance have been important properties to exploit for designing powerful deep learning architectures ( Bengio & Lecun , 1997 ; Gens & Domingos , 2014 ; Cohen & Welling , 2016 ) . They have also inspired some of the previous work on solving TSP problems , although they were often not explicitly discussed like in our work . For instance , Deudon et al . ( 2018 ) use principal component analysis to exploit rotation invariance as a single preprocessing step . In contrast to their work , we compose various preprocessing transformations to be applied at every solving step . To the best of our knowledge , we are the first to propose such technique . In addition , permutation invariance is the motivation for using attention models ( Kool et al. , 2019 ; Deudon et al. , 2018 ) or GNNs ( Ma et al. , 2019 ) in RL solvers . Contrary to Deudon et al . ( 2018 ) , Kool et al . ( 2019 ) select the next city to visit based on the first and last visited cities while Ma et al . ( 2019 ) propose to use relative positions for translation invariance . Our proposition combines those two ideas , but compared to Kool et al . ( 2019 ) , our model is based on GNN , while compared to Ma et al . ( 2019 ) , it has a simpler architecture that does not require an LSTM . For vehicle routing problems , Peng et al . ( 2020 ) propose to remove visited cities from the RL agent ’ s input , a technique we also apply to TSP . In contrast to their work , our GNN model is simpler and we investigate generalization from small instances to very large instances . To summarize , compared to all previous work , our paper investigates in a more systematic fashion the exploitation of equivariance to help training and improve generalization . Several previous studies combine deep learning or RL with various local search techniques to find better solutions : 2-opt ( Deudon et al. , 2018 ; Ma et al. , 2019 ; Da Costa et al. , 2020 ; Wu et al. , 2021b ) , k-opt ( Zheng et al. , 2021 ) , or Monte-Carlo tree search ( Fu et al. , 2021 ) . In contrast to these methods , we use a combined local search technique , which applies several heuristics in an efficient way . Moreover , to the best of our knowledge , no previous work considers interleaving local search with policy gradient updates to obtain a more synergetic final method for solving TSP . For the bin packing problem , Cai et al . ( 2019 ) use RL to learn a perturbative heuristics to provide a good initial solution to a heuristic optimizer . Our interleaved training process is based on a similar idea , but our RL agent learns a constructive heuristics and we provide an intuitive motivation why this approach may work via our smoothed policy gradient . All these machine-learning-based methods can be categorized as learning from a batch of instances or as directly learning on the instance to be solved . Like our work , most methods are part of the first category . However , some recent work belongs to the second category ( Zheng et al. , 2021 ) or are hybrid ( Fu et al. , 2021 ) . For instance , Zheng et al . ( 2021 ) apply RL to make the LKH algorithm ( Helsgaun , 2017 ) , a classic efficient heuristic for TSP , adaptive to the instance to be solved . In Fu et al . ( 2021 ) ’ s method , evaluations learned in a supervised way from a batch of instances are used in Monte-Carlo tree search so that solutions can adaptively be found for the current instance . Being adaptive to the current instance can undoubtedly boost the performance of the solver . We leave for future work the improvement of our solver to become more adaptive to the instance to be solved . 3 PRELIMINARIES . Following previous work , we focus on the symmetric 2D Euclidean TSP , which we recall below . We then explain how a TSP instance can be tackled with RL . We first provide some notations : For any positive integerN ∈ N , [ N ] denotes the set { 1 , 2 , . . . , N } . Vectors and matrices are denoted in bold ( e.g. , x or X ) . For a set of subscripts I , XI denotes the matrix formed by the rows of X whose indices are in I . Traveling Salesman Problem A symmetric 2D Euclidean TSP instance is described by a set of N cities ( identified to the set [ N ] ) and their coordinates in R2 . The goal in this problem is to find the shortest tour that visits each city exactly once . The coordinates of city i are denoted xi ∈ R2 . The matrix whose rows correspond to city coordinates is denoted X ∈ RN×2 . A feasible solution ( i.e. , tour ) of a TSP instance is a permutation σ over [ N ] with length equal to : Lσ ( X ) = N∑ t=1 ‖xσ ( t ) − xσ ( t+1 ) ||2 ( 1 ) where ‖·‖2 denotes the ` 2-norm , σ ( t ) ∈ [ N ] is the t-th visited city in tour σ , and by abuse of notation , σ ( N + 1 ) denotes σ ( 1 ) . Therefore , solving a TSP instance consists in finding the permutation σ that minimizes the tour length Lσ ( X ) defined in Equation ( 1 ) . Since TSP solutions are invariant to scaling , we assume that the city coordinates are in the square [ 0 , 1 ] 2 . In Appendix A , we recall some heuristics ( e.g. , insertion , k-opt ) that have been proposed for solving TSP . RL as a Constructive Heuristic RL can be used to construct a TSP tour σ sequentially . Intuitively , at iteration t ∈ [ N ] , an RL solver ( i.e. , policy ) selects the next unvisited city σ ( t ) to visit based on the current partial tour and the description of the TSP instance ( i.e. , coordinates of cities ) . Therefore , this RL problem corresponds to a repeated N -horizon sequential decision-making problem . As noticed by Kool et al . ( 2019 ) , the decision for the next city to visit only depends on the description of the TSP instance and the first and last visited cities . In addition , we update the description of the TSP instance by removing the coordinates of visited cities . Surprisingly , to the best of our knowledge , no previous work exploits this simplified RL model . Formally , in the RL language , at time step t ∈ [ N ] , an action at represents the next city to visit , i.e. , at = σ ( t ) . Let I1 = [ N ] and It+1 = [ N ] \ { σ ( 1 ) , . . . , σ ( t ) } be the set of remaining cities after t cities have been already visited . Moreover , let J1 = [ N ] and Jt = It ∪ { σ ( 1 ) , σ ( t ) } be the set of unvisited cities in addition of the first and last visited cities ( σ ( 1 ) and σ ( t ) ) . Therefore , at ∈ It for all t ∈ [ N ] . A state st can be represented by a matrix XJt with flags indicating the first and last visited cities . This matrix includes the coordinates of unvisited cities ( XIt ) in addition to the coordinates of the first and last visited cities ( xσ ( 1 ) and xσ ( t ) ) . Note that at t = 1 , no city has been chosen yet , so the initial state s1 only contains the list of city coordinates . A state sN contains the coordinates of the unvisited cities and those of the first and last visited cities . After choosing the last city to visit in state sN , the tour σ is completely generated . The immediate reward r ( st , at ) for an action at in a state st can be defined as the negative length between the last visited city and the next chosen city , since we want the tour length to be small : r ( st , at ) = 0 for t = 1 −||xσ ( t ) − xσ ( t−1 ) ||2 for t = 2 , . . . , N − 1 −||xσ ( N ) − xσ ( N−1 ) ||2 − ||xσ ( 1 ) − xσ ( N ) ||2 for t = N ( 2 ) After choosing the first city a1 = σ ( 1 ) , the reward is zero since no length can be computed . After the final action aN , an additional reward −||xσ ( 1 ) − xσ ( N ) ||2 is added to complete the tour length . In deep RL , a policy πθ is represented as a neural network parametrized by θ . The goal is then to find θ∗ that maximizes the objective function J ( θ ) : θ ? = argmax θ J ( θ ) = argmax θ Eτ∼pθ ( τ ) [ N∑ t=1 rt ] , ( 3 ) where for all t ∈ [ N ] , rt = r ( st , at ) , τ = ( s1 , a1 , s2 , a2 , . . . , sN , aN ) is a complete trajectory , and pθ is the probability over trajectories induced by policy πθ . This objective function J ( θ ) can be optimized by a policy gradient ( Williams , 1992 ) or actor-critic method ( Sutton & Barto , 1998 ) .

Summary: The paper proposes a series of techniques and a new model to learn to solve the TSP using RL. First, a number of preprocessing steps are presented in order to transform a TSP instance into a standardized form. Then, a new encoder-decoder architecture is proposed, based on GNN, MLP and attention modules. The authors propose a modification of the policy gradient algorithms for training by replacing the value of the policy-generated solution by its value after applying a local search heuristic, and use the original value as a baseline. Finally they use a curriculum learning approach to increase the difficulty of the instances seen during training, the difficulty here being represented by increasing the size of the instances, within a given interval. Experimental results are presented on standard synthetic and realistic TSP instances of size up to 1000 nodes.