Targeted sampling of enlarged neighborhood via Monte Carlo tree search for TSP

1 INTRODUCTION . The travelling salesman problem ( TSP ) is a well-known combinatorial optimization problem with various real-life applications , such as transportation , logistics , biology , circuit design . Given n cities as well as the distance dij between each pair of cities i and j , the TSP aims to find a cheapest tour which starts from a beginning city ( arbitrarily chosen ) , visits each city exactly once , and finally returns to the beginning city . This problem is NP-hard , thus being extremely difficult from the viewpoint of theoretical computer science . Due to its importance in both theory and practice , many algorithms have been developed for the TSP , mostly based on traditional operations research ( OR ) methods . Among the existing TSP algorithms , the best exact solver Concorde ( Applegate et al. , 2009 ) succeeded in demonstrating optimality of an Euclidean TSP instance with 85,900 cities , while the leading heuristics ( Helsgaun , 2017 ) and ( Taillard & Helsgaun , 2019 ) are capable of obtaining near-optimal solutions for instances with millions of cities . However , these algorithms are very complicated , which generally consist of many hand-crafted rules and heavily rely on expert knowledge , thus being difficult to generalize to other combinatorial optimization problems . To overcome those limitations , recent years have seen a number of ML based algorithms being proposed for the TSP ( briefly reviewed in the next section ) , which attempt to automate the search process by learning mechanisms . This type of methods do not rely on expert knowledge , can be easily generalized to various combinatorial optimization problems , thus become promising research direction at the intersection of ML and OR . For the TSP , existing ML based algorithms can be roughly classified into two paradigms , i.e . : ( 1 ) End-to-end ML paradigm which uses a ML model alone to directly convert the input instance to a solution . ( 2 ) ML followed by OR paradigm which applies ML at first to provide some additional information , to guide the following OR procedure towards promising regions . Despite its high potential , for the TSP , existing ML based methods are still in its infancy , struggling to solve instances with more than 100 cities , leaving much room for further improvement compared with traditional methods . To this end , we propose a novel framework by combining Monte Carlo tree search ( MCTS ) with a basic OR method ( 2-opt based local search ) using variable neighborhood strategy to solve the TSP . The main contributions are summarized as follows . • Framework : We propose a new paradigm which combines OR and ML using variable neighborhood strategy . Starting from an initial state , a basic 2-opt based local search is firstly used to search within a small neighborhood . When no improvement is possible within the small neighborhood , the search turns into an enlarged neighborhood , where a reinforcement learning ( RL ) based method is used to identify a sample of promising actions , and iteratively choose one action to apply . Under this new paradigm , OR and ML respectively work within disjoint space , being flexible and targeted , and clearly different from the two paradigms mentioned above . More importantly , as a general framework without complicated hand-crafted rules , this framework could not only be applied to the TSP , but also be easily extended to many other combinatorial optimization problems . • Methodology : When we search within an enlarged neighborhood , it is infeasible to enumerate all the actions . We then seek to sample a number of promising actions . To do this automatically , we implement a MCTS method which iterates through simulation , selection and back-propagation steps , to collect useful information that guides the sampling process . To the best of our knowledge , there is only one existing paper ( Shimomura & Takashima , 2016 ) which also uses MCTS to solve the TSP . However , their method is a constructive approach , where each state is a partial TSP tour , and each action adds a city to increase the partial tour , until forming a complete tour . By contrast , our MCTS method is a conversion based approach , where each state is a complete TSP tour , and each action converts the original state to a new state ( also a complete TSP tour ) . The methodology and implementation details of our MCTS are very different from the MCTS method developed in ( Shimomura & Takashima , 2016 ) . • Results : We carry out experiments on two sets of public TSP instances . Experimental results ( detailed in Section 4 ) show that , on both data sets our MCTS algorithm obtains ( within reasonable time ) statistically much better results with respect to all the existing learning based algorithms . These results clearly indicate the potential of our new method for solving the TSP . The rest of this paper is organized as follows : Section 2 briefly reviews the existing learning based methods on the TSP . Section 3 describes in detail the new paradigm and the MCTS method . Section 4 provides and analyzes the experimental results . Finally Section 5 concludes this paper . 2 RELATED WORKS . In this section , we briefly review the existing ML based algorithms on the TSP , and then extend to several other highly related problems . Non-learned methods are omitted , and interested readers please find in ( Applegate et al. , 2009 ) , ( Rego et al. , 2011 ) , ( Helsgaun , 2017 ) and ( Taillard & Helsgaun , 2019 ) for an overlook of the leading TSP algorithms . The idea of applying ML to solve the TSP is not new , dated back to several decades ago . Hopfield & Tank ( 1985 ) proposed a Hopfield-network , which achieved the best TSP solutions at that time . Encouraged by this progress , neural networks were subsequently applied on many related problems ( surveyed by Smith ( 1999 ) ) . However , these early attempts only achieved limited success , with respect to other state-of-the-art algorithms , possibly due to the lack of high-performance hardware and training data . In recent years , benefited from the rapidly improving hardware and exponentially increasing data , ML ( especially deep learning ) achieved great successes in the field of artificial intelligence . Motivated by these successes , ML becomes again a hot and promising topic for combinatorial optimization , especially for the TSP . A number of ML based algorithms have been developed for the TSP , which can be roughly classified into two paradigms ( possibly with overlaps ) . End-to-end ML paradigm : Vinyals et al . ( 2015 ) introduced a pointer network which consists of an encoder and a decoder , both using recurrent neural network ( RNN ) . The encoder parses each TSP city into an embedding , and then the decoder uses an attention model to predict the probability distribution over the candidate ( unvisited ) cities . This process is repeated to choose a city one by one , until forming a complete TSP tour . The biggest advantage of the pointer network is its ability of processing graphs with different sizes . However , as a supervised learning ( SL ) method , it requires a large number of pre-computed optimal ( at least high-quality ) TSP solutions , being unaffordable for large-scale instances . To overcome this drawback , several successors chose reinforcement learning ( RL ) instead of SL , thus avoiding the requirement of pre-computed solutions . For example , Bello et al . ( 2017 ) implemented an actor-critic RL architecture , which uses the solution quality ( tour length ) as a reward signal , to guide the search towards promising area . Khalil et al . ( 2017 ) proposed a framework which maintains a partial tour and repeatedly calls a RL based model to select the most relevant city to add to the partial tour , until forming a complete tour . Emami & Ranka ( 2018 ) also implemented an actor-critic neural network , and chose Sinkhorn policy gradient to learn policies by approximating a double stochastic matrix . Concurrently , Deudon et al . ( 2018 ) and Kool et al . ( 2019 ) both proposed a graph attention network ( GAN ) , which incorporates attention mechanism with RL to auto-regressively improve the quality of the obtained solution . Specifically , Shimomura & Takashima ( 2016 ) proposed a MCTS algorithm for the TSP , which also belongs to this paradigm . As explained in the introduction section , this existing method is clearly different from our MCTS algorithm ( detailed in Section 3.5 ) . ML followed by OR paradigm : Applying ML alone is difficult to achieve satisfactory performance , thus it is recommended to combine ML and OR to form hybrid algorithms ( Bengio et al. , 2018 ) . Following this idea , Nowak et al . ( 2017 ) proposed a supervised approach , which trains a graph neural network ( GNN ) to predict an adjacency matrix ( heat map ) over the cities , and then attempts to convert the adjacency matrix to a feasible TSP tour by beam search ( OR based method ) . Joshi et al . ( 2019 ) followed this framework , but chose deep graph convolutional networks ( GCN ) instead of GNN to build heat map , and then tried to construct tours via highly parallelized beam search . Additionally , several algorithms belonging to above paradigm were further enhanced with OR based algorithms , thus also belonging to this paradigm . For example , the solution obtained by ML is post-optimized by sampling in ( Bello et al. , 2017 ) and ( Kool et al. , 2019 ) , or by 2-opt based local search in ( Deudon et al. , 2018 ) . Overall , this hybrid paradigm performs statistically much better than the end-to-end ML paradigm , showing the advantage of combining ML with OR . In addition to the works focused on the classic TSP , there are several ML based methods recently proposed for solving other related problems , such as the decision TSP ( Prates et al. , 2019 ) , the multiple TSP ( Kaempfer & Wolf , 2019 ) , and the vehicle routing problem ( Nazari et al. , 2018 ) , etc . Finally , for an overall survey about machine learning for combinatorial optimization , please refer to ( Bengio et al. , 2018 ) and ( Guo et al. , 2019 ) . 3 METHOD . 3.1 FRAMEWORK . The proposed new paradigm for combining OR and ML to solve the TSP is outlined in Fig . 1 . Specifically , the search process is considered as a Markov Decision Process ( MDP ) , which starts from an initial state π , and iteratively applies an action a to reach a new state π∗ . At first , the MDP explores within a small neighborhood , and tries to improve the current state by applying 2-opt based local search . When no further improvement is possible within the small neighborhood , the MDP turns into an enlarged neighborhood , which consists of a large number of possible actions , being infeasible to enumerate one by one . To improve search efficiency , MCTS is launched to iteratively sample a number of promising actions and choose an improving action to apply . When MCTS fails to find an improving action , the MDP jumps into an unexplored search space , and launches a new round of 2-opt local search and MCTS again . This process is repeated until the termination condition is met , then the best found state is returned as the final solution .

Summary: The paper proposes an algorithm for the Travelling Salesman Problem that starts with a random tour and iteratively improves it using 2-opt local search, followed by Monte Carlo Tree Search with k-opt moves to search a larger neighborhood of solutions. Results show that the algorithm matches the optimal objective value on a dataset of 2D Euclidean TSP instances with 20, 50, and 100 cities, and achieves better objective value than S2V DQN on a subset of instances from TSPLIB with 50 to 318 cities. It is interesting that the algorithm is fairly simple (apart from the definition of the local moves) and still able to achieve competitive objective values on small instances.