A Graph Neural Network Assisted Monte Carlo Tree Search Approach to Traveling Salesman Problem

We present a graph neural network assisted Monte Carlo Tree Search approach for the classical traveling salesman problem ( TSP ) . We adopt a greedy algorithm framework to construct the optimal solution to TSP by adding the nodes successively . A graph neural network ( GNN ) is trained to capture the local and global graph structure and give the prior probability of selecting each vertex every step . The prior probability provides a heuristic for MCTS , and the MCTS output is an improved probability for selecting the successive vertex , as it is the feedback information by fusing the prior with the scouting procedure . Experimental results on TSP up to 100 nodes demonstrate that the proposed method obtains shorter tours than other learning-based methods . 1 INTRODUCTION . Traveling Salesman Problem ( TSP ) is a classical combinatorial optimization problem and has many practical applications in real life , such as planning , manufacturing , genetics ( Applegate et al. , 2006b ) . The goal of TSP is to find the shortest route that visits each city once and ends in the origin city , which is well-known as an NP-hard problem ( Papadimitriou , 1977 ) . In the literature , approximation algorithms were proposed to solve TSP ( Lawler et al. , 1986 ; Goodrich & Tamassia , 2015 ) . In particular , many heuristic search algorithms were made to find a satisfactory solution within a reasonable time . However , the performance of heuristic algorithms depends on handcrafted heuristics to guide the search procedure to find competitive tours efficiently , and the design of heuristics usually requires substantial expertise of the problem ( Johnson & McGeoch , 1997 ; Dorigo & Gambardella , 1997 ) . Recent advances in deep learning provide a powerful way of learning effective representations from data , leading to breakthroughs in many fields such as speech recognition ( Lecun et al. , 2015 ) . Efforts of the deep learning approach to tackling TSP has been made under the supervised learning and reinforcement learning frameworks . Vinyals et al . ( Vinyals et al. , 2015 ) introduced a pointer network based on the Recurrent Neural Network ( RNN ) to model the stochastic policy that assigns high probabilities to short tours given an input set of coordinates of vertices . Dai et al . ( Dai et al. , 2017 ) tackled the difficulty of designing heuristics by Deep Q-Network ( DQN ) based on structure2vec ( Dai et al. , 2016b ) , and a TSP solution was constructed incrementally by the learned greedy policy . Most recently , Kool et al . ( Kool et al. , 2019 ) used Transformer-Pointer Network ( Vaswani et al. , 2017 ) to learn heuristics efficiently and got close to the optimal TSP solution for up to 100 vertices . These efforts made it possible to solve TSP by an end-to-end heuristic algorithm without special expert skills and complicated feature design . In this paper , we present a new approach to solving TSP . Our approach combines the deep neural network with the Monte Carlo Tree Search ( MCTS ) , so that takes advantage of the powerful feature representation and scouting exploration . A graph neural network ( GNN ) is trained to capture the local and global graph structure and predict the prior probability , for each vertex , of whether this vertex belongs to the partial tour . Besides node features , we integrate edge information into each update-layer in order to extract features efficiently from the problem whose solution relies on the edge weight . Similar to above-learned heuristic approaches , we could greedily select the vertex according to the biggest prior probability and yet the algorithm may fall into the local optimum because the algorithm has only one shot to compute the optimal tour and never goes back and reverses the decision . To overcome this problem , we introduce a graph neural network assisted Monte Carlo Tree Search ( GNN-MCTS ) to make the decision more reliable by a large number of scouting simulations . The trained GNN is used to guide the MCTS procedure that effectively reduces the complexity of the search space and MCTS provides a more reliable policy to avoid stuck in a local optimum . Experimental results on TSP up to 100 vertices demonstrate that the proposed method obtains shorter tours than other learning-based methods . The remainder of the paper is organized as follows : After reviewing related work in Section 2 , we briefly give a preliminary introduction to TSP in Section 3 . Our approach is formulated in Section 4 . Experimental results are given in Section 5 , followed by the conclusion in Section 6 . 2 RELATED WORK . The TSP is a well studied combinatorial optimization problem , and many learning-based algorithms have been proposed . In 1985 , Hopfield et al . proposed a neural network to solve the TSP ( Hopfield & Tank , 1985 ) . This is the first time that researchers attempted to use neural networks to solve combinatorial optimization problems . Since the impressive results produced by this approach , many researchers have made efforts on improving the performance ( Bout & Miller , 1988 ; Brandt et al. , 1988 ) . Many shallow network architectures were also proposed to solve the combinatorial optimization problem ( Favata & Walker , 1991 ; Fort , 1988 ; Angniol et al. , 1988 ; Kohonen , 1982 ) . Recent years , deep neural networks have been adopted to solve the TSP and many works have achieved remarkable results . We summarize the existing learning-base methods from the following aspects . ENCODER AND DECODER Vinylas et al . ( Vinyals et al. , 2015 ) proposed a neural architecture called Pointer Net ( Ptr-Net ) to learn the conditional probability of a tour using a mechanism of the neural attention . Instead of using attention to blend hidden units of an encoder to a context vector , they used attention as pointers to the input vertices . The parameters of the model are learned by maximizing the conditional probabilities for the training examples in a supervised way . Upon test time , they used a beam search procedure to find the best possible tour . Two flaws exist in the method . First , Ptr-Net can only be applied to solve problems of a small scale ( n≤ 50 ) . Second , the beam search procedure might generate invalid routes . Bello et al . ( Bello et al. , 2017 ) proposed a framework to tackle TSP using neural networks and reinforcement learning . Similar to Vinylas et al. , they employed the approach of Ptr-Net as a policy model to learn a stochastic policy over tours . Furthermore , they masked the visited vertices to avoid deriving invalid routes and added a glimpse which aggregates different parts of the input sequence to improve the performance . Instead of training the model in a supervised way , they introduced an Actor-Critic algorithm to learn the parameters of the model and empirically demonstrated that the generalization is better compared to optimizing a supervised mapping of labeled data . The algorithm significantly outperformed the supervised learning approach ( Vinyals et al. , 2015 ) with up to 100 vertices . Kool et al . ( Kool et al. , 2019 ) introduced an efficient model and training method for TSP and other routing problems . Compared to ( Bello et al. , 2017 ) , they removed the influence on the input order of the vertices by replacing recurrence ( LSTMs ) with attention layers . The model can include valuable information about the vertices by multi-head attention mechanism which plays an important role in the setting where decisions relate directly to the vertices in a graph . Similar to ( Bello et al. , 2017 ) , they applied a reinforcement learning method to train the model . Instead of learning a value function as a baseline , they introduced a greedy rollout policy to generate baseline and empirically showed that the greedy rollout baseline can improve the quality and convergence speed for the approach . They improved the state-of-art performance among 20 , 50 , and 100 vertices . Independent of the work of Kool et al. , Deudon et al . ( Deudon et al. , 2018 ) also proposed a framework which uses attention layers and reinforcement learning algorithm ( Actor-Critic ) to learn a stochastic policy . They combined the machine learning methods with an existing heuristic algorithm , i.e. , 2-opt to enhance the performance of the framework . GRAPH EMBEDDING Dai et al . ( Dai et al. , 2017 ) proposed a framework , which combines reinforcement learning with graph embedding neural network , to construct solutions incrementally for TSP and other combinatorial optimization problems . Instead of using a separate encoder and decoder , they introduced a graph embedding network based on the structure2vec ( Dai et al. , 2016a ) to capture the current state of the solution and the structure of a graph . Furthermore , they used Q-learning parameterized by the graph embedding network to learn a greedy policy that outputs which vertex being inserted into the partial tour . They adopt the farthest strategy ( Rosenkrantz et al. , 2013 ) to get the best insertion position of the partial tour . Nowak et al . ( Nowak et al. , 2017 ) propose a supervised manner to directly output a tour as an adjacency matrix based on a Graph Neural Network and then convert the matrix into a feasible solution by beam search . The author only reports an optimality gap of 2.7 % for n = 20 and slightly worse than the auto-regressive data-driven model ( Vinyals et al. , 2015 ) . The performance of the above-mentioned methods was suffered due to the greedy policy which selects the vertex according to the biggest prior probability or the value . In this paper , we introduce a new Monte Carlo Tree Search-based algorithm to overcome this problem . 3 PRELIMINARIES . TRAVELING SALESMAN PROBLEM Let G ( V , E , w ) denotes a weighted graph , where V is the set of vertices , E the set of edges and w : E → R+ the edge weight function , i.e. , w ( u , v ) is the weight of edge ( u , v ) ∈ E. We use S = { v1 , v2 , ... , vi } to represent an ordered tour sequence that starts with v1 and ends with vi , and S̄ = V \ S the set of candidate vertices for addition , condition on S. The target of TSP is to find a tour sequence with the lowest cost , i.e. , c ( G , S ) = ∑|S|−1 i=1 w ( S ( i ) , S ( i + 1 ) ) + w ( S ( |S| ) , S ( 1 ) ) when |S| = |V | . 4 PROPOSED APPROACH . For a graph , our goal is to construct a tour solution by adding vertices successively . A natural approach is to train a deep neural network of some form to decide which vertex being added to the partial tour at each step . That is , a neural network f would take the graph G and the partial tour sequence S as input , and the output f ( G|S ) would be a prior probability that indices how likely each vertex to be selected . Intuitively , we can use the prior probability in a greedy way , i.e. , selecting vertex with the biggest probability , to generate the tour sequence incrementally . However , deriving tours in this way might fall into the local optimum because the algorithm has only one shot to compute the optimal tour and never goes back and reverses the decision . To overcome this problem , we enhance the policy-decisions by MCTS assisted with the deep neural network . We begin in Section 4.1 by introducing how to transform TSP into a Markov Decision Process ( MDP ) . Then in Section 4.2 , we describe the GNN architecture for parameterizing f ( G|S ) . Finally , Section 4.3 describes GNN-MCTS for combinatorial optimization problems , especially the TSP . The overall approach is illustrated in Figure 1 .

Summary: The paper proposes learning a TSP solver that incrementally constructs a tour by adding one city at a time to it using a graph neural network and MCTS. The problem is posed as a reinforcement learning problem, and the graph neural network parameters are trained to minimize the tour length on a training set of TSP instances. A graph neural network architecture called Static Edge Graph Neural Networks is introduced which takes into account the graph of all cities in a given problem instance as well as the partial tour constructed so far in an episode. The network predicts probabilities for the remaining cities to be selected as the next city in the tour, which is then used to compute a value function that guides MCTS. Results on synthetic TSP instances with 20, 50, and 100 cities show that the approach is able to achieve better objective values than prior learning-based approaches. Applying AlphaZero-like approaches to TSP is an interesting test case for understanding how well they can work on hard optimization problems.