Solving NP-Hard Problems on Graphs with Extended AlphaGo Zero

1 INTRODUCTION . There is no polynomial-time algorithm found for NP-hard problems [ 7 ] , but they often arise in many real-world optimization tasks . Therefore , a variety of algorithms have been developed in a long history , including approximation algorithms [ 2 , 14 ] , meta-heuristics based on local searches such as simulated annealing and evolutionary computation [ 15 , 10 ] , general-purpose exact solvers such as CPLEX 1 and Gurobi [ 16 ] , and problem-specific exact solvers [ 1 , 25 ] . Recently , machine learning approaches have been actively investigated to solve combinatorial optimization , with the expectation that the combinatorial structure of the problem can be automatically learned without complicated hand-crafted heuristics . In the early stage , many of these approaches focused on solving specific problems [ 17 , 5 ] such as the traveling salesperson problem ( TSP ) . Khalil et al . [ 19 ] proposed a general framework to solve combinatorial problems by a combination of reinforcement learning and graph embedding , which attracted attention for the following two reasons : It does not require any knowledge on graph algorithms other than greedy selection based on network outputs . Furthermore , it learns algorithms without any training dataset . Thanks to these advantages , the framework can be applied to a diverse range of problems over graphs and it also performs much better than previous learning-based approaches . However , we observed poor empirical performance on some graphs having different characteristics ( e.g. , synthetic graphs and real-world graphs ) than random graphs that were used for training , possibly because of the limited exploration space of their Q-learning method . In this paper , to overcome its weakness , we propose a novel solver , named CombOpt Zero . CombOpt Zero is inspired by AlphaGo Zero [ 33 ] , a superhuman engine of Go , which conducts Monte Carlo Tree Search ( MCTS ) to train deep neural networks . AlphaGo Zero was later generalized to AlphaZero [ 34 ] so that it can handle other games ; however , its range of applications is limited to two-player games whose state is win/lose ( or possibly draw ) . We extend AlphaGo Zero to a bunch of combinatorial 1www.cplex.com problems by a simple normalization technique based on random sampling . In the same way as AlphaGo Zero , CombOpt Zero automatically learns a policy network and value network by self-play based on MCTS . We train our networks for five kinds of NP-hard tasks and test on different instances including standard random graphs ( e.g. , the Erdős-Renyi model [ 11 ] and the Barabási-Albert model [ 3 ] ) , benchmark graphs , and real-world graphs . We show that , with only a greedy selection on the policy network , CombOpt Zero has a better generalization to a variety of graphs than the existing method , which indicates that the MCTS-based training strengthens the exploration of various actions . When more computation time is allowed , using the MCTS at test time with the full use of both the policy network and value network significantly improves the performance . Furthermore , we combine our framework with several graph neural network models [ 21 , 39 , 28 ] , and experimentally demonstrate that an appropriate choice of models contributes to improving the performance with a significant margin . 2 BACKGROUND . In this section , we introduce the background which our work is based on . 2.1 MACHINE LEARNING FOR COMBINATORIAL OPTIMIZATION . Machine learning approaches for combinatorial optimization problems have been studied in the literature , starting from Hopfield & Tank [ 17 ] , who applied a variant of neural networks to small instances of Traveling Salesperson Problem ( TSP ) . With the success of deep learning , more and more studies were conducted including Bello et al . [ 5 ] , Kool et al . [ 23 ] for TSP and Wang et al . [ 37 ] for MAXSAT . Khalil et al . [ 19 ] proposed an end-to-end reinforcement learning framework S2V-DQN , which attracted attention because of promising results in a wide range of problems over graphs such as MINIMUMVERTEXCOVER and MAXCUT . Another advantage of this method is that it does not require domain knowledge on specific algorithms or any training dataset . It optimizes a deep Q-network ( DQN ) where the Q-function is approximated by a graph embedding network , called structure2vec ( S2V ) [ 9 ] . The DQN is based on their reinforcement learning formulation , where each action is picking up a node and each state represents the “ sequence of actions ” . In each step , a partial solution S ⊂ V , i.e. , the current state , is expanded by the selected vertex v∗ = argmaxv∈V ( h ( S ) ) Q ( h ( S ) , v ) to ( S , v ∗ ) , where h ( · ) is a fixed function determined by the problem that maps a state to a certain graph , so that the selection of v will not violate the problem constraint . For example , in MAXIMUMINDEPENDENTSET , h ( S ) corresponds to the subgraph of the input graph G = ( V , E ) induced by V \ ( S ∪N ( S ) ) , where N ( S ) is the open neighbors of S. The immediate reward is the change in the objective function . The Q-network , i.e , S2V learns a fixed dimensional embedding for each node . In this work , we mitigate the issue of S2V-DQN ’ s generalization ability . We follow the idea of their reinforcement learning setting , with a different formulation , and replace their Q-learning by a novel learning strategy inspired by AlphaGo Zero . Note that although some studies combine classic heuristic algorithms and learning-based approaches ( using dataset ) to achieve the state-of-the-art performance [ 26 , 12 ] , we stick to learning without domain knowledge and dataset in the same way as S2V-DQN . 2.2 ALPHAGO ZERO . AlphaGo Zero [ 35 ] is a well-known superhuman engine designed for use with the game of Go . It trains a deep neural network fθ with parameter θ by reinforcement learning . Given a state ( game board ) , the network outputs fθ ( s ) = ( p , v ) , where p is the probability vector of each move and v ∈ [ −1 , 1 ] is a scalar denoting the state value . If v is close to 1 , the player who takes a corresponding action from state s is very likely to win . The fundamental idea of AlphaGo Zero is to enhance its own networks by self-play . For this self-play , a special version of Monte Carlo Tree Search ( MCTS ) [ 22 ] , which we describe later , is used . The network is trained in such a way that the policy imitates the enhanced policy by MCTS π , and the value imitates the actual reward from self play z ( i.e . z = 1 if the player wins and z = −1 otherwise ) . More formally , it learns to minimize the loss L = ( z − v ) 2 +CrossEntropy ( p , π ) + creg‖θ‖22 , ( 1 ) where creg is a nonnegative constant for L2 regularization . MCTS is a heuristic search on game trees . In AlphaGo Zero , the search tree is a rooted tree , where each node corresponds to a state and the root is the initial state . Each edge ( s , a ) denotes action a at state s and stores a tuple ( N ( s , a ) , W ( s , a ) , Q ( s , a ) , P ( s , a ) ) , where N ( s , a ) is the visit count , W ( s , a ) and Q ( s , a ) are the total and mean action value respectively , and P ( s , a ) is the prior probability . One iteration of MCTS consists of three parts : select , expand , and backup . First , from the root node , we keep choosing an action that maximizes an upper confidence bound Q ( s , a ) + cpuctP ( s , a ) √∑ a′ N ( s , a ′ ) 1 +N ( s , a ) , ( 2 ) where cpuct is a non-negative constant ( select ) . Once it reaches to unexplored node s , then the edge values are initialized using the network prediction ( p , v ) = fθ ( s ) ( expand ) . After expanding a new node , each visited edge is traversed and its edge values are updated ( backup ) so that Q maintain the mean of state evaluations over simulations : Q ( s , a ) = 1N ( s , a ) ∑ s′|s , a→s′ vs′ , where the sum is taken over those states reached from s after taking action a . After some iterations , the probability vector π is calculated by πa = N ( s0 , a ) 1/τ∑ bN ( s0 , b ) 1/τ for each a ∈ As0 , where τ is a temperature parameter . AlphaGo Zero defeated its previous engine AlphaGo [ 33 ] with 100-0 score without human knowledge , i.e. , the records of the games of professional players and some known techniques in the history of Go . We are motivated to take advantage of the AlphaGo Zero technique in our problem setting since we also aim at training deep neural networks for discrete optimization without domain knowledge . However , we can not directly apply AlphaGo Zero , which was designed for two-player games , to combinatorial optimization problems . Section 3.2 and 3.3 explains how we resolve this issue . 2.3 GRAPH NEURAL NETWORK . A Graph Neural Network ( GNN ) is a neural network that takes graphs as input . Kipf & Welling [ 21 ] proposed the Graph Convolutional Network ( GCN ) inspired by spectral graph convolutions . Because of its scalability , many variants of spatial based GNN were proposed . Many of them can be described as a Message Passing Neural Network ( MPNN ) [ 13 ] . They recursively aggregate neighboring feature vectors to obtain node embeddings that capture the structural information of the input graph . The Graph Isomorphism Network ( GIN ) [ 39 ] is one of the most expressive MPNNs in terms of graph isomorphism . Although they have a good empirical performance , some studies point out the limitation of the representation power of MPNNs [ 39 , 30 ] . Maron et al . [ 27 ] proposed an Invariant Graph Network ( IGN ) using tensor representations of a graph and was shown to be universal [ 29 , 18 ] . Since it requires a high-order tensor in middle layers , which is impractical , Maron et al . [ 28 ] proposed 2-IGN+ , a scalable and powerful model . All of these models , as well as S2V [ 9 ] used in S2V-DQN , are compared in the experiments to test the difference in the performance for combinatorial optimization . The detail of each model is described in Appendix B . 3 METHOD . In this section , we give a detailed explanation of our algorithm to solve combinatorial optimization problems over graphs . First , we introduce our reinforcement learning formulation for NP-hard problems . Then , we explain the basic ideas of our proposed CombOpt Zero in light of the difference between 2-player games and our formulation . Finally , we describe the whole algorithm .

Summary: There has been a sequence of recent works on learning heuristics for combinatorial optimization problems on graphs by treating them as Markov decision processes, and learning by reinforcement a good policy. Since the dynamics of these problems can be readily simulated, in this paper, the authors propose to use AlphaZero, a MCTS variant, with a GNN architecture to learn and predict good solutions. They compare against the approach of Khalil et al. (2017), as well as hand-made heuristics, on three benchmark problems (Min Vertex Cover, Max Cut, Max Clique). There are results on Max Independent Set and Min Feedback Vertex Set in the appendix as well.