Deep Auto-Deferring Policy for Combinatorial Optimization

Designing efficient algorithms for combinatorial optimization appears ubiquitously in various scientific fields . Recently , deep reinforcement learning ( DRL ) frameworks have gained considerable attention as a new approach : they can automatically train a good solver while relying less on sophisticated domain knowledge of the target problem . However , the number of stages ( until reaching the final solution ) required by existing DRL solvers is proportional to the size of the input graph , which hurts their scalability to large-scale instances . In this paper , we seek to resolve this issue by proposing a novel design of DRL ’ s policy , coined auto-deferring policy ( AutoDP ) , automatically stretching or shrinking its decision process . Specifically , it decides whether to finalize the value of each vertex at the current stage or defer to determine it at later stages . We apply the proposed AutoDP framework to the maximum independent set ( MIS ) problem and its variants under various scenarios . Our experimental results demonstrate significant improvement of AutoDP over the current state-of-the-art DRL scheme in terms of computational efficiency and approximation quality . The reported performance of our generic DRL scheme is also comparable with that of the existing solvers for MIS , e.g. , AutoDP outperforms them for the Barabási-Albert graph with two million vertices . 1 INTRODUCTION . Combinatorial optimization is an important mathematical field addressing fundamental questions of computation , where its popular examples include the maximum independent set ( MIS , Miller & Muller 1960 ) , satisfiability ( SAT , Schaefer 1978 ) and traveling salesman problem ( TSP , Voigt 1831 ) . Such problems also arise in various applied fields , e.g. , sociology ( Harary & Ross , 1957 ) , operations research ( Feo et al. , 1994 ) and bioinformatics ( Gardiner et al. , 2000 ) . However , most combinatorial optimization problems are NP-hard to solve , i.e. , exact solutions are typically intractable to find in practical situations . To alleviate this issue , there have been huge efforts in designing fast heuristic solvers ( Biere et al. , 2009 ; Knuth , 1997 ; Mezard et al. , 2009 ) that generate approximate solutions for such scenarios . Recently , the remarkable progress in deep learning has stimulated increased interest in learning such heuristics based on deep neural networks ( DNNs ) . Such learning-based approaches are attractive since one could automate the design of approximation algorithms with less reliance on sophisticated knowledge . As the most straight-forward way , supervised learning schemes can be used for training DNNs to imitate the solutions obtained from existing solvers ( Vinyals et al. , 2015 ; Li et al. , 2018 ; Selsam et al. , 2019 ) . However , such a direction can be criticized , for its quality and applicability are bounded by those of existing solvers . An ideal direction is to discover new solutions in a fully unsupervised manner , potentially outperforming those based on domain-specific knowledge . To this end , deep reinforcement learning ( DRL ) schemes have been studied in the literature ( Bello et al. , 2016 ; Khalil et al. , 2017 ; Deudon et al. , 2018 ; Kool et al. , 2019 ) as a Markov decision process ( MDP ) can be naturally designed with rewards derived from the optimization objective of the target problem . Then , the corresponding agent can be trained based on existing training schemes of DRL , e.g. , Bello et al . ( 2016 ) trained the so-called pointer network for the TSP based on actor-critic training . Such DRL-based methods are especially attractive since they can even solve unexplored problems where domain knowledge is scarce and no efficient heuristic is known . However , the existing methods struggle to compete with the existing highly optimized solvers . In particular , the gap grows larger when the problem requires solutions with higher dimensions or more complex structures . Our motivation stems from the observation that existing DRL-based solvers lack efficient policies for generating solutions to combinatorial problems . Specifically , they are mostly based on emulating greedy iterative heuristics ( Bello et al. , 2016 ; Khalil et al. , 2017 ) and become too slow for training on large graphs . Their choice seems inevitable since an algorithm that generates a solution based on a single feed-forward pass of DNN is potentially hard to train due to large variance in reward signals coming from high dimensional solutions . Contribution . In this paper , we propose a new scalable DRL framework , coined auto-deferring policy ( AutoDP ) , designed towards solving combinatorial problems on large graphs . We particularly focus on applying AutoDP to the MIS problem ( and its variants ) which attempts to find a maximum set of vertices in the graph where no pair of vertices are adjacent to each other . Our choice of the MIS problem is motivated by its hardness and applicability . First , the MIS problem is impossible to approximate in polynomial time by a constant factor ( unless P=NP ) ( Hastad , 1996 ) , in contrast to ( Euclidean or metric ) TSP which can be approximated by a factor of 1.5 ( Christofides , 1976 ) . Next , it has wide applications including classification theory ( Feo et al. , 1994 ) , computer vision ( Sander et al. , 2008 ) and communication algorithms ( Jiang & Walrand , 2010 ) . The main novelty of AutoDP is automatically stretching the determination of the solution throughout multiple steps . In particular , the agent iteratively acts on every undetermined vertex for either ( a ) determining the membership of the vertex in the solution or ( b ) deferring the determination to be made in later steps ( see Figure 1 for illustration ) . Inspired by the celebrated survey propagation ( Braunstein et al. , 2005 ) for solving the SAT problem , AutoDP could be interpreted as prioritizing the “ easier ” decisions to be made first , which in turn simplifies the harder ones by eliminating the source of uncertainties . Compared to the greedy strategy ( Khalil et al. , 2017 ) which determines the membership of a single vertex at each step , our framework brings significant speedup by allowing determinations on as many vertices as possible to happen at once . Based on such speedup , AutoDP can solve the optimization problem by generating a large number of candidate solutions in a limited time budget , then reporting the best solution among them . In such a scenario , it is beneficial for the algorithm to generate diverse candidates . To this end , we additionally give a novel diversification bonus to our agent during training , which explicitly encourages the agent to generate a large variety of solutions . Specifically , we create a “ coupling ” of MDPs to generate two solutions for the given MIS problem and reward the agents for a large deviation between the solutions . The resulting reward efficiently stimulates the agent to explore high-dimensional input spaces and to improve the performance at the evaluation . We empirically validate the AutoDP method on various types of graphs including the Erdös-Rényi ( Erdős & Rényi , 1960 ) model , the Barabási-Albert ( Albert & Barabási , 2002 ) model , the SATLIB ( Hoos & Stützle , 2000 ) benchmark and real-world graphs ( Hamilton et al. , 2017 ; Yanardag & Vishwanathan , 2015 ; Leskovec & Sosič , 2016 ) . Our algorithm shows consistent superiority over the existing state-of-the-art DRL method ( Khalil et al. , 2017 ) both in terms of speed and quality of the solution , and can compete with the existing MIS solver ( ReduMIS , Lamm et al . 2017 ) under a similar time budget . For example , AutoDP even outperforms ReduMIS in the Barabási-Albert graph with two million vertices using a smaller amount of time . Furthermore , we also show that our fully learning-based scheme generalizes well even to graph types unseen during training and works well even for other variants of the MIS problem : the maximum weighted independent set ( MWIS ) problem and the prize collecting maximum independent set ( PCMIS ) problem ( see Appendix B ) . This sheds light on its potential of being a generic solver that works for arbitrary large-scale graphs . 2 RELATED WORKS . The maximum independent set ( MIS ) problem is a prototypical NP-hard task where its optimal solution can not be approximated by a constant factor in polynomial time ( unless P = NP ) ( Hastad , 1996 ) , although it admits a nearly linear factor approximation algorithm ( Boppana & Halldórsson , 1992 ) . It is also known to be a W [ 1 ] -hard problem in terms of fixed-parameter tractability ( Downey & Fellows , 2012 ) . Since the problem is NP-hard , existing methods ( Tomita et al. , 2010 ; San Segundo et al. , 2011 ) for exactly solving the MIS problem often suffers from a prohibitive amount of computation in large graphs . To resolve this issue , a wide range of solvers have been developed for approximately solving the MIS problem ( Grosso et al. , 2008 ; Andrade et al. , 2012 ; Dahlum et al. , 2016 ; Lamm et al. , 2017 ; Chang et al. , 2017 ; Hespe et al. , 2019 ) . Notably , Lamm et al . ( 2017 ) developed a combination of an evolutionary algorithm with graph kernelization techniques for the MIS problem . Later , Chang et al . ( 2017 ) and Hespe et al . ( 2019 ) further improved the graph kernelization technique by introducing new reduction rules and parallelization based on graph partitioning , respectively . In the context of solving combinatorial optimization using neural networks , Hopfield & Tank ( 1985 ) first applied the Hopfield-network for solving the traveling salesman problem ( TSP ) . Since then , several works also tried to utilize neural networks in different forms , e.g. , see Smith ( 1999 ) for a review of such papers . Such works were mostly used for solving combinatorial optimization through online learning , i.e. , training was performed for each problem instance separately . More recently , ( Vinyals et al. , 2015 ) and ( Bello et al. , 2016 ) proposed to solve TSP using an attention-based neural network trained in an offline way . They showed promising results that stimulated many other works to use neural networks for solving combinatorial problems ( Khalil et al. , 2017 ; Selsam et al. , 2019 ; Deudon et al. , 2018 ; Amizadeh et al. , 2018 ; Li et al. , 2018 ; Kool et al. , 2019 ) . Importantly , Khalil et al . ( 2017 ) proposed a reinforcement learning framework for solving the minimum vertex cover problem , which is equivalent to solving the MIS problem . They query the agent for each vertex to add as a new member of the vertex cover at each step of the Markov decision process . However , such a procedure often leads to a prohibitive amount of computation on graphs with large vertex covers . Next , Li et al . ( 2018 ) aim for developing a supervised learning framework for solving the MIS problem . At an angle , their framework is similar to ours since they use hand-designed rules to defer the solution generation procedure at each step . However , it is hard to fairly compare with ours since the supervised learning scheme is highly sensitive to the quality of solutions obtained from existing solvers and is often too expensive to apply , e.g. , for the MIS-variants .

Summary: This paper aims at solving graph-based combinatorial optimization problems using a new paradigm for Deep Reinforcement Learning. In contrast with standard Markov Decision Processes used for combinatorial DRL, the authors advocate the use of “deferred” MDPs capturing more complex actions (which can choose a subset of nodes), and more sophisticated transitions, which combine an update phase together with a cleanup phase. For the DRL architecture, an Actor-Critic framework is trained using a proximal policy optimization approach. The paper specifically focuses on the Maximum Independent Set (MIS) problem. Comparative results on various instances reveal that this approach clearly outperforms the S2V-DQN algorithm and is competitive with some usual combinatorial solvers (CPLEX, ReduMIS).