A Bi-Level Framework for Learning to Solve Combinatorial Optimization on Graphs

1 Introduction . Combinatorial Optimization ( CO ) is a family of long-standing optimization problems . A large portion of CO problems is NP-hard due to the combinatorial nature , raising challenges for traditional ( exact ) solvers on even medium-sized problems . Heuristic algorithms are often adopted to approximately solve CO problems within an acceptable time , and there is a growing trend adopting modern datadriven approaches to solve CO problems that achieve better and faster results [ 30 ] . The major line of works solving CO with machine learning ( ML ) is single-level [ 8 , 25 , 29 , 30 , 39 , 41 , 57 , 59 , 64 ] , where the prediction of ML module lies in the solution space , assuming the model has enough capacity learning the input-output mapping of the CO problem . However , achieving such an assumption is non-trivial , leading to the following two aspects of challenges . On the one hand , it is challenging to design a model with enough capacity with limited computational resources , and existing models are usually tailored for specific problems which require heavy trailand-error [ 25 , 57 , 59 ] . On the other hand , training such a heavy model requires either supervision from high-quality labels [ 31 , 57 , 60 ] which are infeasible to obtain for large-sized problems due to the NP-hard nature , or reinforcement learning ( RL ) [ 8 , 30 , 38 , 39 ] which might be unstable due to the challenges of large action space and sparse reward especially for large-sized problems [ 52 ] . ∗Part of the work was done while the first author was working as an intern at Ant Group . †Junchi Yan is the corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . An alternative approach resorts to a hybrid machine learning and traditional optimization pipeline [ 11 , 18 , 26 , 31 , 51 , 60 , 61 ] hoping to utilize the power of traditional optimization methods . However , designing a general hybrid MLCO approach is still non-trivial , as existing methods [ 11 , 60 ] usually require domain-specific knowledge for the model design . It is again challenging to obtain highquality supervision labels , and existing methods are based on either problem-specific surrogate labels [ 18 , 31 , 60 ] , or learned with RL while the challenges of RL still exist [ 11 , 51 ] . In this paper , we propose a general hybrid MLCO approach over graphs . We first reduce the complexity of deep learning model by reformulating the original CO into a bi-level optimization , whose objective is to minimize the long-term upper-level objective by optimizing the graph structure , and the lower-level problem is handled by an existing heuristic . We resort to RL and the traditional heuristic can be absorbed as part of the environment , and it is shown that the sparse reward issue is mitigated for the resulting RL problem compared to previous RL-based methods [ 8 , 30 , 38 , 39 ] . Specifically , our model is built with standard building blocks : the input graph is encoded by Graph Convolutional Network ( GCN ) [ 32 ] , and the actor and critic modules are based on ResNet blocks [ 22 ] and attention models [ 55 ] . All modules are learned with the Proximal Policy Optimization ( PPO ) algorithm [ 48 ] . The contributions of this paper include : • To combine the best of the two worlds , we propose a general hybrid approach that integrates traditional heuristic solvers with machine learning algorithm . • We propose a bi-level optimization formulation for learning to solve CO on graphs . The upper-level optimization adopts a reinforcement learning agent to adaptively modify the graphs , while the lowerlevel optimization involves traditional learning-free heuristics to solve combinatorial optimization tasks on the modified graphs . Our approach does not require ground truth labels . • The experiments for graphs up to thousands of nodes on several popular tasks such as Directed Acrylic Graph scheduling ( DAG scheduling ) , Graph Edit Distance ( GED ) , and Hamiltonian Cycle Problem ( HCP ) show that our method notably surpasses both traditional learning-free heuristics and single-level learning method . Our method generalizes well to graphs of different sizes while having comparable overhead to the single-level learning methods . 2 Related Work . Here we discuss related works listed in recent surveys [ 4 , 62 , 63 ] . We present their methodologies and compare with our method . Learning the end-to-end solution as a sequence . The major line of existing MLCO works mainly focus on tackling the problem end-to-end by predicting a sequence of solutions [ 25 , 29 , 30 , 35 , 39 , 41 , 57 , 59 , 64 ] . The pioneering work [ 57 ] is designated for traveling salesman problem ( TSP ) , whereby the sequence-to-sequence Pointer Network ( PtrNet ) model is learned with supervised learning . In [ 30 ] , a graph-to-sequence framework is proposed with graph embedding [ 10 ] and Q-learning [ 40 ] for general CO over graphs , and this general framework inspires the major line of MLCO works with applications to DAG scheduling [ 39 ] , graph matching [ 36 ] , and job shop scheduling [ 64 ] . [ 35 ] extends [ 30 ] by problem reduction and supervised learning-based tree search . The framework in [ 30 ] is treated as the RL baseline in this paper . Our approach differs from these single-level end-to-end RL methods [ 30 , 36 , 38 , 39 ] , which lack the flexibility to borrow the power of traditional methods and often suffer from the fundamental issues in RL : sparse reward and large action space . Learning to rewrite end-to-end solutions . Another line of end-to-end learning works predicts rewriting strategies of an existing solution [ 8 , 38 ] , where the model predictions also lie in the solution space . The agents learn to improve an existing solution in a decision sequence , and problems like job scheduling , expression simplification and routing problems have been studied . These learning-based local search heuristics also fall within the single-level paradigm , while our method incorporates another optimization over the graph structure . The sparse reward issue also exists for [ 8 , 38 ] , as a long episode is usually required for searching for a satisfying result . One-shot unsupervised end-to-end learning . There is also a growing trend applying one-shot unsupervised methods for the input-output mapping of CO , with maximum clique and graph cut solved by [ 29 ] , and quadratic assignment problem solved by [ 59 ] . However , the generalization ability of these methods is still an open question , and complicated constraints are often non-trivial to be encoded in the network ’ s prediction . Moreover , such a one-shot end-to-end network often calls for much higher model capacity compared with our multi-round alternating optimization paradigm . Hybrid of machine learning and traditional solvers . Different from learning the solution endto-end , researchers also propose hybrid machine learning and traditional solver approaches . ML modules are studied as sub-routines for traditional solvers , especially predicting branching strategies for branch-and-bound with either supervised learning [ 18 , 31 ] or reinforcement learning [ 51 ] . In [ 60 ] , the heuristic routine in A * algorithm is replaced by graph neural network to solve graph edit distance . However , these methods are tailored for special problems , and our aim is to develop a more general framework , where the learning part and heuristic module are two peers alternatively performed . Bi-level optimization . Our method is based on bi-level optimization , which is a family of optimization problems where a lower-level optimization is nested inside an upper-level optimization . Bi-level optimization is in general NP-hard [ 27 , 56 ] , and the applications of bi-level optimization can be found ranging from multi-player games [ 27 ] to vision tasks [ 37 ] , and there is a loosely relevant attempt [ 2 ] adopting supervised learning to solve a bi-level optimization for transportation . 3 Our Approach . In this paper , we propose a Bi-level Hybrid ( BiHyb ) machine learning and traditional heuristic approach . Sec . 3.1 shows both single-level and bi-level formulations of CO , and Sec . 3.2 shows the RL approach to the bi-level CO . 3.1 Bi-level Reformulation of Combinatorial Optimization . Without loss of generality , we consider the classic single-level CO with a single graph G as : min x f ( x|G ) s.t . hi ( x , G ) ≤ 0 , for i = 1 ... I ( 1 ) where x denotes the decision variable ( i.e . solution ) , f ( x|G ) denotes the objective function given input graph G and hi ( x , G ) ≤ 0 represents the set of constraints . For example , in DAG scheduling , the constraints enforce that the solution x , i.e . the execution order of the DAG job nodes , lies in the feasible space and does not conflict the topological dependency structure of G. The popular framework of existing MLCO methods regards Eq . 1 as a straight-forward end-to-end learning task , and various training methods have been developed , including : 1 ) supervised learning [ 41 , 57 ] obtaining training labels by solving small-scaled Eq . 1 with traditional solvers , however it is nearly infeasible to solve larger NP-hard problems ; 2 ) unsupervised learning [ 29 , 59 ] adopting the continuous relaxation of Eq . 1 as the learning objective , but existing methods face challenges when dealing with complicated constraints ; and 3 ) reinforcement learning [ 8 , 30 ] by predicting x sequentially , but the reward signal is unavailable until x reaches a complete solution , leading to the sparse reward issue . To ease the challenges introduced by the single-level formulation , we resort to the classic idea of modifying the original problem to aid problem solving , e.g . adding cutting planes for integer programming [ 19 , 54 ] . Our proposed framework is capable of handling CO on graphs if all constraints can be encoded by the graph structure , and our motivation is described by the following hypothesis : Algorithm 1 : Policy Roll-out for Bi-level Learning of Hybrid MLCO Solver ( BiHyb ) Input : Original graph G ; Max number of actions K. 1 G0 ← G ; 2 for k← 0 ... ( K − 1 ) do 3 Predict P ( a1 ) , P ( a2|a1 ) on Gk and sample a1 , a2 ; # upper-level optimization for learning 4 Gk+1← add , delete or modify the edge ( a1 , a2 ) in Gk ; # state-transition 5 xk+1← solve argminxk+1 f ( xk+1|Gk+1 ) by heuristic algorithm ; # lower-level optimization 6 rk ← f ( xk|G ) − f ( xk+1|G ) ; # reward Output : A list of rewards { r0 ... rK−1 } . The optimal solution x∗ to G can be acquired by modifying G. And we show the feasibility of this hypothesis for a family of problems by introducing the following proposition : Proposition . We define G as the set of all graphs that can be modified from G , and X as the set of all feasible solutions of G. If the heuristic algorithm is a surjection from G to X , for G and its optimal solution x∗ , there must exist G∗ ∈ G , such that x∗ is the output of the heuristic by solving G∗ . Proof . By the definition of surjection , since x∗ ∈ X , there must exist at least one graph G∗ ∈ G such that x∗ is the output of the heuristic algorithm by solving G∗ . We take DAG scheduling as an example to clarify this Proposition . Without loss of generality , we define processing the nodes 1 to n sequentially as a feasible solution . Then , we can modify the graph as follows : if the edge connecting i to i+ 1 does not exist , it is added . After adding all edges from 1 to n , processing the nodes from 1 to n sequentially is the only feasible solution , which is also the output of any heuristic algorithm . The above construction method applies for all solutions in X . Some Remarks . Our hypothesis and proposition provide the theoretical grounding of developing graph-modification methods to tackle combinatorial problems . It is worth noting that G∗ only suggests that graph modification is a promising direction and finding G∗ given G is usually NP-hard in practice . In this paper , we propose to improve the solving quality for heuristic algorithms by finding optimized ( not necessarily optimal ) graphs by learning based on the bi-level reformulation of the original single-level problem . Due to practical reasons , we restrict the max number of modifications . In the bi-level reformulation of the original single-level problem , an optimized graph G′ is introduced : min x′ , G′ f ( x′|G ) s.t . Hj ( G′ , G ) ≤ 0 , for j = 1 ... J x′ ∈ argmin x′ { f ( x′|G′ ) : hi ( x′ , G′ ) ≤ 0 , for i = 1 ... I } ( 2 ) where f ( x′|G ) , f ( x′|G′ ) are the objectives for upper- and lower-level problems , respectively . The lower-level problem is the CO given the optimized graph G′ , which is solved by a heuristic algorithm . The solved decision variable x′ is further fed to the upper-level problem , whose objective f ( x′|G ) denotes the original CO objective computed by x′ . The upper-level constraints Hj ( G′ , G ) ≤ 0 ensure that the feasible space of G′ is a subset of G , and G′ has at most K modification steps from G. The upper-level problem optimizes G′ by an RL agent , by regarding Eq . 2 as the environment .

Summary: This paper presents an interesting idea to solve Combinatorial Optimization on Graphs by using Reinforcement Learning in order to modify the underlying graph in order to make it easier for a heuristic algorithm. The reward for the RL algorithm is simply the increase found by the heuristic algorithm between the original graph and the modified one. The authors test their algorithm on several real-world dataset related to DAG scheduling, graph edit distance and hamiltonian cycle problem.