NN-Baker: A Neural-network Infused Algorithmic Framework for Optimization Problems on Geometric Intersection Graphs

1 Introduction . Many tasks in science and engineering can be naturally modeled by combinatorial optimization problems over graphs , such as maximum independent set , minimum vertex cover , minimum multiway cut , maximum clique , and so on . These problems are often NP-hard . Hence there has been great effort devoted to developing efficient approximation algorithms . However , many such problems are hard to approximate in the general setting as well : for example , it is known that the maximum independent set problem is NP-hard , and it is even NP-hard to approximate within n1−ε on n-vertex graphs , for any fixed ε > 0 [ 1 ] . On the other hand , many of these hard problems admit PTAS in the geometric setting when the graphs are assumed to be induced by points in fixed-dimensional Euclidean space . Here , a PTAS ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( polynomial-time approximation scheme ) is a polynomial time algorithm which for any fixed ε > 0 , can approximate a maximization problem within a factor of ( 1 + ε ) ( or within a factor of ( 1− ε ) for a minimization problem ) . One most prominent example is the travelling salesman problem ( TSP ) , where in general there is no PTAS available ( and currently the best known approximation algorithm achieves a factor of 3/2 − α , for some α > 10−36 [ 2 ] ) . However a PTAS , more specifically in this case a near linear time ( 1 + ε ) -approximation algorithm , was developed for the Euclidean TSP problem in a ground breaking work by Arora [ 3 ] . The maximum independent set problem is known to remain NP-hard in the geometric setting [ 4 ] , but it is known that it also admits a PTAS [ 5 , 6 , 7 ] . Nevertheless , for problems of large size , even these PTAS are still not practical , especially since many of them involve large constants that are exponential in some fixed parameter ( e.g , having a term like n 1 ε where ( 1 + ε ) is the approximation factor ) . In applications , practitioners often rely on handcrafted heuristic search strategies to find high-quality solutions . Recently , there has been a surge on using machine learning ( ML ) for combinatorial optimization problems ; see the review paper by Bengio et al . [ 8 ] which also provides a nice categorization of different ways that a ML component may contribute to the combinatorial optimization problems . Earlier such approaches focus on several graph combinatorial optimization problems , including TSP , minimum vertex cover and maximum independent sets ; see e.g. , [ 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ] . More recently , there has been a range of approaches developed to tackle more general mixed integer linear programming problems ( MILP ) ; e.g , [ 17 , 18 , 19 , 20 , 21 ] . See Section 1.1 for more detailed description of some related work . In general , an ML framework can be trained to solve an optimization problem in an end-to-end manner ( e.g , [ 10 , 11 , 12 , 15 , 16 ] ) . Alternatively , many recent approaches use ML module as a component within a specific algorithmic framework to help make hopefully better ( either in terms of quality or efficiency ) decisions during the execution of this algorithm : such as using imitation learning or reinforcement learning to learn a good policy to decide which variable to branch in a branch-and-bound algorithm [ 18 , 20 ] . Despite the tremendous progress in combining ML with optimization problems , theoretical understanding remains limited : Does a proposed ML pipeline have the expressiveness capacity to solve the combinatorial optimization problem exactly or approximately ? What is a suitable model for input data distribution to talk about generalization ? Consider the capacity question : While neural networks are known to have many universal approximation results for real or complex valued continuous functions ( or some other special families of functions ) , e.g , [ 22 , 23 , 24 , 25 ] , combinatorial optimization problems , say finding the maximum independent set , can not be easily modeled to fit into such function approximation frameworks . Nevertheless , a very interesting recent work [ 13 ] shows that for graph combinatorial optimization problems , the so-called vector-vector consistent graph neural networks ( VVC -GNNs ) , can solve the same family of problems as the distributed local algorithms in the so-called port-numbering models . Leveraging the literature in distributed local algorithms [ 26 , 27 ] , this leads to several positive results on the capacity of GNNs for approximating minimum vertex cover or maximum matching problems with certain constant factors , however only for graphs with bounded degrees – intuitively , the depth of the GNN will depend on the bound ∆ on the maximum node degree of input graphs . Unfortunately , the connection to distributed local algorithms also leads to negative results : roughly speaking , these constant factor approximations for the special family of constant-degree graphs are the best that a GNN can do for these graph optimization problems . See [ 13 ] for details . Our work . The aforementioned results ( on the capacity of GNNs ) are for the case where GNNs are used to solve an optimization problem in an end-to-end manner , and other than the bound on max-degree , the input graphs are abstract graphs . In this paper , we advocate the study of using ML for optimization problems in the geometric setting where ( graph ) optimization problems are induced by points in a fixed dimensional Euclidean space Rd . This is a rather common scenario in practice , such as solving TSP in road networks spanned by cities in R2 , or solving maximum independent set in a communication network spanned by sensor nodes in R2/R3 ( i.e , the unit-ball model that we will introduce in Section 2.1 ) . Such graphs can also be the result of an embedding of an input arbitrary graph into a certain latent space . At the same time , the geometric setting brings special structures to the problem at hand , which an algorithm and also ML can then leverage . In particular , using the maximum independent set ( MIS ) problem as an example , we first propose what we call the Baker-paradigm in Section 2 , which is an approximation framework for geometric optimization problems inspired by Baker ’ s work in [ 28 ] . We show that the Baker-paradigm gives a bi-criteria approximation for the MIS in the Euclidean setting ( Theorem 2.1 ) . The running time is only linear in the size of input point set , but exponential in terms of the parameters . This framework is general and can be extended to several other geometric optimization problems ( Section 2.3 ) . A key advantage of our Baker-paradigm is that it decomposes the problem into ( at most a linear number of ) small sub-problems of fixed size ( independent of size of input graphs ) . For the family of such fixed-size sub-problems , we can now design neural networks with universal approximation guarantees to solve them . Using such a neural network to replace ( Step-2 ) of our Baker-paradigm , we then obtain a mixed algorithmic-ML framework , which we refer to as NN-Baker , that has the capacity to produce a bi-criteria approximation of MIS within any constant factor in near linear time ( i.e , a bi-criteria PTAS ) ; see Section 3.1 and Theorem 3.2 . Note that while Theorem 2.1 already gives a near-linear time bi-criteria PTAS for MIS , the constant involved in the time complexity is exponential in the approximation parameters , making it inefficient in practice . In contrast , the NN-Baker will replace the costly component by a neural network component , and only call this neural network at most n times . Other than calls to neural networks , the time needed is Θ ( n ) where the constant contains only terms polynomial in the approximation paramters . The resulting mixed algorithmic-ML framework is very efficient , as we show in Section 4 . In Section 3.2 , we provide two instantiations of the NN component for our NN-Baker : based on CNN and GNN respectively . We present a range of experimental results in Section 4 to show the effectiveness of both the CNN-Baker and GNN-Baker frameworks . Note that our NN-Baker can be used together with other SOA NN framework to solve combinatorial optimization problems and to further improve them ( sometimes significantly ) . For example , we deploy different SOA GNNs for graph optimization problems to instantiate ( Step-2 ) in NN-Baker , including TGS of [ 12 ] in the supervised setting , LwD of [ 16 ] in the reinforcement learning setting , as well as Erdős-GNN [ 15 ] in the unsupervised setting . We show that as the problem size increases , the performance of each original GNN decreases ( the GNN is always trained and tested on problems of similar sizes ) . However , using GNN-Baker significantly improves the performance of the original GNN as the problem size increases – This is partly because , independent of the problem size , in ( Step 2 ) our GNN-Baker only needs to train and test the GNN component on a small graph ( of bounded size ) . Thus the trained GNN can adapt to the problem structure much better and require much fewer training samples . Our NN-Baker is , to our best knowledge , the first ( bi-criteria ) PTAS for a combinatorial problem for an ML-based approach ( in terms of expressiveness ) . The recent line of work of using a ML component ( e.g a GNN trained by imitation learning ) to make branching decisions within the branchand-bound algorithmic framework [ 18 , 20 ] may solve the exact problem given enough running time . However , the number of times the algorithm calls the NN component may be exponential in the input size . Instead , our NN-Baker framework calls the neural network ( which has a bounded size ) only a linear number of times . Our approach can open new directions to design NN-infused algorithmic frameworks with theoretical guarantees , by for example , leveraging divide and conquer paradigm and replacing certain algorithmic components by neural components . 1.1 More on related work . The idea of using neural networks to tackle optimization problems traces back to the 1980 ’ s . One of the most important frameworks in this direction is the Hopfield Neural Network ( HNN ) [ 29 , 30 ] . In particular , the HNN is a feedback neural network with pre-specified weights ( whose assignments depend on the optimization problem at hand ) , which encodes a dynamic system whose associated energy function characterize the optimization problem to be solved . To use it to solve an optimization problem , one starts with an initial state , and iterates till convergence . As branch-and-bound ( B & B ) has been proven to be a powerful framework in solving optimization problems , especially in MILP ( mixed integer-linear programing ) problems , researchers proposed different machine learning algorithms to boost B & B . [ 17 ] provided a list of features of variables and designed a learning-to-rank model on selecting branching variables . [ 18 , 20 , 21 ] developed GNN approaches to learn the policy of choosing branching variables after formulating MILP problems as graphs by imitation learning or reinforcement learning . Besides MILP problems , GNNs are also applied on graph combinatorial optimization problems . [ 9 ] encodes the input graphs as sequences and takes an attention RNN to process the sequences . It can be used to compute convex hulls or solve problems like TSP . [ 10 , 11 , 31 , 32 , 16 ] takes reinforcement learning on graphs to solve routing problems like TSP , and other problems like maximum independent set . [ 12 ] solves graph theory problems by supervised learning setup after solving a set of cases as training set by existing solvers , while [ 15 ] introduces unsupervised approaches by designing loss function and training setup based on objective functions and variables constraints . In addition , there are works [ 13 , 33 , 34 ] that study the power of GNN on solving different kinds of combinatorial optimization problems .

Summary: A machine learning framework for constructing heuristics for certain geometric graph optimization problems is proposed. Such graphs are derived from points on a 2d grid, for example. This geometric structure is typically useful for the design of approximation algorithms. This insight inspires the main question addressed in this paper: does the geometric structure also help improve ML approaches for graph optimization problems such as Maximum Independent Set (MIS), both theoretically and practically?  The authors answer affirmatively by leveraging an existing algorithmic paradigm from Baker (1994). The key idea is to decompose the space into a number of independent subproblems (linear in the graph size) of bounded sizes, then heuristically solving those small subproblems using an ML model. The ML model may be a convolutional network that see the subproblem is a pixel grid, or a graph neural network that acts directly on the small graph. The ML model may also be trained by supervision or without.   Theoretically, it is shown that if the ML model has universal approximation guarantees (e.g., 1-layer neural network), meaning that it can accurately approximate the function mapping from subproblem to subproblem optimum, then one obtains a bicriteria approximation guarantee on the global optimization solution, which is assembled appropriately from subproblem solutions.  Experimentally, it is shown that this decomposition approach can be combined with three different ML model classes, improving their original performance on a number of random problem families for MIS and vertex cover.