NN-Baker: A Neural-network Infused Algorithmic Framework for Optimization Problems on Geometric Intersection Graphs

1 Introduction . Many tasks in science and engineering can be naturally modeled by combinatorial optimization problems over graphs , such as maximum independent set , minimum vertex cover , minimum multiway cut , maximum clique , and so on . These problems are often NP-hard . Hence there has been great effort devoted to developing efficient approximation algorithms . However , many such problems are hard to approximate in the general setting as well : for example , it is known that the maximum independent set problem is NP-hard , and it is even NP-hard to approximate within n1−ε on n-vertex graphs , for any fixed ε > 0 [ 1 ] . On the other hand , many of these hard problems admit PTAS in the geometric setting when the graphs are assumed to be induced by points in fixed-dimensional Euclidean space . Here , a PTAS ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( polynomial-time approximation scheme ) is a polynomial time algorithm which for any fixed ε > 0 , can approximate a maximization problem within a factor of ( 1 + ε ) ( or within a factor of ( 1− ε ) for a minimization problem ) . One most prominent example is the travelling salesman problem ( TSP ) , where in general there is no PTAS available ( and currently the best known approximation algorithm achieves a factor of 3/2 − α , for some α > 10−36 [ 2 ] ) . However a PTAS , more specifically in this case a near linear time ( 1 + ε ) -approximation algorithm , was developed for the Euclidean TSP problem in a ground breaking work by Arora [ 3 ] . The maximum independent set problem is known to remain NP-hard in the geometric setting [ 4 ] , but it is known that it also admits a PTAS [ 5 , 6 , 7 ] . Nevertheless , for problems of large size , even these PTAS are still not practical , especially since many of them involve large constants that are exponential in some fixed parameter ( e.g , having a term like n 1 ε where ( 1 + ε ) is the approximation factor ) . In applications , practitioners often rely on handcrafted heuristic search strategies to find high-quality solutions . Recently , there has been a surge on using machine learning ( ML ) for combinatorial optimization problems ; see the review paper by Bengio et al . [ 8 ] which also provides a nice categorization of different ways that a ML component may contribute to the combinatorial optimization problems . Earlier such approaches focus on several graph combinatorial optimization problems , including TSP , minimum vertex cover and maximum independent sets ; see e.g. , [ 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ] . More recently , there has been a range of approaches developed to tackle more general mixed integer linear programming problems ( MILP ) ; e.g , [ 17 , 18 , 19 , 20 , 21 ] . See Section 1.1 for more detailed description of some related work . In general , an ML framework can be trained to solve an optimization problem in an end-to-end manner ( e.g , [ 10 , 11 , 12 , 15 , 16 ] ) . Alternatively , many recent approaches use ML module as a component within a specific algorithmic framework to help make hopefully better ( either in terms of quality or efficiency ) decisions during the execution of this algorithm : such as using imitation learning or reinforcement learning to learn a good policy to decide which variable to branch in a branch-and-bound algorithm [ 18 , 20 ] . Despite the tremendous progress in combining ML with optimization problems , theoretical understanding remains limited : Does a proposed ML pipeline have the expressiveness capacity to solve the combinatorial optimization problem exactly or approximately ? What is a suitable model for input data distribution to talk about generalization ? Consider the capacity question : While neural networks are known to have many universal approximation results for real or complex valued continuous functions ( or some other special families of functions ) , e.g , [ 22 , 23 , 24 , 25 ] , combinatorial optimization problems , say finding the maximum independent set , can not be easily modeled to fit into such function approximation frameworks . Nevertheless , a very interesting recent work [ 13 ] shows that for graph combinatorial optimization problems , the so-called vector-vector consistent graph neural networks ( VVC -GNNs ) , can solve the same family of problems as the distributed local algorithms in the so-called port-numbering models . Leveraging the literature in distributed local algorithms [ 26 , 27 ] , this leads to several positive results on the capacity of GNNs for approximating minimum vertex cover or maximum matching problems with certain constant factors , however only for graphs with bounded degrees – intuitively , the depth of the GNN will depend on the bound ∆ on the maximum node degree of input graphs . Unfortunately , the connection to distributed local algorithms also leads to negative results : roughly speaking , these constant factor approximations for the special family of constant-degree graphs are the best that a GNN can do for these graph optimization problems . See [ 13 ] for details . Our work . The aforementioned results ( on the capacity of GNNs ) are for the case where GNNs are used to solve an optimization problem in an end-to-end manner , and other than the bound on max-degree , the input graphs are abstract graphs . In this paper , we advocate the study of using ML for optimization problems in the geometric setting where ( graph ) optimization problems are induced by points in a fixed dimensional Euclidean space Rd . This is a rather common scenario in practice , such as solving TSP in road networks spanned by cities in R2 , or solving maximum independent set in a communication network spanned by sensor nodes in R2/R3 ( i.e , the unit-ball model that we will introduce in Section 2.1 ) . Such graphs can also be the result of an embedding of an input arbitrary graph into a certain latent space . At the same time , the geometric setting brings special structures to the problem at hand , which an algorithm and also ML can then leverage . In particular , using the maximum independent set ( MIS ) problem as an example , we first propose what we call the Baker-paradigm in Section 2 , which is an approximation framework for geometric optimization problems inspired by Baker ’ s work in [ 28 ] . We show that the Baker-paradigm gives a bi-criteria approximation for the MIS in the Euclidean setting ( Theorem 2.1 ) . The running time is only linear in the size of input point set , but exponential in terms of the parameters . This framework is general and can be extended to several other geometric optimization problems ( Section 2.3 ) . A key advantage of our Baker-paradigm is that it decomposes the problem into ( at most a linear number of ) small sub-problems of fixed size ( independent of size of input graphs ) . For the family of such fixed-size sub-problems , we can now design neural networks with universal approximation guarantees to solve them . Using such a neural network to replace ( Step-2 ) of our Baker-paradigm , we then obtain a mixed algorithmic-ML framework , which we refer to as NN-Baker , that has the capacity to produce a bi-criteria approximation of MIS within any constant factor in near linear time ( i.e , a bi-criteria PTAS ) ; see Section 3.1 and Theorem 3.2 . Note that while Theorem 2.1 already gives a near-linear time bi-criteria PTAS for MIS , the constant involved in the time complexity is exponential in the approximation parameters , making it inefficient in practice . In contrast , the NN-Baker will replace the costly component by a neural network component , and only call this neural network at most n times . Other than calls to neural networks , the time needed is Θ ( n ) where the constant contains only terms polynomial in the approximation paramters . The resulting mixed algorithmic-ML framework is very efficient , as we show in Section 4 . In Section 3.2 , we provide two instantiations of the NN component for our NN-Baker : based on CNN and GNN respectively . We present a range of experimental results in Section 4 to show the effectiveness of both the CNN-Baker and GNN-Baker frameworks . Note that our NN-Baker can be used together with other SOA NN framework to solve combinatorial optimization problems and to further improve them ( sometimes significantly ) . For example , we deploy different SOA GNNs for graph optimization problems to instantiate ( Step-2 ) in NN-Baker , including TGS of [ 12 ] in the supervised setting , LwD of [ 16 ] in the reinforcement learning setting , as well as Erdős-GNN [ 15 ] in the unsupervised setting . We show that as the problem size increases , the performance of each original GNN decreases ( the GNN is always trained and tested on problems of similar sizes ) . However , using GNN-Baker significantly improves the performance of the original GNN as the problem size increases – This is partly because , independent of the problem size , in ( Step 2 ) our GNN-Baker only needs to train and test the GNN component on a small graph ( of bounded size ) . Thus the trained GNN can adapt to the problem structure much better and require much fewer training samples . Our NN-Baker is , to our best knowledge , the first ( bi-criteria ) PTAS for a combinatorial problem for an ML-based approach ( in terms of expressiveness ) . The recent line of work of using a ML component ( e.g a GNN trained by imitation learning ) to make branching decisions within the branchand-bound algorithmic framework [ 18 , 20 ] may solve the exact problem given enough running time . However , the number of times the algorithm calls the NN component may be exponential in the input size . Instead , our NN-Baker framework calls the neural network ( which has a bounded size ) only a linear number of times . Our approach can open new directions to design NN-infused algorithmic frameworks with theoretical guarantees , by for example , leveraging divide and conquer paradigm and replacing certain algorithmic components by neural components . 1.1 More on related work . The idea of using neural networks to tackle optimization problems traces back to the 1980 ’ s . One of the most important frameworks in this direction is the Hopfield Neural Network ( HNN ) [ 29 , 30 ] . In particular , the HNN is a feedback neural network with pre-specified weights ( whose assignments depend on the optimization problem at hand ) , which encodes a dynamic system whose associated energy function characterize the optimization problem to be solved . To use it to solve an optimization problem , one starts with an initial state , and iterates till convergence . As branch-and-bound ( B & B ) has been proven to be a powerful framework in solving optimization problems , especially in MILP ( mixed integer-linear programing ) problems , researchers proposed different machine learning algorithms to boost B & B . [ 17 ] provided a list of features of variables and designed a learning-to-rank model on selecting branching variables . [ 18 , 20 , 21 ] developed GNN approaches to learn the policy of choosing branching variables after formulating MILP problems as graphs by imitation learning or reinforcement learning . Besides MILP problems , GNNs are also applied on graph combinatorial optimization problems . [ 9 ] encodes the input graphs as sequences and takes an attention RNN to process the sequences . It can be used to compute convex hulls or solve problems like TSP . [ 10 , 11 , 31 , 32 , 16 ] takes reinforcement learning on graphs to solve routing problems like TSP , and other problems like maximum independent set . [ 12 ] solves graph theory problems by supervised learning setup after solving a set of cases as training set by existing solvers , while [ 15 ] introduces unsupervised approaches by designing loss function and training setup based on objective functions and variables constraints . In addition , there are works [ 13 , 33 , 34 ] that study the power of GNN on solving different kinds of combinatorial optimization problems .

Summary: The paper proposes a polynomial time approximation scheme that can utilize neural networks to efficiently solve combinatorial problems like maximum independent set and minimum vertex cover on geometric interesection graphs. The method decomposes a problem instance into multiple smaller instances of bounded size located on a grid, which are then solved with the help of a neural network. The neural network is either a graph neural network applied on the intersection graph or a CNN that treats the problem as a segmentation task. To discretize the output of the network the authors use a greedy  rounding strategy that preserves feasibility.