Learning Hard Optimization Problems: A Data Generation Perspective

1 Introduction . Constrained optimization ( CO ) is in daily use in our society , with applications ranging from supply chains and logistics , to electricity grids , organ exchanges , marketing campaigns , and manufacturing to name only a few . Two classes of hard optimization problems of particular interest in many fields are ( 1 ) combinatorial optimization problems and ( 2 ) nonlinear constrained problems . Combinatorial optimization problems are characterized by discrete search spaces and have solutions that are combinatorial in nature , involving for instance , the selection of subsets or permutations , and the sequencing or scheduling of tasks . Nonlinear constrained problems may have continuous search spaces but are often characterized by highly nonlinear constraints , such as those arising in electrical power systems whose applications must capture physical laws such as Ohm ’ s law and Kirchhoff ’ s law in addition to engineering operational constraints . Such CO problems are often NP-Hard and may be computationally challenging in practice , especially for large-scale instances . While the AI and Operational Research communities have contributed fundamental advances in optimization in the last decades , the complexity of some problems often prevents them from being adopted in contexts where many instances must be solved over a long-term horizon ( e.g. , multiyear planning studies ) or when solutions must be produced under time constraints . Fortunately , in many practical cases , including the scheduling and energy problems motivating this work , one is interested in solving many problem instances sharing similar patterns . Therefore , the application of deep learning methods to aid the solving of computationally challenging constrained optimization problems appears to be a natural approach and has gained traction in the nascent area at the intersection 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . between CO and ML [ 5 , 19 , 31 ] . In particular , supervised learning frameworks can train a model using pre-solved CO instances and their solutions . However , learning the underlying combinatorial structure of the problem or learning approximations of optimization problems with hard physical and engineering constraints may be an extremely difficult task . While much of the recent research at the intersection of CO and ML has focused on learning good CO approximations in jointly training prediction and optimization models [ 3 , 18 , 22 , 25 , 32 ] and incorporating optimization algorithms into differentiable systems [ 1 , 27 , 34 , 20 ] , learning the combinatorial structure of CO problems remains an elusive task . Beside the difficulty of handling hard constraints , which will almost always exhibit some violations , two interesting challenges have emerged : the presence of multiple , often symmetric , solutions , and the learning of approximate solution methods . The first challenge recognizes that an optimization problem may not have a unique solution . This challenge is illustrated in Figure 1 , where the various y ( i ) represent optimal solutions to CO instances x ( i ) and C the feasible space . As a result , a combinatorial number of possible datasets may be generated . While equally valid as optimal solutions , some sets follow patterns which are more meaningful and recognizable . Symmetry breaking is of course a major area of combinatorial optimization and may alleviate some of these issues . But different instances may not break symmetries in the same fashion , thus creating datasets that are harder to learn . The second challenge comes from realities in the application domain . Because of time constraints , the solution technique may return a sub-optimal solution . Moreover , modern com- binatorial optimization techniques often use randomization and large neighborhood search to produce high-quality solutions quickly . Although these are widely successful , different runs for the same , or similar , instances may produce radically different solutions . As a result , learning the solutions returned by these approximations may be inherently more difficult . These effects may be viewed as a source of noise that obscures the relationships between training data and their target outputs . Although this does not raise issues for optimization systems , it creates challenging learning tasks . This paper demonstrates these relations , connects the variation in the training data to the ability of a model to approximate it , and proposes a method for producing ( exact or approximate ) solutions to optimization problems that are more amenable to supervised learning tasks . More concretely , the paper makes the following contributions : 1 . It shows that the existence of co-optimal or approximated solutions obtained by solving hard CO problems to construct training datasets challenges the learnability of the task . 2 . To overcome this limitation , it introduces the problem of optimal dataset design , which is cast as a bilevel optimization problem . The optimal dataset design problem is motivated using theoretical insights on the approximation of functions by neural networks , relating the properties of a function describing a training dataset to the model capacity required to represent it . 3 . It introduces a tractable algorithm for the generation of datasets that are amenable to learning , and empirical demonstration of marked improvements to the accuracy of trained models , as well as the ability to satisfy constraints at inference time . 4 . Finally , it provides state-of-the-art accuracy results at vastly enhanced computational runtime on learning two challenging optimization problems : Job Shop Scheduling problems and Optimal Power Flow problems for energy networks . To the best of the authors ’ knowledge , this work is the first to highlight the issue of learnability in the face of co-optimal or approximate solutions obtained to generate training data for learning to approximate hard CO problems . The observations raised in this work may result in a broader impact as , in addition to approximating hard optimization problems , the optimal dataset generation strategy introduced in this paper may be useful to the line of work on integrating CO as differentiable layers for predictive and prescriptive analytics , as well as for physics constrained learning problems such as when approximating solutions to systems of partial differential equation . 2 Related work . The integration of CO models into ML pipelines to meld prediction and decision models has recently seen the introduction of several nonoverlapping approaches , surveyed by Kotary et al . [ 19 ] . The application of ML to boost performance in traditional search methods through branching rules and enhanced heuristics is broadly overviewed by Bengio et al . [ 5 ] . Motivated also by the need for fast approximations to combinatorial optimization problems , the training of surrogate models via both supervised and reinforcement learning is an active area for which a thorough review is provided by Vesselinova et al . [ 31 ] . Designing surrogate models with the purpose to approximate hard CO problem has been studied in a number of works , including [ 6 , 12 , 15 ] . An important aspect for the learned surrogate models is the prediction of solutions that satisfy the problem constraints . While this is a very difficult task in general , several methodologies have been devised . In particular , Lagrangian loss functions have been used for encouraging constraint satisfaction in several applications of deep learning , including fairness [ 30 ] and energy problems [ 15 ] . Other methods iteratively modify training labels to encourage satifaction of constraints during training [ 12 ] . This work focuses on an orthogonal direction with respect to the literature reviewed above . Rather than devising a new methodology for effectively producing a surrogate model that approximate some hard optimization problem , it studies the machine learning task from a data generation perspective . It shows that the co-optimality and symmetries of a hard CO problem may be viewed as a source of noise that obscures the relationships between training data and the target outputs and proposes an optimal data generation approach to mitigate these important issues . 3 Preliminaries . A constrained optimization ( CO ) problem poses the task of minimizing an objective function f : Y × X → R+ of one or more variables y ∈ Y ⊆ Rn , subject to the condition that a set of constraints Cx are satisfied between the variables and where x ∈ X ⊆ Rm denotes a vector of input data that specifies the problem instance : O ( x ) = argmin y f ( y , x ) subject to : y ∈ Cx . ( 1 ) An assignment of values y which satisfies Cx is called a feasible solution ; if , additionally f ( y , x ) ≤ f ( w , x ) for all feasible w , it is called an optimal solution . A particularly common constraint set arising in practical problems takes the form C = { y : Ay ≤ b } , whereA ∈ Rm×n and b ∈ Rm . In this case , C is a convex set . If the objective f is an affine function , the problem is referred to as linear program ( LP ) . If , in addition , some subset of a problem ’ s variables are required to take integer values , it is called mixed integer program ( MIP ) . While LPs with convex objectives belong to the class of convex problems , and can be solved efficiently with strong theoretical guarantees on the existence and uniqueness of solutions [ 7 ] , the introduction of integral constraints ( y ∈ Nn ) results in a much more difficult problem . The feasible set in MIP consists of distinct points in y ∈ Rn , not only nonconvex but also disjoint , and the resulting problem is , in general , NP-Hard . Finally , nonlinear programs ( NLPs ) are optimization problems where some of the constraints or the objective function are nonlinear . Many NLPs are nonconvex and can not be efficiently solved [ 24 ] . The methodology introduced in this paper is illustrated on hard MIP and nonlinear program instances . 4 Problem setting and goals . This paper focuses on learning approximate solutions to problem ( 1 ) via supervised learning . The task considers datasets χ = { ( x ( i ) , y ( i ) ) } Ni=1 consisting of N data points with x ( i ) ∈ X being a vector of input data , as defined in equation ( 1 ) , and y ( i ) ∈ O ( x ( i ) ) being a solution of the optimization task . A desirable , but not always achievable , property is for the solutions y ( i ) to be optimal . The goal is to learn a model fθ : X → Y , where θ is a vector of real-valued parameters , and whose quality is measured in terms of a nonnegative , and assumed differentiable , loss function ` : Y ×Y → R+ . The learning task minimizes the empirical risk function ( ERM ) : min θ J ( fθ ; χ ) = 1 N N∑ i=1 ` ( fθ ( x ( i ) ) , y ( i ) ) , ( 2 ) with the desired goal that the predictions also satisfy the problem constraints : fθ ( x ( i ) ) ∈ Cx . While the above task is difficult to achieve due to the presence of constraints , the paper adopts a Lagrangian approach [ 14 ] , which has been shown successful in learning constrained representations , along with projection operators , commonly applied in constrained optimization to ensure constraint satisfaction of an assignment . For a given point ŷ , e.g. , representing the model prediction , a projection operator πC ( ŷ ) finds the closest feasible point y ∈ C to ŷ under a p-norm : πC ( ŷ ) def = argmin y ‖y − ŷ‖p subject to : y ∈ C. The full description of the Lagrangian based approach and the projection method adopted is delegated to the Appendix A .

Summary: This paper proposes a method to optimize training data generation for supervised learning approaches to constrained optimization. First the authors show that the existence of various optimal solutions to the problem can make the learning more challenging. Then they Then they proceed with some theoretical insights that link this challenge to the representation capacity of ReLU neural networks. The paper introduces a method for training data generation for CO learning such that CO instances that are close (in terms of specifications) have solutions that are close too. Numerical validation is presented on a job shop scheduling problem (combinatorial optimization) and an Optimal Power Flow problem (non-linear, non-convex constrained pb).