Title: Dual Lagrangian Learning for Conic Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Conic optimization

optimization proxies

duality

self-supervised learning
Abstract: This paper presents Dual Lagrangian Learning (DLL), a principled learning methodology for dual conic optimization proxies.
DLL leverages conic duality and the representation power of ML models to provide high-duality, dual-feasible solutions, and therefore valid Lagrangian dual bounds, for linear and nonlinear conic optimization problems.
The paper introduces a systematic dual completion procedure, differentiable conic projection layers, and a self-supervised learning framework based on Lagrangian duality.
It also provides closed-form dual completion formulae for broad classes of conic problems, which eliminate the need for costly implicit layers.
The effectiveness of DLL is demonstrated on linear and nonlinear conic optimization problems.
The proposed methodology significantly outperforms a state-of-the-art learning-based method, and achieves 1000x speedups over commercial interior-point solvers with optimality gaps under 0.5\% on average.

Dual Lagrangian Learning for Conic Optimization

Abstract

This paper presents Dual Lagrangian Learning (DLL), a principled learning methodology for dual conic optimization proxies. DLL leverages conic duality and the representation power of ML models to provide high-duality, dual-feasible solutions, and therefore valid Lagrangian dual bounds, for linear and nonlinear conic optimization problems. The paper introduces a systematic dual completion procedure, differentiable conic projection layers, and a self-supervised learning framework based on Lagrangian duality. It also provides closed-form dual completion formulae for broad classes of conic problems, which eliminate the need for costly implicit layers. The effectiveness of DLL is demonstrated on linear and nonlinear conic optimization problems. The proposed methodology significantly outperforms a state-of-the-art learning-based method, and achieves 1000x speedups over commercial interior-point solvers with optimality gaps under 0.5% on average.

1 Introduction

From power systems and manufacturing to supply chain management, logistics and healthcare, optimization technology underlies most aspects of the economy and society. Over recent years, the substantial achievements of Machine Learning (ML) have spurred significant interest in combining the two methodologies. This integration has led to the development of new optimization algorithms (and the revival of old ones) taylored to ML problems, as well as new ML techniques for improving the resolution of hard optimization problems [1]. This paper focuses on the latter (ML for optimization), specifically, the development of so-called optimization proxies , i.e., ML models that provide approximate solutions to parametric optimization problems, see e.g., [2].

In that context, considerable progress has been made in learning primal solutions for a broad range of problems, from linear to discrete and nonlinear, non-convex optimization problems. State-of-the-art methods can now predict high-quality, feasible or close-to-feasible solutions for various applications [2]. This paper complements these methods by learning dual solutions which, in turn, certify the (sub)optimality of learned primal solutions. Despite the fundamental role of duality in optimization, there is no dedicated framework for dual optimization proxies, which have seldom received any attention in the literature. The paper addresses this gap by proposing, for the first time, a principled learning methodology that combines conic duality theory with Machine Learning. As a result, it becomes possible, for a large class of optimization problems, to design a primal proxy to deliver a high-quality primal solution and an associated dual proxy to obtain a quality certificate.

1.1 Contributions and outline

The core contribution of the paper is the Dual Lagrangian Learning (DLL) methodology for learning dual-feasible solutions for parametric conic optimization problems. DLL leverages conic duality to design a self-supervised Lagrangian loss for training dual conic optimization proxies. In addition, the

paper proposes a general dual conic completion using differential conic projections and implicit layers to guarantee dual feasibility, which yields stronger guarantees than existing methods for constrained optimization learning. Furthermore, it presents closed-form analytical solutions for conic projections, and for dual conic completion across broad classes of problems. This eliminates the need for implicit layers in practice. Finally, numerical results on linear and nonlinear conic problems demonstrate the effectiveness of DLL, which a outperforms state-of-the-art learning baseline, and yields significant speedups over interior-point solvers.

The rest of the paper is organized as follows. Section 2 presents the relevant literature. Section 3 introduces notations and background material. Section 4 presents the DLL methodology, which comprises the Lagrangian loss, dual completion strategy, and conic projections. Section 5 reports numerical results. Section 6 discusses possible the limitations of DLL and possible extensions, and Section 7 concludes the paper.

2 Related works

Constrained Optimization Learning The vast majority of existing works on optimization proxies focuses on learning primal solutions and, especially, on ensuring their feasibility. This includes, for instance, physics-informed training loss [3, 4, 5], mimicking the steps of an optimization algorithm [6, 7, 8], using masking operations [9, 10], or designing custom projections and feasibility layers [11, 12]. The reader is referred to [2] for an extensive survey of constrained optimization learning.

Only a handful of methods offer feasibility guarantees, and only for convex constraints; this latter point is to be expected since satisfying non-convex constraints is NP-hard in general. Implicit layers [13] have a high computational cost, and are therefore impractical unless closed-form solutions are available. DC3 [5] uses equality completion and inequality correction, and is only guaranteed to converge for convex constraints and given enough correction steps. LOOP-LC [14] uses a gauge mapping to ensure feasibility for bounded polyhedral domains. RAYEN [12] and the similar work in [15] use a line search-based projection mechanism to handle convex constraints. All above methods employ equality completion, and the latter three [14, 12, 15] assume knowledge of a strictly feasible point, which is not always available.

Dual Optimization Learning To the authors' knowledge, dual predictions have received very little attention, with most works using them to warm-start an optimization algorithm. In [16], a primal-dual prediction is used to warm-start an ADMM algorithm, while These works consider specific applications, and do not provide dual feasibility guarantees. More recently, [6, 8] attempt to mimic the (dual) steps of an augmented Lagrangian method, however with the goal of obtaining high-quality primal solutions.

In the mixed-integer programming (MIP) setting, [17] and [18] use a dual prediction as warm-start in a column-generation algorithm, for cutting-stock and unit-commitment problems, respectively. In a similar fashion, [19] consider a (combinatorial) Lagrangian relaxation of Traveling Salesperson Problem (TSP) Most recently, [20] consider learning Lagrangian multipliers for mixed-integer linear programs. Therein, a machine learning model predicts Lagrange multipliers, and a Lagrangian subproblem is solved to obtained a Lagrangian dual bound. This approach only supports linear constraints for the Lagrangian, and it requires an external combinatorial solver to solve the subproblem, which may be NP-hard in general.

The first work to explicitly consider dual proxies in the context of conic optimization, and to offer dual feasibility guarantees, is [21], which learns a dual proxy for a second-order cone relaxation of the AC Optimal Power Flow. Klamkin et al. [22] later introduce a dual interior-point learning algorithm to speed-up the training of dual proxies for bounded linear programming problems. In contrast, this paper proposes a general methodology for conic optimization problems, thus generalizing the approach in [21], and provides more extensive theoretical results. The dual completion procedure used in [22, Lemma 1] is a special case of the one proposed in this paper.

3 Background

This section introduces relevant notations and standard results on conic optimization and duality, which lay the basis for the proposed learning methodology. The reader is referred to [23] for a thorough overview of conic optimization.

3.1 Notations

Unless specified otherwise, the Euclidean norm of a vector x ∈ R n is denoted by ∥ x ∥ = √ x ⊤ x . The positive and negative part of x ∈ R are denoted by x + = max(0 , x ) and x -= max(0 , -x ) . The identity matrix of order n is denoted by I n , and e denotes the vector of all ones. The smallest eigenvalue of a real symmetric matrix X is λ min ( X ) .

Given a set X ⊆ R n , the interior and closure of X are denoted by int X and by cl X , respectively. The Euclidean projection onto convex set C is denoted by Π C , where

The set K⊆ R n is a cone if x ∈ K , λ ≥ 0 ⇒ λx ∈ K . The dual cone of K is

whose negative K ◦ = -K ∗ is the polar cone of K . A cone K is self-dual if K = K ∗ , and it is pointed if K∩ ( -K ) = { 0 } . All cones considered in the paper are proper cones, i.e., closed, convex, pointed cones with non-empty interior. A proper cone K defines conic inequalities ⪰ K and ≻ K as

3.2 Conic optimization

Consider a (convex) conic optimization problem of the form

where A ∈ R m × n , b ∈ R m , c ∈ R n , and K is a proper cone. All convex optimization problems can be formulated in conic form. A desirable property of conic formulations is that is enables the use of principled conic duality theory [23]. Namely, the conic dual problem reads

The dual problem (5) is a conic problem, and the dual of (5) is (4). Weak conic duality always holds, i.e., any dual-feasible solution provides a valid lower bound on the optimal value of (4), and vice-versa. When strong conic duality holds, e.g., under Slater's condition, both primal/dual problems have the same optimal value and a primal-dual optimal solution exists [23].

Conic optimization encompasses broad classes of problems such linear and semi-definite programming. Most real-life convex optimization problems can be represented in conic form using only a small number of cones [24], which are supported by off-the-shelf solvers such as Mosek, ECOS, or SCS. These so-called 'standard' cones comprise the non-negative orthant R + , the second-order cone Q and rotated second-order cone Q r , the positive semi-definite cone S + , the power cone P and the exponential cone E ; see Appendix B for algebraic definitions.

4 Dual Lagrangian Learning (DLL)

This section presents the Dual Lagrangian Learning (DLL) methodology, illustrated in Figure 1, for learning dual solutions for conic optimization problems. DLL combines the representation power of artificial neural networks (or, more generally, any differentiable program), with conic duality theory, thus providing valid Lagrangian dual bounds for general conic optimization problems. To the best of the authors' knowledge, this paper is the first to propose a principled self-supervised framework with dual guarantees for general conic optimization problems.

Dual Completion

Artificial Neural Network

Conic Projection

Lagrangian Dual Bound

Gradient step

DLL exploits three fundamental building blocks: (1) a dual conic completion procedure that provides dual-feasible solutions and, hence, valid Lagrangian dual bounds; (2) fast and differentiable conic projection layers; and (3) a self-supervised learning algorithm that emulates the steps of a dual Lagrangian ascent algorithm.

4.1 Dual Conic Completion

Consider a conic optimization problem in primal-dual form

where y ∈K ∗ and z ∈C ∗ are the dual variables associated to constraints (6b) and (6c), respectively. The proposed dual conic completion, outlined in Theorems 1 and 2 below, takes as input ˆ y ∈K ∗ , and recovers ˆ z ∈C ∗ such that (ˆ y, ˆ z ) is feasible for (7). The initial assumption that ˆ y ∈K ∗ can be enforced through a projection step, which will be described in Section 4.2.

Theorem 1 (Dual conic completion) . Assume that ∀ ˆ y ∈ K ∗ , ∃ x : Hx ≻ C h and the problem

is bounded. Then, ∀ ˆ y ∈ K ∗ , ∃ ˆ z ∈C ∗ : A ⊤ ˆ y + H ⊤ ˆ z = c , i.e., (ˆ y, ˆ z ) is feasible for (7) .

Theorem 2 (Optimal dual completion) . Let ˆ y ∈ K ∗ , and let ˆ z be dual-optimal for (8) .

Then, L (ˆ y, ˆ z ) = b ⊤ ˆ y + h ⊤ ˆ z is a valid dual bound on the optimal value of (6) , and L (ˆ y, ˆ z ) is the strongest dual bound that can be obtained after fixing y = ˆ y in (7) .

It is important to note the theoretical differences between the proposed dual completion, and applying a generic method, e.g., DC3 [5], LOOP-LC [14] or RAYEN [12], to the dual problem (7). First, LOOP-LC is not applicable here, because it only handles linear constraints and requires a compact feasible set, which is not the case in general for (7). Second, unlike RAYEN, Theorem 1 does not require an explicit characterization of the affine hull of the (dual) feasible set, nor does it assume knowledge of a strictly feasible point. In fact, Theorem 1 applies even if the feasible set of (7) has an empty interior. Third, the proposed dual completion enforces both linear equality constraints (7b) and conic constraints (7c). In contrast, the equality completion schemes used in DC3 and RAYEN enforce equality constraints but need an additional mechanism to handle inequality constraints. Fourth, the optimal completion outlined in Theorem 2 provides guarantees on the strength of the Lagrangian dual bound L (ˆ y, ˆ z ) . This is a major difference with DC3 and RAYEN, whose correction mechanism does not provide any guarantee of solution quality. Overall, the fundamental difference between generic methods and the proposed optimal dual completion, is that the former only exploit dual feasibility constraints (7b) -(7c) , whereas DLL also exploits (dual) optimality conditions, thus providing additional guarantees.

Another desirable property of the proposed dual completion procedure, is that it does not require the user to formulate the dual problem (7) explicitly, as would be the case for DC3 or RAYEN. Instead,

the user only needs to identify a set of primal constraints that satisfy the conditions of Theorem 1. For instance, it suffices to identify constraints that bound the set of primal-feasible solutions. This is advantageous because practitioners typically work with primal problems rather than their dual. The optimal dual completion can then be implemented via an implicit optimization layer. Thereby, in a forward pass, ˆ z is computed by solving the primal-dual pair (8)-(27) and, in a backward pass, gradient information is obtained via the implicit function theorem [25].

The main limitations of implicit layers are their numerical instability and their computational cost, both in the forward and backward passes. To eliminate these issues, closed-form analytical solutions are presented next for broad classes of conic optimization problems; other examples are presented in the numerical experiments of Section 5.

Example 1 (Bounded variables) . Consider a conic optimization problem with bounded variables

where l < u are finite lower and upper bounds on all variables x . The dual problem is

and the optimal dual completion is ˆ z l = | c -A ⊤ ˆ y | + , ˆ z u = | c -A ⊤ ˆ y | -.

The assumption in Example 1 that all variables have finite bounds holds in most -if not all- real-life settings, where decision variables are physical quantities (e.g. budgets or production levels) that are naturally bounded. The resulting completion procedure is a generalization of that used in [22] for linear programming (LP) problems.

Example 2 (Trust region) . Consider the trust region problem [26]

where r ≥ 0 , ∥·∥ is a norm, and ∥ x ∥≤ r ⇔ ( r, x ) ∈C = { ( t, x ) | t ≥∥ x ∥} . The dual problem is

where ∥·∥ ∗ is the dual norm and C ∗ = { ( t, x ) | t ≥∥ x ∥ ∗ } [27]. The optimal dual completion is ˆ z = c -A ⊤ ˆ y , ˆ z 0 = ∥ ˆ z ∥ ∗ .

Example 3 (Convex quadratic objective) . Consider the convex quadratic conic problem

where Q = F ⊤ F is positive definite. The problem can be formulated as the conic problem

whose dual is

The optimal dual completion is ˆ z = F -⊤ ( c -A ⊤ ˆ y ) , ˆ z 0 = 1 / 2 ∥ ˆ z ∥ 2 2 .

4.2 Conic Projections

The second building block of DLL are differentiable conic projection layers. Note that DLL only requires a valid projection onto K ∗ , which need not be the Euclidean projection Π K ∗ . Indeed, the latter may be computationally expensive and cumbersome to differentiate. For completeness, the paper presents Euclidean and non-Euclidean projection operators, where the latter are simple to implement, computationally fast, and differentiable almost everywhere. Closed-form formulae are presented for each standard cone in Appendix B, and an overview is presented in Table 1.

4.2.1 Euclidean projection

Let K be a proper cone, and ¯ x ∈ R n . By Moreau's decomposition [28],

which is a reformulation of the KKT conditions of the projection problem (1), i.e.,

It then follows that Π K ∗ (¯ x ) = -Π K ◦ ( -¯ x ) , by invariance of Moreau's decomposition under orthogonal transformations. Thus, it is sufficient to know how to project onto K to be able to project onto K ∗ and K ◦ . Furthermore, (16) shows that Π K is identically zero on the polar cone K ◦ . In a machine learning context, this may cause gradient vanishing issues and slow down training.

4.2.2 Radial projection

Given an interior ray ρ ≻ K 0 , the radial projection operator Π ρ K is defined as

The name stems from the fact that Π ρ K traces ray ρ from ¯ x until K is reached. Unlike the Euclidean projection, it requires an interior ray, which however only needs to be determined once per cone . The radial projection can then be computed, in general, via a line search on λ or via an implicit layer. Closed-form formulae for standard cones and their duals are presented in Appendix B.

4.3 Self-Supervised Dual Lagrangian Training

The third building block of DLL is a self-supervised learning framework for training dual conic optimization proxies. In all that follows, let ξ = ( A,b, H, h, c ) denote the data of an instance (6), and assume a distribution of instances ξ ∼ Ξ . Next, let M θ be a differentiable program parametrized by θ , e.g., an artificial neural network, which takes as input ξ and outputs a dual-feasible solution (ˆ y, ˆ z ) . Recall that dual feasibility of (ˆ y, ˆ z ) can be enforced by combining the dual conic projection presented in Section 4.2, and the optimal dual completion outlined in Theorem 2.

The proposed self-supervised dual lagrangian training is formulated as

where L (ˆ y, ˆ z, ξ ) = b ⊤ ˆ y + h ⊤ ˆ z is the Lagrangian dual bound obtained from (ˆ y, ˆ z ) by weak duality. Thereby, the training problem (19) seeks the value of θ that maximizes the expected Lagrangian dual bound over the distribution of instances Ξ , effectively mimicking the steps of a (sub)gradient algorithm. Note that, instead of updating (ˆ y, ˆ z ) directly, the training procedure computes a (sub)gradient ∂ θ L (ˆ y, ˆ z, ξ ) to update θ , and then obtains a new prediction (ˆ y, ˆ z ) through M θ . Also note that formulation (19) does not required labeled data, i.e., it does not require pre-computed dual-optimal solutions. Furthermore, it applies to any architecture that guarantees dual feasibility of (ˆ y, ˆ z ) , i.e., it does not assume any specific projection nor completion procedure.

5 Numerical experiments

This section presents numerical experiments on linear and nonlinear optimization problems; detailed problem formulations, model architectures, and other experiment settings, are reported in Appendix

All gaps are in %; best values are in bold. ∗ Mean optimal value on test set; obtained with Gurobi.

C. The code used for experiments is available under an open-source license. 1 The proposed DLL methodology is evaluated against applying DC3 to the dual problem (7) as a baseline. Thereby, linear equality constraints (7b) and conic inequality constraints (7c) are handled by DC3's equality completion and inequality correction mechanisms, respectively. The two approaches (DLL and DC3) are evaluated in terms of dual optimality gap and training/inference time. The dual optimality gap is defined as ( L ∗ - L (ˆ y, ˆ z )) / L ∗ , where L ∗ is the optimal value obtained from a state-of-the-art interior-point solver.

5.1 Linear Programming Problems

5.1.1 Problem formulation and dual completion

The first set of experiments considers continuous relaxations of multi-dimensional knapsack problems [29, 30], which are of the form

where p ∈ R n + , W ∈ R m × n + , and b ∈ R m + . The dual problem reads

where y ∈ R m and z l , z u ∈ R n . Since variables x is bounded, the closed-form completion presented in Example 1 applies. Namely, ˆ z l = |-p -W ⊤ ˆ y | + and ˆ z u = |-p -W ⊤ ˆ y | -, where ˆ y ∈ R m -.

5.1.2 Numerical results

Table 2 reports, for each combination of m,n : the average optimal value obtained by Gurobi (Opt val), as well as the average (avg), standard-deviation (std) and maximum (max) optimality gaps achieved by DC3 and DLL on the test set. First, DLL significantly outperforms DC3, with average gaps ranging from 0.07% to 1.93%, compared with 19.58%-274.78% for DC3, an improvement of about two orders of magnitude. A similar behavior is observed for maximum optimality gaps. The rest of the analysis thus focuses on DLL. Second, an interesting trend can be identified: optimality gaps tend to increase with m and decrease with n . This effect may be explained by the fact that increasing m increases the output dimension of the FCNN; larger output dimensions are typically harder to predict. In addition, a larger n likely provides a smoothing effect on the dual, whose solution becomes easier to predict. The reader is referred to [30] for probabilistic results on properties of multi-knapsack problems.

Next, Table 3 reports computing time statistics for Gurobi, DC3 and DLL. Namely, the table reports, for each combination of m,n , the time it takes to execute each method on all instances in the test set. First, DLL is 3-10x faster than DC3, which is caused by DC3's larger output dimension ( m + n , compared to m for DLL), and its correction steps. Furthermore, unsurprisingly, both DC3 and DLL

yield substantial speedups compared to Gurobi, of about 3 orders of magnitude. Note however that Gurobi's timings could be improved given additional CPU cores, although both ML-based methods remain significantly faster using a single GPU.

5.2 Nonlinear Production and Inventory Planning Problems

5.2.1 Problem formulation and dual completion

The second set of experiments considers the nonlinear resource-constrained production and inventory planning problem [31, 32]. In primal-dual form, the problem reads

where r, d, f ∈ R n are positive vectors, and b > 0 . Primal variables are x, t ∈ R n , and the dual variables associated to constraints (22b) and (22c) are y ∈ R -, and π, σ, τ ∈ R n , respectively.

Note that (22c) implies x, t ≥ 0 . Next, let y ≤ 0 be fixed, and consider the problem

Problem (24) is immediately strictly feasible, and bounded since ( d -yr ) , f > 0 and x, t ≥ 0 . Hence, Theorems 1 and 2 apply, and there exists a dual-optimal completion to recover π, σ, τ . A closed-form completion is then achieved as follows. First, constraints (23b) and (23c) yield π = d -ry and τ = f . Next, note that σ only appears in constraint (23e) and has negative objective coefficient. Further noting that (23e) can be written as σ 2 j ≤ 2 π j τ j , it follows that σ j = -√ 2 π j τ j at the optimum.

5.2.2 Numerical Results

Table 4 reports optimality gap statistics for DC3 and DLL. Similar to the linear programming setting, DLL substantially outperforms DC3, with average optimality gaps ranging from 0.23% to 1.03%, compared with 70.76%-87.01% for DC3. In addition, DLL exhibits smaller standard deviation and maximum optimality gaps than DC3. These results can be explained by several factors. First, the neural network architecture used in DC3 has output size n +1 , compared to 1 for DLL; this is because DLL leverages a more efficient dual completion procedure. Second, a closer examination of DC3's output reveals that it often fails to satisfy the (conic) inequality constraints (23d) and (23e). More generally, DC3 was found to have much slower convergence than DLL during training. While the performance of DC3 may benefit from more exhaustive hypertuning, doing so comes at a significant computational and environmental cost. This further highlights the benefits of DLL, which requires minimal tuning and is efficient to train.

† Time to solve all instances in the test set, using one CPU core. ‡ Time to run inference on all instances in the test set, using one V100 GPU.

Finally, Table 5 reports computing time statistics for Mosek, a state-of-the-art conic interior-point solver, DC3 and DLL. Abnormally high times are observed for Mosek and n =10 , 20 . These are most likely caused by congestion on the computing nodes used in the experiments, and are discarded in the analysis. Again, DC3 and DLL outperform Mosek by about three orders of magnitude. Furthermore, DLL is about 10x faster than DC3 for smaller instances ( n ≤ 100 ), and about 2x faster for the largest instances ( n =1000 ). This is caused by DC3's larger output dimension and correction steps.

6 Discussion

6.1 Mixed-Integer Nonlinear Programming Setting

The proposed Dual Lagrangian Learning framework directly extends to the mixed-integer nonlinear programming (MINLP) setting. Consider a general MINLP problem of the form

where X ⊆ R n denotes a possibly discrete domain. Given Lagrange multipliers λ ∈ R m and µ ∈ R p + associated to equality and inequality constraints, the corresponding Lagrangian dual bound is

Note that (26) is the MINLP counterpart of (8) in the conic setting.

The dual Lagrangian function L ( λ, µ ) is concave, non-smooth, and admits sub-differentials of the form ∂ λ L = -h (¯ x ) , ∂ µ L = -g (¯ x ) , where ¯ x is an optimal solution of (26). The self-supervised learning framework of Section 4.3 can then be applied out of the box, wherein an ML model predicting ( λ, µ ) is trained in a self-supervised fashion by maximizing the dual bound L ( λ, µ ) . This approach is followed in, e.g., [20] for mixed-integer linear problems, and [6, 8] for nonlinear problems.

Despite natural similarities between the (convex) conic and MINLP settings, several intrinsic limitations appear in the latter. First, although the domain of ( λ, µ ) ∈ R m × R p + is simple, and can be enforced via, e.g., ReLU activations, evaluating L ( λ, µ ) is not. Indeed, this requires solving the MINLP problem (26) to optimality , which is NP-hard in general. In contrast, the proposed dual conic completion can be performed efficiently, and closed-form solutions are available for broad classes of problems. Second, (sub)gradient information ∂ L is obtained from an optimal solution of (26), which poses obvious limitations if (26) is solved approximately. Third, arbitrary values of λ, µ may result in

(26) being unbounded, yielding a dual bound of -∞ and no usable gradient information. In contrast, in the conic setting, Theorem 1 provides sufficient conditions under which dual completion is always possible. Finally, an intrinsic limitation in the MINLP setting is the absence, in general, of strong duality. Therefore, even predicting a dual-optimal ( λ, µ ) may be insufficient to prove optimality, thus requiring additional computation such as branching. In contrast, the strong conic duality theorem [23] offers a robust foundation to obtain high-quality dual bounds efficiently.

6.2 Limitations

The main theoretical limitation of the paper is that it considers convex conic optimization problems, and therefore does not consider discrete decisions nor general non-convex constraints. Since convex relaxations are typically used to solve non-convex problems to global optimality, the proposed approach is nonetheless still useful in non-convex settings. Furthermore, as pointed out in Section 6.1, the DLL framework extends naturally to the MINLP setting, by leveraging Lagrangian duality for discrete and/or nonlinear problems. However, this approach suffers from several theoretical and computational limitations.

On the practical side, the optimal dual completion presented in Section 4.1 requires, in general, the use of an implicit layer, which is typically not tractable for large-scale problems. In the absence of a known closed-form optimal dual completion, it may still be possible to design efficient completion strategies that at least ensure dual feasibility. One such strategy is to introduce artificial large bounds on all primal variables, and use the completion outlined in Example 1. Finally, all neural network architectures considered in the experiments are fully-connected neural networks. Thus, a separate model is trained for each input dimension. Nevertheless, the DLL methodology is applicable to graph neural network architectures, which would support arbitrary problem size. The use of GNN models in the DLL context is a promising avenue for future research.

7 Conclusion

The paper has proposed Dual Lagrangian Learning (DLL), a principled methodology for learning dual conic optimization proxies. Thereby, a systematic dual conic completion, differentiable conic projection layers, and a self-supervised dual Lagrangian training framework have been proposed. The effectiveness of DLL has been demonstrated on numerical experiments that consider linear and nonlinear conic problems, where DLL significantly outperforms DC3 [5], and achieves 1000x speedups over commercial interior-point solvers.

One of the main advantages of DLL is its simplicity. The proposed dual completion can be stated only in terms of primal constraints, thus relieving users from the need to explicitly write the dual problem. DLL introduces very few hyper-parameters, and requires minimal tuning to achieve good performance. This results in simpler models and improved performance, thus delivering computational and environmental benefits.

DLLopens the door to multiple avenues for future research, at the intersection of ML and optimization. The availability of high-quality dual-feasible solutions naturally calls for the integration of DLL in existing optimization algorithms, either as a warm-start, or to obtain good dual bounds fast. Multiple optimization algorithms have been proposed to optimize Lagrangian functions, which may yield more efficient training algorithms in DLL. Finally, given the importance of conic optimization in numerous real-life applications, DLL can provide a useful complement to existing primal proxies.

Acknowledgments

This research was partially supported by NSF awards 2007164 and 2112533, and ARPA-E PERFORM award DE-AR0001280.

References

Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: A methodological tour d'horizon. European Journal of Operational Research , 290(2):405-421, 2021. ISSN 0377-2217. doi: https://doi.org/10.1016/j.ejor.2020.07.063.

James Kotary, Ferdinando Fioretto, Pascal Van Hentenryck, and Bryan Wilder. End-to-end constrained optimization learning: A survey. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 4475-4482. International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 10.24963/ ijcai.2021/610. URL https://doi.org/10.24963/ijcai.2021/610 . Survey Track.

Ferdinando Fioretto, Terrence WK Mak, and Pascal Van Hentenryck. Predicting AC Optimal Power Flows: Combining deep learning and lagrangian dual methods. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 630-637, 2020. URL https: //doi.org/10.1609/aaai.v34i01.5403 .

Xiang Pan, Tianyu Zhao, Minghua Chen, and Shengyu Zhang. DeepOPF: A deep neural network approach for security-constrained DC optimal power flow. IEEE Transactions on Power Systems , 36(3):1725-1735, 2020.

Priya L. Donti, David Rolnick, and J. Zico Kolter. DC3: A learning method for optimization with hard constraints. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , 2021. URL https://iclr.cc/virtual/2021/ poster/2868 .

Seonho Park and Pascal Van Hentenryck. Self-supervised primal-dual learning for constrained optimization. Proceedings of the AAAI Conference on Artificial Intelligence , 37(4):4052-4060, 2023. doi: 10.1609/aaai.v37i4.25520. URL https://ojs.aaai.org/index.php/AAAI/ article/view/25520 .

Chendi Qian, Didier Ch´ etelat, and Christopher Morris. Exploring the power of graph neural networks in solving linear optimization problems. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , volume 238 of Proceedings of Machine Learning Research , pages 1432-1440. PMLR, 02-04 May 2024. URL https://proceedings.mlr.press/v238/qian24a.html .

James Kotary and Ferdinando Fioretto. Learning constrained optimization with deep augmented lagrangian methods, 2024.

Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id=Bk9mxlSFx .

Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial optimization algorithms over graphs. Advances in neural information processing systems , 30, 2017.

Wenbo Chen, Mathieu Tanneau, and Pascal Van Hentenryck. End-to-End Feasible Optimization Proxies for Large-Scale Economic Dispatch. IEEE Transactions on Power Systems , pages 1-12, 2023. doi: 10.1109/TPWRS.2023.3317352.

Jesus Tordesillas, Jonathan P How, and Marco Hutter. Rayen: Imposition of hard convex constraints on neural networks, 2023.

Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J. Zico Kolter. Differentiable convex optimization layers. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´ e-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/ paper_files/paper/2019/file/9ce3c52fc54362e22053399d3181c638-Paper.pdf .

Meiyi Li, Soheil Kolouri, and Javad Mohammadi. Learning to Solve Optimization Problems With Hard Linear Constraints. IEEE Access , 11:59995-60004, 2023. doi: 10.1109/ACCESS. 2023.3285199.

Andrei V Konstantinov and Lev V Utkin. A new computationally simple approach for implementing neural networks with output hard constraints. In Doklady Mathematics , pages 1-9. Springer, 2024.

Terrence W.K. Mak, Minas Chatzos, Mathieu Tanneau, and Pascal Van Hentenryck. Learning regionally decentralized ac optimal power flows with admm. IEEE Transactions on Smart Grid , 14(6):4863-4876, 2023.

Sebastian Kraul, Markus Seizinger, and Jens O. Brunner. Machine learning-supported prediction of dual variables for the cutting stock problem with an application in stabilized column generation. INFORMS Journal on Computing , 35(3):692-709, 2023. doi: 10.1287/ijoc.2023.1277.

Nagisa Sugishita, Andreas Grothey, and Ken McKinnon. Use of machine learning models to warmstart column generation for unit commitment. INFORMS Journal on Computing , 2024. doi: 10.1287/ijoc.2022.0140.

Augustin Parjadis, Quentin Cappart, Bistra Dilkina, Aaron Ferber, and Louis-Martin Rousseau. Learning Lagrangian Multipliers for the Travelling Salesman Problem, 2023.

Francesco Demelas, Joseph Le Roux, Mathieu Lacroix, and Axel Parmentier. Predicting lagrangian multipliers for mixed integer linear programs. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 10368-10384. PMLR, 21-27 Jul 2024. URL https://proceedings.mlr.press/v235/demelas24a.html .

Guancheng Qiu, Mathieu Tanneau, and Pascal Van Hentenryck. Dual Conic Proxies for AC Optimal Power Flow. In Power Systems Computations Conference , 2024.

Michael Klamkin, Mathieu Tanneau, and Pascal Van Hentenryck. Dual interior-point optimization learning, 2024.

Aharon Ben-Tal and Arkadi Nemirovski. Lectures on modern convex optimization: analysis, algorithms, and engineering applications . SIAM, 2001.

M. Lubin, E. Yamangil, R. Bent, and J. P. Vielma. Extended Formulations in Mixed-Integer Convex Programming. In Q. Louveaux and M. Skutella, editors, Proceedings of the 18th Conference on Integer Programming and Combinatorial Optimization (IPCO 2016) , volume 9682 of Lecture Notes in Computer Science , pages 102-113, 2016.

Akshay Agrawal, Shane Barratt, Stephen Boyd, Enzo Busseti, and Walaa M. Moursi. Differentiating through a Cone Program. Journal of Applied and Numerical Optimization , 1(2):107-115, August 2019.

Jorge Nocedal and Stephen J Wright. Numerical optimization . Springer, 1999.

Stephen P Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and trends® in Optimization , 1(3):127-239, 2014.

Arnaud Freville and G´ erard Plateau. An efficient preprocessing procedure for the multidimensional 0-1 knapsack problem. Discrete Applied Mathematics , 49(1):189-212, 1994. ISSN 0166-218X. doi: 10.1016/0166-218X(94)90209-7. Special Volume Viewpoints on Optimization.

Arnaud Freville. The multidimensional 0-1 knapsack problem: An overview. European Journal of Operational Research , 155(1):1-21, 2004. ISSN 0377-2217. doi: 10.1016/S0377-2217(03) 00274-1.

Hans Ziegler. Solving certain singly constrained convex optimization problems in production planning. Operations Research Letters , 1(6):246-252, 1982. ISSN 0167-6377. doi: 10.1016/ 0167-6377(82)90030-X.

MOSEK Aps. The MOSEK Modeling Cookbook , 2023. URL https://docs.mosek.com/ modeling-cookbook/index.html .

A Proofs for Section 4 (Dual Lagrangian Learning (DLL))

Theorem 1 (Dual conic completion) . Assume that ∀ ˆ y ∈ K ∗ , ∃ x : Hx ≻ C h and the problem

is bounded. Then, ∀ ˆ y ∈ K ∗ , ∃ ˆ z ∈C ∗ : A ⊤ ˆ y + H ⊤ ˆ z = c , i.e., (ˆ y, ˆ z ) is feasible for (7) .

Proof. Let ˆ y ∈ K ∗ , and recall that (8) is bounded and strictly feasible. By strong duality, its dual

is solvable [BTN01]. Therefore, there exists a feasible solution ˆ z for (27). By construction, ˆ z ∈ C ∗ and A ⊤ ˆ y + H ⊤ ˆ z = c , hence (ˆ y, ˆ z ) is feasible for (7).

∗

Theorem 2 (Optimal dual completion) . Let ˆ y ∈ K , and let ˆ z be dual-optimal for (8) . Then, L (ˆ y, ˆ z ) = b ⊤ ˆ y + h ⊤ ˆ z is a valid dual bound on the optimal value of (6) , and L (ˆ y, ˆ z ) is the strongest dual bound that can be obtained after fixing y = ˆ y in (7) .

Proof. First, recall that ˆ z exists by strong conic duality; see proof of Theorem 1. Furthermore, (ˆ y, ˆ z ) is feasible for (7) by construction. Thus, by weak duality, the Lagrangian bound L (ˆ y ) = b ⊤ ˆ y + h ⊤ ˆ z is a valid dual bound on the optimal value of (6). Finally, fixing y = ˆ y in (7) yields

which is equivalent to (27). Hence, its optimal value is b ⊤ ˆ y + h ⊤ ˆ z = L (ˆ y, ˆ z ) by definition of ˆ z .

Example 1 (Bounded variables) . Consider a conic optimization problem with bounded variables

where l < u are finite lower and upper bounds on all variables x . The dual problem is

and the optimal dual completion is ˆ z l = | c -A ⊤ ˆ y | + , ˆ z u = | c -A ⊤ ˆ y | -.

Proof. Let ˆ y ∈ K ∗ be fixed. Fixing y = ˆ y in the dual problem yields

Eliminating z l = z u +( c -A ⊤ ˆ y ) , the problem becomes

z

u

0

.

(34)

Since l < u , i.e., the objective coefficient of z u is negative, and the problem is a maximization problem, it follows that z u must be as small as possible in any optimal solution. Hence, at the optimum, ˆ z u = max(0 , -( c -A ⊤ y )) = | c -A ⊤ ˆ y | -, and ˆ z l = | c -A ⊤ ˆ y | + .

Example 2 (Trust region) . Consider the trust region problem [NW99]

where r ≥ 0 , ∥·∥ is a norm, and ∥ x ∥≤ r ⇔ ( r, x ) ∈C = { ( t, x ) | t ≥∥ x ∥} . The dual problem is

where ∥·∥ ∗ is the dual norm and C ∗ = { ( t, x ) | t ≥∥ x ∥ ∗ } [BV04]. The optimal dual completion is ˆ z = c -A ⊤ ˆ y , ˆ z 0 = ∥ ˆ z ∥ ∗ .

Proof. The relation ˆ z = c -A ⊤ ˆ y is immediate from the dual equality constraint A ⊤ y + z = c . Next, observe that z 0 appears only in the constraint ( z 0 , z ) ∈ C ∗ , and has negative objective coeffient. Hence, z 0 must be as small as possible in any optimal solution. This yields ˆ z 0 = ∥ ˆ z ∥ ∗ .

Example 3 (Convex quadratic objective) . Consider the convex quadratic conic problem

where Q = F ⊤ F is positive definite. The problem can be formulated as the conic problem

whose dual is

The optimal dual completion is ˆ z = F -⊤ ( c -A ⊤ ˆ y ) , ˆ z 0 = 1 / 2 ∥ ˆ z ∥ 2 2 .

Proof. The proof uses the same argument as the proof of Example 2. Namely, ˆ z = F -⊤ ( c -A ⊤ ˆ y ) is immediate from the dual equality constraint A ⊤ y + F ⊤ z = c . Note that F ⊤ is non-singular because Q is positive definite. Finally, z 0 has negative objective coefficient, and only appears in the conic constraint (1 , z 0 , z ) ∈ Q 2+ n r . Therefore, at the optimum, one must have 2ˆ z 0 = ∥ ˆ z ∥ 2 2 , which concludes the proof.

B Standard cones

This section presents standard cones and their duals, as well as corresponding Euclidean and radial projections. The reader is referred to [CKV22] for a more exhaustive list of non-standard cones, and to [PB + 14, Sec. 6.3] for an overview of Euclidean projections onto standard cones.

B.1 Non-negative orthant

The non-negative orthant is defined as R n + = { x ∈ R n : x ≥ 0 } . It is a self-dual cone, and forms the basis of linear programming [BTN01].

Euclidean projection The Euclidean projection on R n + is

where the max and ReLU operations are performed element-wise.

Radial projection The radial projection with ray e , applied coordinate-wise, is equivalent to the Euclidean projection.

B.2 Conic quadratic cones

Conic quadratic cones include the second-order cone (SOC)

and the rotated second-order cone (RSOC)

Both cones are self-dual, i.e., Q ∗ = Q and Q ∗ r = Q r . The RSOC is the main building block of conic formulations of convex quadratically-constrained optimization problems.

Euclidean projection The Euclidean projection on Q n is given by

where δ = ∥ (¯ x 2 , ..., ¯ x n ) ∥ 2 .

Radial projection Given interior ray ρ = (1 , 0 , ..., 0) ≻ Q n 0 , the radial projection is

Note that, in the worst case, computing Π Q (¯ x ) requires O (2 n ) operations, and modifies all coordinates of ¯ x . In contrast, computing Π ρ Q (¯ x ) requires only O ( n ) operations, and only modifies the first coordinate of ¯ x .

Closed-form formulae for Euclidean and radial projections onto Q n r are derived from (38) and (39).

B.3 Positive Semi-Definite cone

The cone of positive semi-definite (PSD) matrices of order n is defined as

Note that all matrices in S n + are symmetric, hence all their eigenvalues are real. The PSD cone is self-dual, and generalizes the non-negative orthant and SOC cones [BV04].

Euclidean projection The Euclidean projection onto S n + is given by

where ¯ X ∈ R n × n is symmetric with eigenvalue decomposition ¯ X = ∑ i λ i v i v ⊤ i . Note that the Euclidean projection onto the PSD code thus requires a full eigenvalue decomposition, which has complexity O ( n 3 ) .

Radial projection The radial projection considered in the paper uses ρ = I n ∈ int( S n + ) . This yields the closed-form projection

Note that the radial projection only requires computing the smallest eigenvalue of ¯ X , which is typically much faster than a full eigenvalue decomposition, and only modifies the diagonal of ¯ X .

B.4 Exponential Cone

The 3-dimensional exponential cone is a non-symmetric cone defined as

The exponential cone is useful to model exponential and logarithmic terms, which occur in, e.g., relative entropy, logistic regression, or logarithmic utility functions.

Euclidean projection To the best of the authors' knowledge, there is no closed-form, analytical formula for evaluating Π E nor Π E ∗ , which instead require a numerical method, see, e.g., [PB + 14] and [Fri23] for completeness.

Radial projection To avoid any root-finding operation, the paper leverages the fact that x 1 , x 2 > 0 , ∀ ( x 1 , x 2 , x 3 ) ∈ E . Note that one can enforce ¯ x 1 , ¯ x 2 > 0 via, e.g., softplus activation. A radial projection is then obtained using ρ = (0 , 0 , 1) , which yields

This approach does not require any root-finding, and is therefore more amenable to automatic differentiation. The validity of Eq. (45) is immediate from the representation

Similarly, assuming ¯ y 1 > 0 and ¯ y 3 < 0 , the radial projection onto E ∗ reads

whose dual cone is

B.5 Power Cone

Given 0 <α< 1 , the 3-dimensional power cone is defined as

Power cones are non-symmetric cones, which allow to express power other than 2 , e.g., p -norms with p ≥ 1 . Note that P 1 / 2 is a scaled version of the rotated second-order cone Q 3 r . The 3-dimensional power cone P α is sufficient to express more general, high-dimensional power cones. The dual power cone is

Euclidean projection The Euclidean projection onto the power cone P α is described in [Hie15]. Similar to the exponential cone, it requires a root-finding operation.

Radial projection The proposed radial projection is similar to the one proposed for E . Assuming ¯ x 2 , ¯ x 3 > 0 , and using ρ = (1 , 0 , 0) , the radial projection reads

A similar approach is done to recover y ∈ P ∗ α after scaling the first two coordinates of y . This technique can be extended to the more general power cones.

C Experiment Details

C.1 Common experiment settings

All neural network architectures considered here are fully-connected neural networks (FCNNs). Thus, a separate model is trained for each input dimension. Note that the proposed DLL methodology is applicable to graph neural network architectures, which would support arbitrary problem size. The use GNN models in the DLL context is a promising avenue for future research; a systematic comparison of the performance of GNN and FCNN architectures is, however, beyond the scope of this work.

All ML models are trained in a self-supervised fashion following the training scheme outlined in Section 4.3, and training is performed using the Adam optimizer [KB15]. The training scheme uses a patience mechanism where the learning rate η is decreased by a factor 2 if the validation loss does not improve for more than N p epochs. The initial learning rate is η =10 -4 . Training is stopped if either the learning rate reaches η min =10 -7 , or a maximum N max epochs is reached. Every ML model considered in the experiments was trained in under an hour.

A limited, manual, hypertuning was performed by the authors during preliminary experiments. It was found that DLL models require very little hypertuning, if any, to achieve satisfactory performance. In contrast, DC3 was found to require very careful hypertuning, even just to ensure its numerical stability. It is also important to note that DC3 introduces multiple additional hyperparameters, such as the number of correction steps, learning rate for the correction steps, penalty coefficient for the soft penalty loss, etc. These additional hyperparameters complicate the hypertuning task, and result in additional computational needs. Given the corresponding economical and environmental costs, only limited hypertuning of DC3 was performed.

Finally, it was observed that DC3 often fail to output dual-feasible solutions, which therefore do not valid dual bounds. Therefore, to ensure a fair comparison, the dual solution produced by DC3 is fed to the dual optimal completion of DLL, thus ensuring dual feasibility and a valid dual bound. This is only performed at test time, with a negligible overhead since the dual completion uses a closed-form formula. All optimality gaps for DC3 are reported for this valid dual bound.

C.2 Linear programming problems

C.2.1 Problem formulation

The first set of experiments considers the continuous relaxation of multi-dimensional knapsack problems [FP94, Fre04], which are of the form

where n denotes the number of items, m denotes the number of resources, p ∈ R n + is the value of each item, b i is the amount of resource i , and W ij denotes the amount of resource i used by item j . The dual problem reads

C.2.2 Data generation

For each number of items n ∈{ 100 , 200 , 500 } and number of resources m ∈{ 5 , 10 , 30 } , a total of 16384 instances are generated using the same procedure as the MIPLearn library [SXQG + 23]. Each instance is solved with Gurobi, and the optimal dual solution is recorded for evaluation purposes. This dataset is split in training, validation and testing sets, which contain 8192, 4096 and 4096 instances, respectively.

C.2.3 DLL implementation

The DLL architecture considered here is a fully-connected neural network (FCNN); a separate model is trained for each combination ( m,n ) . The FCNN model takes as input the flatted problem data ( b, p, W ) ∈ R 1+ n + n × m , and outputs y ∈ R m . The FCNN has two hidden layers of size 2( m + n ) and sigmoid activation; the output layer uses a negated softplus activation to ensure y ≤ 0 . The dual completion procedure follows Example (1).

Hyperparameters The patience parameter is N p =32 , and the maximum number of training epochs is N max =1024 .

C.2.4 DC3 implementation

The DC3 architecture consists of an initial FCNN which takes as input ( b, p, W ) , and outputs y, z l . Then, z u is recovered by equality completion as z u = p + W ⊤ y + z l . The correction then applies gradient steps ( y, z l ) ← ( y, z l ) -γ ∇ ϕ ( y, z l ) where

The corresponding gradients ∇ ϕ ( y, z l ) were computed analytically. After applying corrections, the dual equality completion is applied one more time to recover z u , and the final soft loss is

which considers both the dual objective value, and the violation of inequality constraints. Note that the dual objective b ⊤ y -e ⊤ z u is not a valid dual bound in general, because y, z l , z u may not be dual-feasible.

Hyperparameters The maximum number of correction steps is 10 , the learning rate for correction is γ =10 -4 . The soft penalty weight is set to ρ = 10 ; this parameter was found to have a high impact on the numerical stability of training. The patience parameter is N p =32 , and the maximum number of training epochs is N max =1024 .

C.3 Nonlinear Production and Inventory Planning Problems

C.3.1 Problem formulation

The original presentation of the resource-constrained production and inventory planning problem [Zie82] uses the nonlinear convex formulation

where n is the number of items to be produced, x ∈ R n , b ∈ R denotes the available resource amount, and r j > 0 denotes the resource consumption rate of item j . The objective function captures production and inventory costs. Namely, d j = 1 2 c p j c r j and f j = c o j D j , where c p j , c r j , c o j > 0 and D j > 0 denote per-unit holding cost, rate of holding cost, ordering cost, and total demand for item j , respectively.

The problem is reformulated in conic form [MOS23a] as

Note that the dual problem contains 1 + 3 n variables, 2 n equality constraints, 1 linear inequality constraints, and n conic inequality constraints. Therefore, DC3 must predict n +1 dual variables, then recover 2 n variables by equality completion, and correct for n +1 inequality constraints. In contrast, by exploiting dual optimality conditions, DLL eliminates 3 n dual variables, thus reducing the output dimension of the initial prediction from n +1 to 1 , and eliminates the need for correction.

C.3.2 Data generation

For each n ∈ { 10 , 20 , 50 , 100 , 200 , 500 , 1000 } , 16384 instances are generated using the procedure of [Zie82]. First, D j is sampled from a uniform distribution U [1 , 100] , c p j is sampled from U [1 , 10] , and c r j is sampled from U [0 . 05 , 0 . 2] . Then, c o j = α j c p j and r j = β j c p j , where α, β are sampled from U [0 . 1 , 1 . 5] and U [0 . 1 , 2] , respectively. Finally, the right-hand side is b = η ∑ j r j where η is sampled from U [0 . 25 , 0 . 75] .

Each instance is solved with Mosek, and its solution is recorded for evaluation purposes. The dataset is split into training, validation and testing sets comprising 8192, 4096 and 4096 instances, respectively.

C.3.3 DLL implementation

The DLL architecture consists of an initial FCNN that takes as input ( d, f, r, b ) ∈ R 1+3 n , and output y ∈ R . The FCNNs have two hidden layers of size max(128 , 4 n ) and sigmoid activation. For the output layer, a negated softplus activation ensures y ≤ 0 . The dual completion outlined in Section 5.2 then recovers ( π, σ, τ ) .

whose dual problem is

Hyperparameters The patience parameter is N p =128 , and the maximum number of training epochs is N max =4096 . The patience mechanism is deactivated for the first 1024 epochs; this latter setting has little impact of the performance of DLL, and was introduced to avoid premature termination for DC3.

C.3.4 DC3 implementation

The DLL architecture consists of an initial FCNN that takes as input ( d, f, r, b ) ∈ R 1+3 n , and outputs y, σ ∈ R . The FCNNs have two hidden layers of size max(128 , 4 n ) and sigmoid activation, and the output layer has linear activation.

The equality completion step recovers π = d -ry and τ = f . The correction step then apply gradient steps to minimize the violations ϕ ( y, σ ) = ϕ y ( y, σ ) + ϕ π ( y, σ ) + ϕ σ ( y, σ ) , where

Note that, to express ϕ σ ( y, σ ) , conic constraints (23e) are converted to their nonlinear programming equality, because DC3 does not handle conic constraints. Gradients for ϕ are computed analytically, and implemented directly in the inequality correction procedure. The final soft loss is then

Hyperparameters The maximum number of correction steps is 10 , the learning rate for correction is γ =10 -5 , and the soft loss penalty parameter is ρ = 10 . The patience parameter is N p =128 , and the maximum number of training epochs is N max =4096 . The patience mechanism is deactivated for the first 1024 epochs; this latter setting has little impact of the performance of DLL, and was introduced to avoid premature termination for DC3.

Overall, DC3 was found to experience substantial numerical instability, and failed to produce dualfeasible solutions on a majority of instances. Increasing the number of correction steps helps alleviate this issue, at the cost of more expensive inference and back-propagation. Increasing the learning rate for correction ( γ ) was also found to yield smaller violations, yet resulted in degraded numerical stability. Finally, increasing the number of correction steps also increases GPU memory requirements, which can further affect training performance.

C.3.5 Convergence plots

Figures 2 and 3 show the progress of the Lagrangian dual bound obtained by DLL and DC3 throughout training. The figures report the average Lagrangian dual bound on the training and validation set, as a function of the number of epochs (Figure 2) and training time (Figure 3). The difference in training times, for a same number of epochs, is explained by the longer inference and back-propagation times for DC3 (see also Table 5).

Both figures show that DLL exhibits a faster convergence, with performance plateau-ing after about 1,000 training epochs. The performance of DC3 degrades as the instances become larger (from n = 10 to n = 500 ): the plots exhibit a more erratic behavior, especially in the first 500 training epochs. On the smallest instances (n= 10 and n= 20 ), the behavior stabilizes after about 500 epochs, yet progress remains slow compared to DLL.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract mentions the 3 building blocks of the proposed methodology, which are described in Section 4. The numerical results that are mentioned in the abstract reflect the results presented in Section 5.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Theoretical and practical limitations are discussed in Section 6.2.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions are stated in the paper and in theorem, and proofs are provided in Appendix.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experiment details are provided in Appendix C. These include:

Computing resources used for experiments (CPU and GPU models)

Problem formulations and data-generation procedures

Neural architecture used in the experiments

Detailed training scheme

Hyper-parameters used for the final results

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All the data used in the experiments is publicly available and/or synthetically generated. We have cited sources whenever using a data-generation procedure proposed elsewhere. We intend to release our code upon acceptance of the paper.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental details are provided in Appendix C

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: When reporting optimality gaps in Section 5 (Tables 2 and 4), we report averages, standard deviations, and maximum across the test set. Computing times reported in Tables 3 and 5 were evaluated over multiple runs.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Compute resources and training time are reported in Appendix C.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the code of ethics, and do not see any deviation to report.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Broader impact is discussed in Section 6 (Limitations) and Section 7 (Conclusion).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Not applicable.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All prior codes / methods have been cited.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

References for the Appendix

[BTN01] Aharon Ben-Tal and Arkadi Nemirovski. Lectures on modern convex optimization: analysis, algorithms, and engineering applications . SIAM, 2001.

[BV04] Stephen P Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

[CKV22] Chris Coey, Lea Kapelevich, and Juan Pablo Vielma. Solving natural conic formulations with hypatia.jl. INFORMS Journal on Computing , 34(5):2686-2699, 2022.

[FP94] Arnaud Freville and G´ erard Plateau. An efficient preprocessing procedure for the multidimensional 0-1 knapsack problem. Discrete Applied Mathematics , 49(1):189-212, 1994. Special Volume Viewpoints on Optimization.

[Fre04] Arnaud Freville. The multidimensional 0-1 knapsack problem: An overview. European Journal of Operational Research , 155(1):1-21, 2004.

[Fri23] Henrik A. Friberg. Projection onto the exponential cone: a univariate root-finding problem. Optimization Methods and Software , 38(3):457-473, 2023.

[GO18] LLC Gurobi Optimization. Gurobi optimizer reference manual, 2018.

[Hie15] Le Thi Khanh Hien. Differential properties of euclidean projection onto power cone. Mathematical Methods of Operations Research , 82:265-284, 2015.

[Inn18] Michael Innes. Don't unroll adjoint: Differentiating ssa-form programs. CoRR , abs/1810.07951, 2018.

[ISF + 18] Michael Innes, Elliot Saba, Keno Fischer, Dhairya Gandhi, Marco Concetto Rudilosso, Neethu Mariya Joy, Tejan Karmali, Avik Pal, and Viral Shah. Fashionable modelling with flux. CoRR , abs/1811.01457, 2018.

[KB15] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

[MOS23a] MOSEK Aps. The MOSEK Modeling Cookbook , 2023.

[MOS23b] MOSEK Aps. The MOSEK optimization toolbox for Julia manual. Version 10.1.23. , 2023.

[NW99] Jorge Nocedal and Stephen J Wright. Numerical optimization . Springer, 1999.

[PAC17] PACE. Partnership for an Advanced Computing Environment (PACE) , 2017.

[PB + 14] Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and trends® in Optimization , 1(3):127-239, 2014.

[SXQG + 23] Alinson Santos Xavier, Feng Qiu, Xiaoyi Gu, Berkay Becu, and Santanu S. Dey. MIPLearn: An Extensible Framework for Learning- Enhanced Optimization, June 2023.

[Zie82] Hans Ziegler. Solving certain singly constrained convex optimization problems in production planning. Operations Research Letters , 1(6):246-252, 1982.

Peer reviews:
comments: SUMMARY:
The authors propose Dual Langrangian Learning for learning dual conic optimization problems. Dual conic completion, differentiable conic projection layers, and a self supervised Lagrangian training framework are discussed.

STRENGTHS:
- Presents a unique framework for the dual optimization. No dedicated framework for dual optimization proxies are known to date
-Extensive theoretical formulations

WEAKNESSES:
- The practical use of the framework is unclear especially with regard to scalability and computational costs involved

QUESTIONS:
- It is unclear how the utility of this framework can be tested for problem-solving in the real world. While DC3 has been selected as a comparable technique, it remains to be seen whether dual optimization is always beneficial -- for e.g. assuming a problem can be solved both in the primal and dual, are there necessary computational benefits to solving in the dual? How scalable are the proposed algorithmic frameworks?

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the broad category of machine learning (ML) for optimization, where ML-based approaches are used to obtain solutions or bounds on the solutions of optimization problems. In particular, the paper addresses a particular subclass of problem - conic problems, and even more specifically, the dual solution of these problems.\n\nMost \"ML-for-optimization\" frameworks have thus far focused on finding the solution to primal problems, which is intuitive because the variables in primal problems are typically the variables of interest for most real-world problems. However, learning dual solutions can help certify or bound the primal solution, and in some cases, one may be interested in the dual solution directly (for example, when learning shadow prices in economic optimization problems). This paper uses a conic projection layer in addition to the dual completion step to ensure dual feasibility.\n\nThe paper compares the method (DLL) to other existing methods in terms of their ability to learn solutions to dual problems and shows superiority, mostly due to the fact that DLL includes dual optimality conditions in addition to dual feasibility constraints. An alternative to Euclidean projection is also provided (Radial projection), for the projection layer, which allows for computation once per cone and attempts to avoid gradient vanishing issues during training.\n\nSTRENGTHS:\nThe paper claims that the model can guarantee dual feasibility, which is significant for yielding bounds on the primal solution. In addition, the authors show a closed-form, analytical solution for conic projections and for dual conic completion, outside of the ML model development.\n\nThe proposed DLL framework doesn't require an explicit formulation of the dual problem; instead, the user must only have to provide a set of primal constraints that satisfy a certain set of conditions. This is convenient but not a main contribution (if the primal conic problem is known, the dual should be relatively easy to find).\n\nEven though the most challenging and time consuming optimization problems in practice are nonconvex optimization problems, many of these problems can be cast as conic problems, and thus the method can be applied to solve quite a large array of problems that may occur in practice.\n\nVersus other methods, the DLL framework requires far less hyperparameter tuning and iterative procedures, which contributes to the ease of training and increased computational speed.\n\nWEAKNESSES:\nOne limitation of the method, if a known closed-form optimal dual completion cannot be achieved, is its use of implicit layers. It would be interesting to see how the speedups diminish with very large problems where the implicit layer becomes a bottleneck. \n\nThe comparison with existing approaches is fair since those are the state of the art for learning primal solutions, but it's slightly unfair because those methods were not specifically designed to learn dual solutions. I think this is fine since there are not existing ML-for-optimization methods tailored towards dual solutions in particular, but just to note.\n\nThe work in this paper does seem like a more generalized and extended version of the work in reference [19], where reference [19] was the first to provide a dual proxy for a second-order cone relaxation of the AC optimal power flow (OPF) problem. However, in reading [19], it does not seem to be necessarily limited to the AC OPF problem, although those authors only apply it to such. The approach in the present paper is generalized to a broader class of problems, but is not the first to provide dual feasibility guarantees as determined by a learning model.\n\nQUESTIONS:\nIt would be interesting to also see indications of whether or not strong duality holds in the test cases, if this is not too challenging to check. If so, the optimality gaps reported also should hold in the primal. The authors may also want to come up with some examples of why learning dual solutions directly may be a contribution on its own, outside of just providing a bound on a primal solution.\n\nI would also want to see more of a discussion about the comparison with reference [19], specifically, is [19] already generalizable but the authors only applied it to the AC OPF problem in the paper?\n\nRegarding the use of implicit layers, a discussion (or even better, experiments if time allows) on their computational limitations and how this slows down inference as problem size grows\n\nLIMITATIONS:\nThe authors have sufficiently addressed the limitations of the method. I don't see any potential negative societal impacts of the work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the Dual Lagrangian Learning (DLL) framework that utilizes a fully-connected neural network (FCNN) to provide high-quality dual-feasible solutions that can be used to generate valid Lagrangian dual bounds for linear and nonlinear conic optimization problems without the use of costly implicit layers. This FCNN provides a candidate dual feasible point that is passed through a conic projection step to produce a dual feasible point, then dual completion subproblem is solved to get a complete dual feasible pair $(\\hat{z}, \\hat{y}) which is then used to get a valid Lagrangian dual bound. The subgradient of the Lagrangian at the dual feasible point is used to update the FCNN network weights to provide better candidate points. The dual completion subproblem is solved for a variety of conic optimization problems. The experimental evaluation is done on both linear and nonlinear conic optimization problems and it is shown to outperform the state-of-art method DC3.\n\nSTRENGTHS:\nThe strength of the paper is the simplicity and effectiveness of the DLL framework compared to DC3 and commercial interior point solvers. While previous work used dual feasible points as a warm-start, this work shows how dual feasible points can be used to get valid Lagrangian dual bounds.\n\nWEAKNESSES:\nThe main weakness of the paper is lack of consideration for general non-convex constraints. A minor weakness of the paper is lack of detail about how DC3 was tuned. The authors claim that DC3 was difficult to tune and only limited tuning was done with DC3 due to this. However, on the positive side, they do provide adequate detail of how the hyperparameters for DC3 were set for easy reproducibility.\n\nQUESTIONS:\n1. On Line 515-6, there seems to be a typo. \"do not valid bounds\" should be replaced with \"do not output valid bounds\"\n2. How was the limited tuning of DC3 done? While DC3 has many parameters and is difficult to tune, it is possible that DC3 maybe better with more extensive tuning. It is important to show that a reasonable effort was made to tune DC3 before claiming that its performance is worse than DLL.\n3. How does the quality of the candidates provided by DLL change over time? It would be interesting to see how the Lagrangian dual bound changes with each iteration of DLL. \n4. What considerations did you use when designing the FCNN for DLL? Have you tried other NN architectures other than GNNs?\n\nLIMITATIONS:\nThe authors acknowledge the limitations of their work, which include lack of consideration for general non-convex constraints and suggest Graph Neural Networks (GNNs) as a promising future avenue of research.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Dual Lagrangian Learning (DLL), a machine learning framework designed for dual conic optimization problems. DLL utilizes conic duality and machine learning models to produce high-quality, dual-feasible solutions and corresponding Lagrangian dual bounds for both linear and nonlinear conic optimization problems. It introduces techniques like a systematic dual completion procedure, differentiable conic projection layers, and a self-supervised learning framework based on Lagrangian duality. The methodology is demonstrated to significantly outperform existing state-of-the-art methods and provides a faster alternative to traditional interior-point methods used in commercial solvers.\n\nSTRENGTHS:\nThe paper introduces Dual Lagrangian Learning (DLL), which combines machine learning (ML) techniques with dual conic optimization to enhance the computation of dual feasible solutions and corresponding Lagrangian bounds. This integration addresses both linear and nonlinear conic optimization problems, aiming at providing a new approach as compared to a traditional methods. The methodology applies differentiable programming to optimization, thus enabling gradient-based optimization methods to refine dual solutions directly. The paper is well-structured and articulates complex ideas with clarity.\n\nWEAKNESSES:\nDespite its strengths, the paper has several areas that could be improved:\n\nScope of Application: The focus on continuous problems seems a step back considering the current trend and necessity of addressing mixed-integer programming (MIP) in real-world important applications like power systems and manufacturing, which authors rightfully point out in the beginning. The utility of DLL in these contexts remains unclear, especially without addressing the discrete aspects of such problems.\nIncremental Nature: The contributions, while innovative in combining ML with dual conic optimization, appear incremental when compared to existing research integrating ML with Lagrangian relaxation for MIP. More distinction or comparison with these methods could help in highlighting the unique advantages of DLL.\nNumerical Validation: The experiments provided are based on relatively small problem instances. This limitation might raise concerns about the scalability and practical applicability of DLL. Even by the standards of MIP applications, the sizes of the problems tested appear to be rather small. Extending these experiments to larger, more complex problem instances could enhance the paper's impact and relevance.\n\nQUESTIONS:\nJustification for Focus on Continuous Problems: Can the authors provide a stronger rationale for focusing predominantly on continuous problems given the prevalent interest and necessity for solving MIP in real-world applications?\nComparison with Existing Methods: How does DLL compare in performance and applicability with existing methods that combine ML with Lagrangian relaxation specifically for MIP problems?\nScalability and Practical Applicability: Could the authors elaborate on the potential scalability of DLL? Are there plans to test the methodology on larger-scale problems, particularly those involving discrete decisions common in industrial applications?\n\nLIMITATIONS:\nThe limitations are provided in the paper, but in the view of the above review, they appear to be rather restrictive.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}