Title: IPM-LSTM: A Learning-Based Interior Point Method for Solving Nonlinear Programs
Venue: NeurIPS 2024
Accepted: yes
Keywords: learning to optimize

interior point method

nonlinear constrained programs
Abstract: Solving constrained nonlinear programs (NLPs) is of great importance in various domains such as power systems, robotics, and wireless communication networks. One widely used approach for addressing NLPs is the interior point method (IPM). The most computationally expensive procedure in IPMs is to solve systems of linear equations via matrix factorization. Recently, machine learning techniques have been adopted to expedite classic optimization algorithms. In this work, we propose using Long Short-Term Memory (LSTM) neural networks to approximate the solution of linear systems and integrate this approximating step into an IPM. The resulting approximate NLP solution is then utilized to warm-start an interior point solver. Experiments on various types of NLPs, including Quadratic Programs and Quadratically Constrained Quadratic Programs, show that our approach can significantly accelerate NLP solving, reducing iterations by up to 60% and solution time by up to 70% compared to the default solver.

IPM-LSTM: A Learning-Based Interior Point Method for Solving Nonlinear Programs

Abstract

Solving constrained nonlinear programs (NLPs) is of great importance in various domains such as power systems, robotics, and wireless communication networks. One widely used approach for addressing NLPs is the interior point method (IPM). The most computationally expensive procedure in IPMs is to solve systems of linear equations via matrix factorization. Recently, machine learning techniques have been adopted to expedite classic optimization algorithms. In this work, we propose using Long Short-Term Memory (LSTM) neural networks to approximate the solution of linear systems and integrate this approximating step into an IPM. The resulting approximate NLP solution is then utilized to warm-start an interior point solver. Experiments on various types of NLPs, including Quadratic Programs and Quadratically Constrained Quadratic Programs, show that our approach can significantly accelerate NLP solving, reducing iterations by up to 60% and solution time by up to 70% compared to the default solver.

1 Introduction

Constrained Nonlinear Programs (NLPs) represent a category of mathematical optimization problems in which the objective function, constraints, or both, exhibit nonlinearity. Popular NLP variants encompass Quadratic Programs (QPs), Quadratically Constrained Quadratic Programs (QCQPs), semi-definite programs, among others. These programs are commonly classified as convex or non-convex, contingent upon the characteristics of their objective function and constraints. The versatility of NLPs allows for their application across a wide array of domains, including power systems (Conejo and Baringo, 2018), robotics (Schaal and Atkeson, 2010), and wireless communication networks (Chiang, 2009).

The primal-dual Interior Point Method (IPM) stands as a preeminent algorithm for addressing NLPs (Nesterov and Nemirovskii, 1994; Nocedal and Wright, 1999). It initiates with an infeasible solution positioned sufficiently far from the boundary. Subsequently, at each iteration, the method refines the solution by solving a system of linear equations, thereby directing it towards the optimal solution. Throughout this iterative process, the algorithm progresses towards feasibility and optimality while keeping the iterate well-centered, ultimately converging to the optimal solution. However, a notable computational bottleneck arises during the process of solving linear systems, necessitating matrix decomposition with a runtime complexity of O ( n 3 ) .

Recently, the Learning to Optimize (L2O) (Bengio et al., 2021; Chen et al., 2024; Gasse et al., 2022) paradigm has emerged as a promising methodology for tackling various optimization problems,

IPM-LSTM

min

f(α)

CERn

grad

grad

LSTMcell

LSTMcell

c≥0,

Opt.sol.

C

￥

IPOPT

warm

start

#

miny

1

JKy+Fk12

LSTM

y

k

newiterate

spanning unconstrained optimization (Chen et al., 2022a), linear optimization (Chen et al., 2022b; Li et al., 2024), and combinatorial optimization (Baker, 2019; Gasse et al., 2022; Han et al., 2023). Its ability to encapsulate common optimization patterns renders it particularly appealing. Noteworthy is the application of learning techniques to augment traditional algorithms such as the gradient descent method (Andrychowicz et al., 2016), simplex method (Liu et al., 2024), and IPM (Qian et al., 2024).

We observe that existing works on learning-based IPMs primarily concentrate on solving LPs (Qian et al., 2024). Motivated by the robustness and efficiency of IPMs for general NLPs, we pose the following question:

Can we leverage L2O techniques to expedite IPMs for NLPs?

In this study, we propose the integration of Long Short-Term Memory (LSTM) neural networks to address the crucial task of solving systems of linear equations within IPMs, introducing a novel approach named IPM-LSTM. An illustration of the IPM-LSTM approach is depicted in Figure 1. Specifically, we substitute the conventional method of solving linear systems with an unconstrained optimization problem, leveraging LSTM networks to identify near-optimal solutions for the latter. We integrate a fixed number of IPM iterations into the LSTM loss function and train these networks within the self-supervised learning framework. The substitution is embedded within a classic IPM to generate search directions. Ideally, the primal-dual solution provided by IPM-LSTM should be well-centered with respect to the boundary and associated with a small duality gap. Finally, we utilize such approximate primal-dual solution pairs to warm-start an interior point solver. IPM-LSTM has several attractive features: ( i ) it can be applied to general NLPs ; ( ii ) it strikes a good balance between feasibility and optimality in the returned solutions; ( iii ) it can warm-start and thereby accelerate interior point solvers.

The distinct contributions of our work can be summarized as follows:

Approximating Solutions to Linear Systems via LSTM: This study marks the first attempt to employ learning techniques for approximating solutions of linear systems in IPMs, achieving significant speedup compared to traditional linear algebra approaches.

Two-Stage Framework: We introduce a two-stage L2O framework. In the first stage, IPMLSTM generates high-quality primal-dual solutions. In the second stage, these solutions are used to warm-start an interior point solver. This framework effectively accelerates the solving process of IPMs while yielding optimal solutions.

Empirical Results: Compared with existing L2O algorithms, IPM-LSTM demonstrates favorable performance in terms of solution feasibility and optimality across various NLP types, including QPs and QCQPs. Utilizing these solutions as initial points in the state-of-the-art NLP solver IPOPT (Wächter and Biegler, 2006) reduces iterations by up to 60% and solution time by up to 70% .

2 Related Works

Constrained L2O. Approaches utilizing L2O for constrained optimization can be broadly categorized into two directions: (i) direct learning of the mapping from optimization inputs to full solutions, and (ii) integration of learning techniques alongside or within optimization algorithms (Bengio et al., 2021; Donti et al., 2021). Previous works (Fioretto et al., 2020; Huang et al., 2021; Pan et al., 2023) adopted the former approach, employing a supervised learning scheme to train the mapping. However, this method necessitates a large number of (near-)optimal solutions as training samples, making it resource-intensive. From a self-supervised learning perspective, an intuitive approach is to incorporate the objective and penalization for constraint violation directly into the loss function (Kim et al., 2023; Park and Van Hentenryck, 2023). Nevertheless, such an approach may not guarantee the feasibility of the returned solutions. To address the feasibility issue, notable works such as Donti et al. (2021) first predict a partial solution via neural networks and then complete the full solution by utilizing equality constraints, iteratively correcting the solution towards the satisfaction of inequalities by applying gradient-based methods. However, for general nonlinear inequalities, this correction step may not ensure feasibility (Liang et al., 2023). Other approaches like Li et al. (2023) utilized gauge mappings to enforce feasibility for linear inequalities, while Liang et al. (2023) proposed the homeomorphic projection scheme to guarantee feasibility. However, these methods have limitations; the former is only applicable to linearly constrained problems, and the latter works for problems with feasibility regions homeomorphic to a unit ball. Another critical issue with the approach in Donti et al. (2021) is that the completion step may fail during training when the equality system with some fixed variables becomes infeasible, as highlighted in Han et al. (2024) and Zeng et al. (2024). Consequently, such an approach will not succeed during the training stage. To mitigate this issue, Han et al. (2024) proposed solving a projection problem if the completion step fails. However, the computationally expensive projection step may still be necessary during inference, which hinders its practical value.

Learning-Based IPMs. Primal-dual IPMs are polynomial-time algorithms used for solving constrained optimization problems such as LPs and NLPs. The work of Qian et al. (2024) demonstrated that properly designed Graph Neural Networks (GNNs) can theoretically align with IPMs for LPs, enabling GNNs to function as lightweight proxies for solving LPs. However, extending this alignment to NLPs is challenging as representation learning for general NLPs remains unknown. Another avenue of research in learning-based IPMs involves warm-starting implementation. Previous works like Baker (2019), Diehl (2019) and Zhang and Zhang (2022) addressed alternative current optimal power flow (ACOPF) applications and proposed using learning models, such as GNNs, to learn the mapping between ACOPF and its optimal solutions. These predicted solutions are then utilized as initial points to warm-start an interior point optimizer. However, even if these solutions are close to the optimal ones, they may not be well-centered with respect to the trajectory in IPMs, causing the optimizer to struggle in progressing towards feasibility and optimality (Forsgren, 2006).

3 Approach

3.1 The Classic IPM

We focus on solving the following NLP (1):

where the functions f : R n → R and h : R n → R m are all assumed to be twice continuously differentiable. Problems with general nonlinear inequality constraints can be reformulated in the above form by introducing slack variables. We note that for simplicity, we assume all variables in (1) are non-negative, though NLPs with arbitrary variable bounds can also be handled effectively. Readers are referred to Appendix A for details.

The primal-dual IPM stands as one of the most widely utilized approaches for addressing NLPs. It entails iteratively solving the perturbed Karush-Kuhn-Tucker (KKT) conditions (2) for a decreasing

sequence of parameters µ converging to zero.

where λ ∈ R m and z ∈ R n + denote the corresponding dual variables, diag ( · ) represents a diagonal matrix, and e is a vector of ones. Let F ( x, λ, z ) = 0 denote the system of nonlinear equations in (2). We then employ a one-step Newton's method to solve such a system, aiming to solve systems of linear equations (3).

The IPM commences with an initial solution ( x 0 , λ 0 , z 0 ) such that x 0 , z 0 > 0 . At iteration k , the linear system (3) defined by the current iterate ( x k , λ k , z k ) is solved, with µ := σ [ ( z k ) ⊤ x k ] /n being the perturbation parameter and constant σ ∈ (0 , 1) . A line-search filter step along the direction (∆ x k , ∆ λ k , ∆ z k ) is then performed to ensure boundary condition satisfaction as well as sufficient progress towards objective value improvement or constraint violation reduction. This process iterates until convergence criteria, such as achieving optimality and feasibility within specified tolerances, are met. The IPM is guaranteed to converge to a KKT point with a superlinear rate. A pseudocode of the IPM is presented as Algorithm 1.

Algorithm 1 The classic IPM

Inputs: An initial solution ( x 0 , λ 0 , z 0 ) , σ ∈ (0 , 1) , k ← 0 Outputs: The optimal solution ( x ∗ , λ ∗ , z ∗ )

1: while not converged do

3: Solve the system J k [ (∆ x k ) ⊤ , (∆ λ k ) ⊤ , (∆ z k ) ⊤ ] ⊤ = -F k 4: Choose α k via a line-search filter method

2: Update µ k

5: ( x k +1 , λ k +1 , z k +1 ) ← ( x k , λ k , z k ) + α k (∆ x k , ∆ λ k , ∆ z k )

7: end while

6: k ← k +1

We note that, in classic IPMs, one typically reformulates the system (3) and then solves a reduced system of equations (i.e., augmented system) for greater efficiency. However, in this work, we are interested in the full systems since they are associated with smaller condition numbers (Greif et al., 2014) that are critical to the performance of our proposed approach. Additionally, various techniques have been proposed to enhance the robustness and efficiency of IPMs, including secondorder correction, inertial correction, and feasibility restoration. Interested readers are referred to Wächter and Biegler (2006) for further details about IPMs.

3.2 Approximating Solutions to Linear Systems

The small number of iterations in IPMs does not always guarantee efficiency because, at times, IPMs encounter a high per-iteration cost of linear algebra operations. In the worst-case scenario, the cost of solving a dense optimization problem using a direct linear algebra method to solve the Newton equation system (3) may reach O ( n 3 ) flops per iteration. This motivates us to avoid computing exact solutions to linear systems and instead focus on their approximations. Toward this goal, we consider the following least squares problem (4):

where ∥·∥ denotes the Euclidean norm. If (3) is solvable, then an optimal solution to problem (4) is also the exact solution [ (∆ x k ) ⊤ , (∆ λ k ) ⊤ , (∆ z k ) ⊤ ] ⊤ to system (3). Otherwise, we resort to an approximation of the latter. This perspective is similar to the inexact IPM (Bellavia, 1998; Dexter et al., 2022).

Assumption 1. At iteration k , we could identify some y k such that

where η ∈ (0 , 1) and F 0 ( x k , λ k , z k ) denotes F ( x k , λ k , z k ) with µ = 0 .

To satisfy Assumption 1, the approximate solution y k has to be bounded and accurate enough, regardless of whether J k is invertible.

Proposition 1 (Bellavia (1998)) . If ( x k , λ k , z k ) is generated such that Assumption 1 is satisfied, let ( x ∗ , λ ∗ , z ∗ ) denote a limit point of the sequence { ( x k , λ k , z k ) } , then { ( x k , λ k , z k ) } converges to ( x ∗ , λ ∗ , z ∗ ) and F 0 ( x ∗ , λ ∗ , z ∗ ) = 0 .

Proposition 1 implies that if solutions with specified accuracy for linear systems in Step 3 are found, the IPM would converge.

3.3 The IPM-LSTM Approach

The problem (4) is an unconstrained convex optimization problem. Various L2O methods have been proposed to solve such problems (Chen et al., 2022a; Gregor and LeCun, 2010; Liu et al., 2023). We will employ the LSTM networks in our L2O method for addressing problem (4), hence our approach is called 'IPM-LSTM'.

Model Architecture. LSTM is a type of recurrent neural network designed to effectively capture and maintain long-term dependencies in sequential data (Yu et al., 2019). LSTM networks are commonly considered suitable for solving unconstrained optimization problems due to the resemblance between LSTM recurrent calculations and iterative algorithms (Andrychowicz et al., 2016; Liu et al., 2023; Lv et al., 2017).

The LSTM network consists of T cells parameterized by the same learnable parameters θ . Each cell can be viewed as one iteration of a traditional iterative method, as illustrated in Figure 2. Let ϕ ( y ) := 1 2 ∥ ∥ J k y + F k ∥ ∥ 2 for convenience. The t -th cell takes the previous estimate y t -1 and the gradient ( J k ) ⊤ ( J k y t -1 + F k ) as the input and outputs the current estimate y t :

The T -th cell yields y T as an approximate solution to problem (4). As suggested by Andrychowicz et al. (2016) and Liu et al. (2023), we utilize a coordinate-wise LSTM that shares parameters not only across different LSTM cells but also for all coordinates of y .

yt-1

yt

yt+1

gradt

gradt+1

$

个

tanh

LSTM Cell

×

→

LSTM Cell

tanh

(yt-2,VΦ(yt-2))

(yt-1,VΦ(yt-1))

Model Training. We train the proposed optimizer by finding the optimal θ in (7) on a dataset M of NLPs. Each sample in M is an instance of the optimization problem. During training, we apply the optimizer to each instance M ∈ M , performing K IPM iterations in the outer loop and T LSTM

time steps in the inner loop, generating a sequence of iterates { ( y 1 1 , ..., y 1 T ) , ..., ( y K 1 , ..., y K T ) } where the superscript k denotes the IPM iteration number. We then optimize θ by minimizing the following loss function:

where the subscript M indicates that the corresponding term is associated with instance M . Clearly, our model training falls into the category of self-supervised learning . To mitigate memory issues caused by excessively large computational graphs, we employ truncated backpropagation through time after each IPM iteration during training, as done in Chen et al. (2022a) and Liu et al. (2023).

Preconditioning. The Hessian matrix of ϕ ( y ) is ( J k ) ⊤ J k , whose condition number, κ (( J k ) ⊤ J k ) , is the square of that of J k . Consequently, κ (( J k ) ⊤ J k ) can easily become very large. Since solving system (4) via LSTM networks emulates iterative first-order methods, the value of κ (( J k ) ⊤ J k ) strongly affects the performance of LSTM networks. To address this issue, we employ a simple diagonal preconditioning technique that rescales the Hessian matrix ( J k ) ⊤ J k using the Ruiz scaling method (Ruiz, 2001) to decrease its condition number.

3.4 Two-Stage Framework

To further enhance the solution quality, we propose a two-stage framework that initially obtains a nearoptimal and well-centered primal-dual solution via IPM-LSTM and then utilizes this approximate solution to warm-start an interior point solver. In this study, we select IPOPT (Wächter and Biegler, 2006), an IPM-based solver renowned for its robustness and efficiency in optimizing NLPs.

Our two-stage framework works as follows: Given an NLP (1), we generate an initial point ( x 0 , λ 0 , z 0 ) and formulate the least squares problem (4). Subsequently, the trained LSTM network with T cells solves this problem and returns a search direction. We then employ the simple fractional-to-boundary method (Wächter and Biegler, 2006) to determine the step size and reach the new iterate. This procedure is iterated K times, resulting in a primal-dual solution ( x K , λ K , z K ) . Finally, the obtained solution serves as the warm-start solution for IPOPT, leading to the optimal solution x ∗ upon IPOPT convergence.

4 Experiments

4.1 Experimental Settings

We evaluate our approach and compare its performance against traditional methods as well as L2O algorithms for solving various types of NLPs. Furthermore, we also quantify the warm-starting effect of our proposed two-stage approach. Our code is available at https://github.com/NetSysOpt/IPMLSTM.

Baseline Algorithms. In our experiments, we denote our algorithm by IPM-LSTM and compare it against both traditional optimizers and L2O algorithms. The traditional optimizers considered are: (i) OSQP (Stellato et al., 2020): an ADMM-based solver designed for convex QPs. (ii) IPOPT 3.14.8 (Wächter and Biegler, 2006): a state-of-the-art IPM-based solver for NLPs with the default linear solver MUMPS (Amestoy et al., 2000) and a convergence tolerance of 10 -4 . We also assess several L2O algorithms, including: (i) NN (Donti et al., 2021): a straightforward deep learning approach that integrates the objective function and penalty for constraint violations into the loss function. (ii) DC3 (Donti et al., 2021): an end-to-end method that uses 'completion' steps to maintain equality constraints and 'correction' steps for inequality feasibility. (iii) DeepLDE (Kim et al., 2023): an algorithm that trains neural networks using a primal-dual approach to impose inequality constraints and employs 'completion' for equality constraints. (iv) PDL (Park and Van Hentenryck, 2023): a self-supervised learning method that jointly trains two networks to approximate primal and dual solutions. (v) LOOP-LC (Li et al., 2023): a neural approximator that maps inputs of linearly constrained models to high-quality feasible solutions using gauge maps. (vi) H-Proj (Liang et al., 2023): a method that applies a homeomorphic projection scheme to post-process solutions resulting from the completion step in DC3 .

Datasets. The dataset used in this paper includes randomly generated benchmarks obtained from Chen and Burer (2012), Donti et al. (2021) and Liang et al. (2023), as well as real-world instances

from Globallib (see http://www.minlplib.org). These benchmarks encompass QPs, QCQPs, and simplex non-convex programs. For each case, we generate 10 , 000 samples and divide them into a 10 : 1 : 1 ratio for training, validation, and testing, respectively. All numerical results are reported for the test set.

Model Settings. All LSTM networks have a single layer and are trained using the Adam optimizer (Kingma, 2014). During IPM-LSTM training, an early stopping strategy with a patience of 50 is employed, halting training if no improvement is observed for 50 iterations, while satisfying inequality and equality constraints violation less than 0.005 and 0.01. The learning rate is 0.0001, and the batch size is 128 for each task. Additional IPM-LSTM parameters for each task are provided in Appendix C.

Evaluation Configuration. All our experiments were conducted on an NVIDIA RTX A6000 GPU, an Intel Xeon 2.10GHz CPU, using Python 3.10.0 and PyTorch 1.13.1.

4.2 Computational Results

Convex QPs. We consider convex QPs with both equality and inequality constraints:

where Q 0 ∈ S n + , p j ∈ R n , q j ∈ R , x L i ∈ R ∪{-∞} and x U i ∈ R ∪{ + ∞} . We conduct experiments on two groups of QPs, each instance with 200 variables, 100 inequalities and 100 equalities. The first group is generated in the same way as Donti et al. (2021), where only the right hand sides of equality constraints are perturbed while the second one considers perturbation for all model parameters. Let 'Convex QP (RHS)' denote the former and 'Convex QPs (ALL)' denote the latter. It is noteworthy that, in line with Donti et al. (2021), we also investigate the performance of IPM-LSTM and baseline algorithms on smaller-scale convex QPs. Interested readers are directed to Appendix D for details.

We ran IPM-LSTM and all baseline algorithms on the test set 'Convex QPs (RHS)', and we reported their computational results, which were averaged across 833 instances. The results are presented in Table 1. We denote this experiment by 'End-to-End' for convenience. The columns labeled 'Max Ineq.', 'Mean Ineq.', 'Max Eq.', and 'Mean Eq.' denote the maximum and mean violations for inequalities and equalities, respectively. The columns 'Obj.' and 'Time (s)' represent the final primal objective and the runtime in seconds. Both OSQP and IPOPT solved these instances to guaranteed optimality, with OSQP being significantly more efficient due to its specialization as a QP-specific solver. All L2O baseline algorithms returned solutions very quickly. However, solutions generated by NN and PDL exhibited significant constraint violations. While the solutions from DC3 and DeepLDE were nearly feasible, they corresponded to inferior objective values. On the other hand, LOOP-LC produced feasible and near-optimal solutions, whereas H-Proj generated feasible solutions but with

larger objective values. Solutions identified by IPM-LSTM showed mild constraint violations but yielded superior objective values very close to the optimal values. Clearly, IPM-LSTM effectively balances feasibility and optimality in the returned solutions. This comes at the cost of longer runtime compared to the L2O baseline algorithms, as the baselines typically employ simple multi-layer perceptrons, whereas IPM-LSTM utilizes a neural network with several dozen LSTM cells.

Since IPM-LSTM is designed to provide interior point optimizers with high-quality initial points, we fed the returned primal-dual solution pair to IPOPT and reported the performance in Table 1. For comparison, we also provided IPOPT with initial points generated from other L2O algorithms. The columns labeled 'Ite.' and 'Time (s)' under ' IPOPT (warm-start)' indicate the number of iterations and solver time, respectively, while the column 'Total Time (s)' represents the cumulative time for running both the L2O algorithms and IPOPT . The final column, 'Gain (Ite/Time)', shows the reduction in iteration number and solution time, with the default IPOPT iteration number listed in the 'Ite.' column for reference. When initial solutions from IPM-LSTM and most L2O baseline algorithms (except DeepLDE ) were used, IPOPT converged with fewer iterations and reduced solution time. Notably, IPM-LSTM achieved the most significant reduction in iterations, from 12 . 5 to 7 . 2 , and decreased the average solver time from 0 . 642 seconds to 0 . 37 seconds. Including the computational time for IPM-LSTM , the total runtime was 0 . 545 seconds, reflecting a 15.1% reduction in time. It is worth noting that while IPM-LSTM did not yield the maximum solution time reduction, this was due to its relatively high computational expense.

To our knowledge, there is no existing representation learning approach for general convex QPs. Since the aforementioned L2O baseline algorithms depend on specific representations of QPs, they are not applicable to 'Convex QPs (ALL)'. Therefore, we only provide results for OSQP , IPOPT , and IPM-LSTM . We also report computational results averaged across 833 instances in the test set 'Convex QPs (ALL)', presented in Table 1. The results demonstrate that IPM-LSTM can identify high-quality solutions for general convex QPs, and using these solutions as initial points can reduce iterations by 35 . 7% and solution time by 7 . 5% .

Convex QCQPs. We now turn to convex QCQPs with both equaltity and inequality constraints:

where Q j ∈ S n + , p j ∈ R n , q j ∈ R , x L i ∈ R ∪{-∞} and x U i ∈ R ∪{∞} . Similar to our experiments on convex QPs, we also consider two groups of convex QCQPs, each with 200 variables, 100 inequality constraints, and 100 equality constraints. The first group (denoted as 'Convex QCQPs (RHS)') is generated as described in Liang et al. (2023), with perturbations only to the right-hand sides of the equality constraints. The second group (denoted as 'Convex QCQPs (ALL)') considers perturbations to all parameters. We also refer readers to Appendix D for computational experiments on smaller-sized convex QCQPs.

Weomit OSQP and LOOP-LC since the former cannot handle QCQPs, while the latter is only applicable to linearly constrained problems. We evaluate IPM-LSTM and compare it against the remaining baseline algorithms. The computational results are reported in Table 2. Again, solutions produced by NN and PDL exhibit significant constraint violations, while those from DC3 and H-Proj are of high quality in terms of feasibility and optimality. Once more, the solutions produced by DeepLDE were deemed feasible but exhibited inferior objective values. Conversely, our approach, IPM-LSTM , produced solutions with superior objective values albeit with minor infeasibility. Compared to the baseline algorithms, utilizing solutions from IPM-LSTM to warm-start IPOPT resulted in the most substantial reduction in iterations.

As the aforementioned L2O baseline algorithms are not applicable to 'Convex QCQPs (ALL)', we only report computational results for IPOPT and IPM-LSTM in Table 2. The IPM-LSTM approach produced high-quality approximate solutions to convex QCQPs, and warm-starting IPOPT with these solutions accelerated IPOPT by 11 . 4% , with a 33 . 1% reduction in iterations.

Non-convex QPs. We now consider non-convex QPs of exactly the same form as (8) but with Q 0 being indefinite. We take 8 representative non-convex QPs from the datasets Globallib and

RandQP (Chen and Burer, 2012), each with up to 50 variables and 20 constraints, and perturb the relevant parameters for instance generation. Details can be found in Appendix D.2.

Among all the baselines, only IPOPT is applicable to these general non-convex QPs. Hence, we report computational results for IPOPT and IPM-LSTM in Table 3. IPOPT solved these instances to local optimality, while IPM-LSTM identified high-quality approximate solutions very efficiently. Using these primal-dual approximations to warm-start IPOPT resulted in an iteration reduction of up to 63 . 9% and a solution time reduction of up to 70 . 5% .

Max Vio. denotes the maximum constraint violation.

Simple non-convex programs. Following the approach outlined in Donti et al. (2021), we consider a set of simple non-convex programs where the linear objective term in (8) is substituted with p ⊤ 0 sin( x ) . These instances were generated using the same methodology as Donti et al. (2021). We subject these problems to evaluation using both IPM-LSTM and baseline algorithms. Once again, the computational results underscore the high-quality solutions obtained by IPM-LSTM and its superior warm-starting capability. Detailed results are provided in Appendix D.4.

4.3 Performance Analysis of IPM-LSTM

In Assumption 1, we posit that the linear systems (3) can be solved with an acceptable residual (Condition (5)) and that the solutions are properly bounded (Condition (6)). Although the LSTM network cannot guarantee the satisfaction of these conditions, we empirically assess the validity of Assumption 1 in IPM-LSTM . We plot the progress of ∥ ∥ J k y k + F k ∥ ∥ , η [ ( z k ) ⊤ x k ] /n , ∥ ∥ y k ∥ ∥ , and ∥ ∥ F 0 ( x k , λ k , z k ) ∥ ∥ as the IPM iterations increase in Figure 3(a) and 3(b). Condition (5) is mostly satisfied except during the first few iterations, while Condition (6) is strictly satisfied across all IPM iterations. To assess the precision of the approximate solutions as the LSTM time steps increase, we plot the progress of the residual for linear systems (3) in Figure 3(c). At IPM iteration k , the residual decreases monotonically towards 0 as the LSTM time steps increase, indicating that LSTM networks can produce high-quality approximate solutions to system (3). The approximation quality improves with the number of IPM iterations. As shown in Figure 3(d), the primal objective decreases monotonically towards the optimal value with increasing IPM iterations, empirically demonstrating the convergence of IPM-LSTM . Note that the condition number κ (( J k ) ⊤ J k ) becomes quite large in the later IPM iterations, which can adversely affect the performance of LSTM networks in obtaining approximations to systems (3). Consequently, we terminate IPM-LSTM after a finite number of iterations. Further analysis regarding the number of IPM iterations and LSTM time steps is

presented in Appendix D.5. Additionally, we include the performance of IPM-LSTM under various hyperparameter settings in Appendix D.5.

(a) Condition (5)

(b) Condition (6)

0

10

20

30

40

50

LSTM time step

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

15

.

0

17

.

5

1

2

∥

∥

J

k

y

k

+

F

k

∥

∥

2

IPM iteration 1

IPM iteration 2

IPM iteration 3

IPM iteration 10

IPM iteration 20

IPM iteration 60

IPM iteration 100

(c) Residual

0

20

40

60

80

100

IPM iteration

-

15

-

10

-

5

0

5

10

Objective value

IPM-LSTM

Optimal value

(d) Objective value

5 Limitations and Conclusions

In this paper, we present a learning-based IPM called IPM-LSTM. Specifically, we propose approximating solutions of linear systems in IPMs by solving least square problems using trained LSTM networks. We demonstrate that IPMs with this approximation procedure still converge. The solutions returned by IPM-LSTM are used to warm-start interior point optimizers. Our computational experiments on various types of NLPs, including general QPs and QCQPs, showcase the effectiveness of IPM-LSTM and its ability to accelerate IPOPT. Although IPM-LSTM generates high-quality primal-dual solutions, it is relatively computationally expensive due to the utilization of multi-cell LSTM networks. In future endeavors, we aim to investigate the efficacy of employing low-complexity neural networks to approximate solutions of linear systems within IPMs.

Acknowledgments

This work was supported by the National Key R&D Program of China under grant 2022YFA1003900. Jinxin Xiong and Akang Wang also gratefully acknowledge support from the National Natural Science Foundation of China (Grant No. 12301416), the Shenzhen Science and Technology Program (Grant No. RCBS20221008093309021), the Guangdong Basic and Applied Basic Research Foundation (Grant No. 2024A1515010306) and Longgang District Special Funds for Science and Technology Innovation (LGKCSDPT2023002).

References

Patrick R Amestoy, Iain S Duff, Jean-Yves L'Excellent, and Jacko Koster. Mumps: a general purpose distributed memory sparse solver. In International Workshop on Applied Parallel Computing , pages 121-130. Springer, 2000.

Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems , 29, 2016.

Kyri Baker. Learning warm-start points for ac optimal power flow. In 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP) , pages 1-6. IEEE, 2019.

Stefania Bellavia. Inexact interior-point method. Journal of Optimization Theory and Applications , 96:109-121, 1998.

Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: a methodological tour d'horizon. European Journal of Operational Research , 290(2):405-421, 2021.

Jieqiu Chen and Samuel Burer. Globally solving nonconvex quadratic programming problems via completely positive programming. Mathematical Programming Computation , 4(1):33-52, 2012.

Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. Learning to optimize: A primer and a benchmark. Journal of Machine Learning Research , 23(189):1-59, 2022a.

Xiaohan Chen, Jialin Liu, and Wotao Yin. Learning to optimize: A tutorial for continuous and mixed-integer optimization. Science China Mathematics , pages 1-72, 2024.

Ziang Chen, Jialin Liu, Xinshang Wang, and Wotao Yin. On representing linear programs by graph neural networks. In The Eleventh International Conference on Learning Representations , 2022b.

Mung Chiang. Nonconvex optimization for communication networks. Advances in Applied Mathematics and Global Optimization: In Honor of Gilbert Strang , pages 137-196, 2009.

Antonio J Conejo and Luis Baringo. Power system operations , volume 14. Springer, 2018.

Gregory Dexter, Agniva Chowdhury, Haim Avron, and Petros Drineas. On the convergence of inexact predictorcorrector methods for linear programming. In International Conference on Machine Learning , pages 5007-5038. PMLR, 2022.

Frederik Diehl. Warm-starting ac optimal power flow with graph neural networks. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019) , pages 1-6, 2019.

Priya L Donti, David Rolnick, and J Zico Kolter. Dc3: A learning method for optimization with hard constraints. In International Conference on Learning Representations , 2021.

Stanley C Eisenstat and Homer F Walker. Globally convergent inexact newton methods. SIAM Journal on Optimization , 4(2):393-422, 1994.

Ferdinando Fioretto, Terrence WK Mak, and Pascal Van Hentenryck. Predicting ac optimal power flows: Combining deep learning and lagrangian dual methods. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 630-637, 2020.

Anders Forsgren. On warm starts for interior methods. In System Modeling and Optimization: Proceedings of the 22nd IFIP TC7 Conference held from July 18-22, 2005, in Turin, Italy 22 , pages 51-66. Springer, 2006.

Maxime Gasse, Simon Bowly, Quentin Cappart, Jonas Charfreitag, Laurent Charlin, Didier Chételat, Antonia Chmiela, Justin Dumouchelle, Ambros Gleixner, Aleksandr M Kazachkov, et al. The machine learning for combinatorial optimization competition (ml4co): Results and insights. In NeurIPS 2021 competitions and demonstrations track , pages 220-231. PMLR, 2022.

Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference on international conference on machine learning , pages 399-406, 2010.

Chen Greif, Erin Moulding, and Dominique Orban. Bounds on eigenvalues of matrices arising from interior-point methods. SIAM Journal on Optimization , 24(1):49-83, 2014.

Jiayu Han, Wei Wang, Chao Yang, Mengyang Niu, Cheng Yang, Lei Yan, and Zuyi Li. FRMNet: A Feasibility Restoration Mapping Deep Neural Network for AC Optimal Power Flow. IEEE Transactions on Power Systems , pages 1-11, 2024. ISSN 0885-8950, 1558-0679. doi: 10.1109/TPWRS.2024.3354733. URL https://ieeexplore.ieee.org/document/10411984/ .

Qingyu Han, Linxin Yang, Qian Chen, Xiang Zhou, Dong Zhang, Akang Wang, Ruoyu Sun, and Xiaodong Luo. A gnn-guided predict-and-search framework for mixed-integer linear programming. In The Eleventh International Conference on Learning Representations , 2023.

Wanjun Huang, Xiang Pan, Minghua Chen, and Steven H Low. Deepopf-v: Solving ac-opf problems efficiently. IEEE Transactions on Power Systems , 37(1):800-803, 2021.

Hongseok Kim et al. Self-supervised equality embedded deep lagrange dual for approximate constrained optimization. arXiv preprint arXiv:2306.06674 , 2023.

DP Kingma. Adam: a method for stochastic optimization. In Int Conf Learn Represent , 2014.

Bingheng Li, Linxin Yang, Yupeng Chen, Senmiao Wang, Qian Chen, Haitao Mao, Yao Ma, Akang Wang, Tian Ding, Jiliang Tang, et al. Pdhg-unrolled learning-to-optimize method for large-scale linear programming. In Forty-first International Conference on Machine Learning , 2024.

Meiyi Li, Soheil Kolouri, and Javad Mohammadi. Learning to solve optimization problems with hard linear constraints. IEEE Access , 2023.

Enming Liang, Minghua Chen, and Steven Low. Low complexity homeomorphic projection to ensure neuralnetwork solution feasibility for optimization over (non-) convex set. In Conference on Parsimony and Learning (Recent Spotlight Track) , 2023.

Jialin Liu, Xiaohan Chen, Zhangyang Wang, Wotao Yin, and HanQin Cai. Towards constituting mathematical structures for learning to optimize. In International Conference on Machine Learning , pages 21426-21449. PMLR, 2023.

Tianhao Liu, Shanwen Pu, Dongdong Ge, and Yinyu Ye. Learning to pivot as a smart expert. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 8073-8081, 2024.

Kaifeng Lv, Shunhua Jiang, and Jian Li. Learning gradient descent: Better generalization and longer horizons. In International Conference on Machine Learning , pages 2247-2255. PMLR, 2017.

Yurii Nesterov and Arkadii Nemirovskii. Interior-point polynomial algorithms in convex programming . SIAM, 1994.

Jorge Nocedal and Stephen J Wright. Numerical optimization . Springer, 1999.

Xiang Pan, Minghua Chen, Tianyu Zhao, and Steven H. Low. Deepopf: A feasibility-optimized deep neural network approach for ac optimal power flow problems. IEEE Systems Journal , 17(1):673-683, 2023. doi: 10.1109/JSYST.2022.3201041.

Seonho Park and Pascal Van Hentenryck. Self-supervised primal-dual learning for constrained optimization. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 4052-4060, 2023.

Chendi Qian, Didier Chételat, and Christopher Morris. Exploring the power of graph neural networks in solving linear optimization problems. In International Conference on Artificial Intelligence and Statistics , pages 1432-1440. PMLR, 2024.

Daniel Ruiz. A scaling algorithm to equilibrate both rows and columns norms in matrices. Technical report, CM-P00040415, 2001.

Stefan Schaal and Christopher G Atkeson. Learning control in robotics. IEEE Robotics & Automation Magazine , 17(2):20-29, 2010.

Stellato, G. Banjac, P. Goulart, A. Bemporad, and S. Boyd. OSQP: an operator splitting solver for quadratic programs. Mathematical Programming Computation , 12(4):637-672, 2020. doi: 10.1007/s12532-020-00179-2. URL https://doi.org/10.1007/s12532-020-00179-2 .

Andreas Wächter and Lorenz T Biegler. On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Mathematical programming , 106:25-57, 2006.

Yong Yu, Xiaosheng Si, Changhua Hu, and Jianxun Zhang. A review of recurrent neural networks: Lstm cells and network architectures. Neural computation , 31(7):1235-1270, 2019.

Sihan Zeng, Youngdae Kim, Yuxuan Ren, and Kibaek Kim. QCQP-Net: Reliably Learning Feasible Alternating Current Optimal Power Flow Solutions Under Constraints, January 2024. URL http://arxiv.org/abs/ 2401.06820 . arXiv:2401.06820 [cs, math] version: 1.

Ling Zhang and Baosen Zhang. Learning to solve the ac optimal power flow via a lagrangian approach. In 2022 North American Power Symposium (NAPS) , pages 1-6, 2022. doi: 10.1109/NAPS56150.2022.10012237.

Appendix

A Implementation Details

The general NLP considered in this paper can be formulated as:

where f : R n → R , g : R n → R m ineq , and h : R n → R m eq are twice continuously differentiable functions; s ∈ R m ineq is the slack variable corresponding to the inequality constraints; I L = { i : x L i = -∞} and I U = { i : x U i = ∞} . The Lagrangian function is defined as:

̸

̸

where η ∈ R m ineq and λ ∈ R m eq are the corresponding dual variables. The perturbed KKT conditions are given by:

The nonlinear system extracted from the KKT conditions is represented as:

where

The Jacobian matrix of F ( x, η, λ, s, z L , z U ) is:

where

For convenience, we define the updates of primal and dual variables as:

Then the linear system obtained from one-step Newton's method is shown as:

Initial points. If dual variables η , s , z L , and z U exist, their initial values are set to 1. If the dual variable λ exists, its initial value is set to 0. The initial value of the primal variable x can be formulated as:

The step length. If the dual variables η , s , z L and z U exist, their step lengths are chosen as follows:

If x is bounded, the step sizes for x and λ are chosen to be equal as:

If x is unbounded but inequality constraints exist, the step lengths for x and λ are chosen the same as the step length for s . If x is unbounded and there are no inequality constraints, the step lengths for x and λ are chosen to be 1. In our experiments, additional line search procedures for step lengths, as described in Bellavia (1998), were not implemented in order to save computational time.

B Proof of Proposition 1

Proof. Step 1. Let δ > 0 , suppose that ( x k , λ k , z k ) ∈ N δ ( x ∗ , λ ∗ , z ∗ ) := { ( x, λ, z ) | ∥ ( x, λ, z ) -( x ∗ , λ ∗ , z ∗ ) ∥ < δ } . Define σ := min { σ k } and η := min { η k } , where σ k and η k are chosen to satisfy the update rule (11) in Bellavia (1998). Then from the Assumption 1:

By using the steplength α k obtained from the line search method of Bellavia (1998) and defining:

we obtain:

where Γ = 1+ η max 1 -η max , ( σ k + η k ) ∈ (0 , η max ) and η max ∈ (0 , 1) . Then, there exists a constant Γ independent of k such that (24) holds, whenever y k is bounded by (1 + σ k + η k ) ∥ F 0 ( x k , λ k , z k ) ∥ . Hence, from Theorem 3.5 of Eisenstat and Walker (1994), it follows that ( x k , λ k , z k ) → ( x ∗ , λ ∗ , z ∗ ) .

Step 2. In this step, we aim to prove that the step size α k of the inexact IPM is bounded away from 0. Given that Assumption 1 is satisfied, it follows that ∥ J k y k + F k ∥ and ∥ y k ∥ are both bounded. Then we assume the first two equations of F 0 ( x, λ, z ) are Lipschitz continuous gradient with constant L . The remaining proofs are consistent with Theorem 3.2 in Bellavia (1998).

Step 3. The line search method of Bellavia (1998) enables α k to satisfy:

where β ∈ (0 , 1) . Therefore {∥ F 0 ( x k , λ k , z k ) ∥} is decreasing and bounded, hence, it is convergent. Based on Assumption 1, (24) holds. Furthermore, we assume ∥ F 0 ( x k , λ k , z k ) ∥ ̸ = 0 and δ is chosen sufficiently small so that

is satisfied, whenever ( x 1 , λ 1 , z 1 ) , ( x 2 , λ 2 , z 2 ) ∈ N 2 δ ( x ∗ , λ ∗ , z ∗ ) . Define

then based on Lemma 5.1 of Eisenstat and Walker (1994), the loop of backtracking line search of Bellavia (1998) will terminate with

where θ ∈ (0 , 1) is a control parameter. This implies that the series ∞ ∑ k =0 (1 -ˆ η k ) is divergent. By applying (25) iteratively, we have

Since β > 0 and 1 -ˆ η j ≥ 0 , the divergence of series ∞ ∑ j =0 (1 -ˆ η j ) implies ∥ F 0 ( x k , λ k , z k ) ∥ → 0 .

C Datasets and Parameter setting

The key hyperparameters for each task, including K , T , and the hidden dimension of LSTM networks, are listed in Table 4. Generally, IPM-LSTM demonstrates improved performance with larger values of K and T , although this comes at the cost of increased computational time. Increasing T could enhance the quality of solutions to the linear system. In this study, K is maintained at the same value for both training and testing. Further analysis can be found in Appendix D.5.

D Experimental Results

D.1 Convex QPs

The performance on convex QPs, including 'Convex QPs (RHS)' and 'Convex QPs (ALL)', each instance with 100 variables, 50 inequality constraints, and 50 equality constraints, is shown in Table 5. From Table 5, IPM-LSTM provided the best objective value with acceptable constraint violations.

Also, IPM-LSTM achieved the most significant reduction in iterations when the returned primal-dual solution pair is utilized for warm-starting IPOPT .

D.2 Non-convex QPs

For non-convex QPs (8), all the non-zero elements without any special physical meaning, such as all zeros or all ones, are multiplied by a value generated from a uniform distribution in the range [0 . 8 , 1 . 2] . If the original elements are integers, we will perform an additional rounding operation. We use 'p' and 'r' to represent these operations, respectively, and use 'c' to denote the case without perturbation. The detailed perturbation rules for each instance are shown as:

D.3 Convex QCQPs

The performance on convex QPs, including 'Convex QCQPs (RHS)' and 'Convex QCQPs (ALL)', each instance with 100 variables, 50 inequality constraints, and 50 equality constraints, is shown in Table 7. From Table 7, IPM-LSTM provided the best objective value with acceptable constraint violations. Also, IPM-LSTM achieved the most significant reduction in iterations when the returned primal-dual solution pair is utilized for warm-starting IPOPT .

D.4 A Simple Non-convex Program

The performance on simple non-convex programs (30), including 'Non-convex Programs (RHS)' and 'Non-convex Programs (ALL)', each instance with 100/200 variables, 50/100 inequality constraints, and 50/100 equality constraints, is shown in Table 8. From this table, IPM-LSTM provided the best objective value with acceptable constraint violations. Also, IPM-LSTM achieved the most significant reduction in iterations when the returned primal-dual solution pair is utilized for warm-starting IPOPT .

D.5 Performance Analysis

D.5.1 The Number of LSTM Time Steps

The number of LSTM time steps T is a key hyperparameter which decides the performance-efficiency trade-off of IPM-LSTM . To illustrate this, we conduct experiments on a convex QP (RHS) problem

with 100 variables, 50 inequality constraints, and 50 equality constraints, investigating the quality of approximate solutions under different LSTM time step settings. As shown in Table 9, the IPM-LSTM with deeper LSTM architectures generally yields better approximate solutions (with lower objective values, smaller constraint violation and better warm-start performance) but with longer computational time. Specifically, the IPM-LSTM with 60 and 70 time steps achieves most significant reductions in warm-starting iteration count and total runtime, respectively. However, taking into account the end-to-end solution time, we have opted to set T to 50 in our experiments. At each IPM iteration, as the LSTM network depth increases, ∥ J k y k + F k ∥ decreases (see Figure 5(a)). This indicates an improvement in the quality of solutions to the linear systems. Furthermore, the corresponding IPM-LSTM converges faster (e.g., fewer IPM iterations) when the LSTM network becomes deeper (see Figure 4).

0

20

40

60

80

100

IPM iteration

-

15

-

10

-

5

0

5

10

15

Objective value

T

= 10

T

= 20

T

= 30

T

= 40

T

= 50

T

= 60

T

= 70

T

= 80

T

= 90

Optimal value

D.5.2 Linear System Solutions

In order to provide an more specific presentation of how accurately the LSTM performs, we report the detailed values of Figure 3(a) in Table 10. We can conclude that, ∥ J k y k + F k ∥ is roughly in the same order of magnitude as η [( z k ) ⊤ x k ] /n at each IPM iteration. To reveal the relationship between the error of the linear system solution ∥ J k y k + F k ∥ and the LSTM time steps, hidden dimensions, training sizes and test sizes, we conduct experiments on representative convex QP (RHS) problems with 100 variables, 50 inequality constraints, and 50 equality constraints, and the results are included in Figure 5.

In Figure 5(a), with the number of LSTM time steps increasing, ∥ J k y k + F k ∥ decreases.

In Figure 5(c), a larger training set is more beneficial for model training. The training set size in our experiment is 8,334, and the error in solving the linear system ∥ J k y k + F k ∥ is smaller compared with the case of 4,000 or 6,000 training samples.

In Figure 5(b), we consider LSTMs with 25, 50, 75, and 100 hidden dimensions and find that an LSTM with a hidden dimension of 50, as used in our manuscript, generally performs the best (e.g., the smallest ∥ J k y k + F k ∥ ).

As shown by Figure 5(d), the number of samples in the test set does not affect the performance of LSTM for solving linear systems.

0

20

40

60

80

100

IPM iteration

-

2

.

0

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

log(

∥

∥

J

k

y

k

+

F

k

∥

∥

)

T

= 10

T

= 20

T

= 30

T

= 40

T

= 50

T

= 60

T

= 70

T

= 80

T

= 90

(a) LSTM time steps

0

20

40

60

80

100

IPM iteration

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

log(

∥

∥

J

k

y

k

+

F

k

∥

∥

)

Hidden dimension 25

Hidden dimension 50

Hidden dimension 75

Hidden dimension 100

(b) Hidden dimension

0

20

40

60

80

100

IPM iteration

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

log(

∥

∥

J

k

y

k

+

F

k

∥

∥

)

Training size 4000

Training size 6000

Training size 8334

(c) Training size

0

20

40

60

80

100

IPM iteration

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

log(

∥

∥

J

k

y

k

+

F

k

∥

∥

)

Test size 400

Test size 833

Test size 1000

Test size 2000

(d) Test size

We take the log of the y-axis in Figure 3(a) and plot it in Figure 6(a). Roughly speaking, ∥ J k y k + F k ∥ is smaller than η [( z k ) ⊤ x k ] /n in the first 40 IPM iterations, while ∥ J k y k + F k ∥ surpasses η [( z k ) ⊤ x k ] /n in the later IPM iterations. We increase the number of LSTM time steps and report the computational results in Figure 6(b). From Figure 6, we can claim that with the number of LSTM time steps increasing, ∥ J k y k + F k ∥ becomes smaller.

0

20

40

60

80

100

IPM iteration

-

2

.

0

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

log(

∥

∥

J

k

y

k

+

F

k

∥

∥

)

log(

η

[

(

z

k

)

/latticetop

x

k

]

/n

)

(a) IPM-LSTM (100, 50)

0

20

40

60

80

100

IPM iteration

-

2

.

5

-

2

.

0

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

log(

∥

∥

J

k

y

k

+

F

k

∥

∥

)

log(

η

[

(

z

k

)

/latticetop

x

k

]

/n

)

(b) IPM-LSTM (100, 90)

D.5.3 Condition Numbers

The LSTM approach for solving linear systems is negatively affected by their large condition numbers. To demonstrate this, we consider the least squares problem

We utilize a first-order method (e.g., steepest descent method) to minimize ϕ ( y ) and achieve a linear convergence rate (Nocedal and Wright, 1999), i.e.,

As we discussed in Section 3.3, since solving linear systems via LSTM networks emulates iterative first-order methods, thus the value of κ ( J k ) affects the performance of LSTM networks. However, LSTM networks can empirically achieve a faster convergence rate than traditional first-order algorithms when solving the same least squares problems as shown in the computational studies (Section 3.1) of Andrychowicz et al. (2016). In order to alleviate the effect of large condition numbers, as discussed in Section 3.3, we have employed preconditioning techniques. To illustrate its effect, we conduct experiments on the simple non-convex programs, and report κ ( J k ) and their values after preconditioning (in parantheses) across several IPM iterations (e.g., 1 st , 10 th , 20 th , 50 th , 100 th ) in Table 11. We can conclude that the condition numbers κ ( J k ) remain within reasonable magnitudes even during the later IPM iterations, and are significantly reduced after applying the preconditioning technique.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to Abstract and Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please refer to Section 3.2 and Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to Section 3.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please refer to Section 4.1 for data. Our code will be available to public once our work is accepted, as mentioned in Section 1.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to Section 4.1 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Please refer to Section 4.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research presented in this paper adheres to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper does not present any such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Please refer to Section 4.1.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve any crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve any crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces IPM-LSTM, an approach integrating Long Short-Term Memory (LSTM) neural networks with Interior Point Methods (IPMs) to solve Nonlinear Programs (NLPs). The key innovation lies in approximating solutions to linear systems within IPMs using LSTMs, aiming to accelerate the convergence of classic IPM solvers. This approach leverages the Learning to Optimize (L2O) paradigm, presenting a two-stage framework where LSTM-generated solutions are used to warm-start an IPM solver. The authors compare the IPM-LSTM against traditional solvers and recent L2O methods across various NLPs, including Quadratic Programs and Quadratically Constrained Quadratic Programs. The proposed method reportedly reduces iterations by up to 60% and solution time by up to 70%.\n\nSTRENGTHS:\n1. The integration of LSTM networks to approximate the solutions of linear systems in IPMs is novel.\n2. The paper provides theoretical insights into the convergence properties of the proposed method under specific assumptions, adding to its credibility and understanding.\n3. The paper provides a comprehensive empirical evaluation across several types of NLPs. The results demonstrate improvements over traditional methods in terms of iteration count and computational time, which supports the effectiveness of the proposed method.\n\nWEAKNESSES:\n1. The IPM-LSTM, which uses LSTM to iteratively solve linear systems at each IPM iteration, repeats K times and feeds it into the IPM solver as a warm-start point, needs more justification.\n   1. The decision to use the L2O approach for solving a least squares problem (problem (4)) is not adequately justified. According to Assumption 1, the approximated solution needs to be bounded and accurate enough to guarantee the convergence of the outer loop. From the theoretical side, it is unclear how hard it is to satisfy those conditions by L2O approaches. From the empirical side, as shown in Figure 3(a), the accuracy condition is not always satisfied. If assumption 1 can not be guaranteed, the convergence of approximated IPM can not be guaranteed according to Prop. 1. \n   2. The approach of using an approximated IPM solution instead of directly generating a warm-start point using either NN or L2O raises questions about efficiency and effectiveness. Previous works [1-4] have shown that direct prediction of warm-start points can be more straightforward and computationally efficient. Besides, the convergence of approximated IPM can not be guaranteed, which also raises concerns about the quality of such a warm-start point. The authors need to provide more discussion or experimental comparisons to justify their more complex, iterative approximation method.\n\n[1] R. Sambharya, G. Hall, B. Amos, and B. Stellato, \"End-to-End Learning to Warm-Start for Real-Time Quadratic Optimization\", arXiv preprint arXiv:2212.08260, 2022.\n\n[2] Sambharya R, Hall G, Amos B, et al. Learning to warm-start fixed-point optimization algorithms[J]. Journal of Machine Learning Research, 2024, 25(166): 1-46.\n\n[3] F. Diehl, \"Warm-Starting AC Optimal Power Flow with Graph Neural Networks\", in Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS) Workshop, Vancouver, BC, Canada, Dec. 8 - 14, 2019.\n\n[4] K. Baker, \"Learning Warm-Start Points for AC Optimal Power Flow\", in Proceedings of IEEE 29th Machine Learning for Signal Processing Conference, Pittsburgh, PA, USA, Oct. 13 - 16, 2019.\n\nQUESTIONS:\n1. In the experiments, why not directly use the simple NN prediction as a warm-start point for the IPM solver?\n2. In the simulation part, as the solutions obtained by the proposed IPM-LSTM approach are used as a warm-start point to IPM, why are the equality constraints not satisfied? The authors are suggested to explain the experimental setup clearly.\n3. In Table I, the proposed IPM-LSTM approach has a larger constraint violation with a longer solution time as compared to the OSQP algorithm. Does this mean the proposed approach can not exceed state-of-the-art methods for solving convex QP problems?\n\nLIMITATIONS:\nThe authors have discussed the limitation in Sec. 5.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method called IPM-LSTM, which integrates machine learning techniques into interior point methods (IPM). Specifically, the authors propose training a RNN model, LSTM, to quickly approximate the solution of linear systems within IPM. This approach is numerically validated on several convex and nonconvex QPs.\n\nSTRENGTHS:\n- The topic is important as IPM plays a crucial role in solving linear and nonlinear programs, which have extensive applications in scientific computing.\n- The idea of accelerating a subroutine of IPM, rather than applying deep neural nets end-to-end to solve optimization problems, is insightful. The convergence of IPM is typically fast (superlinear convergence), leaving little room for improvement. However, solving the linear system in IPM is usually a computational bottleneck, making it worthwhile to accelerate with machine learning.\n- The experimental results are promising. IPM-LSTM clearly outperforms other learning-based baseline methods in terms of the objective function.\n\nWEAKNESSES:\nMy main concern about this paper is Assumption 1, which requires the accuracy of the linear system solution to increase as the number of iterations $k$ increases. Based on this assumption, exact convergence is derived, as shown in Proposition 1. However, the empirical results in Section 4 indicate that IPM-LSTM does not achieve exact optimality, revealing a gap between theory and practice. To address this gap, I suggest:\n- Reporting the error of LSTM at each iteration. This would provide readers with an understanding of how accurately the LSTM performs. Additionally, it would be beneficial to report the relationship between the error of the linear system solution with the size, training, and testing overhead of the LSTM.\n- Using a log scale y-axis for Figure 3a for better precision. For example, 0.01 is 10 times greater than 0.001, but this difference is not reflected in the linear scale of Figure 3a.\n- Modifying Assumption 1 to better align with practice. For example, assuming a fixed error on the right-hand sides of equations (5) and (6). Based on this relaxed assumption, a result similar to Proposition 1 could be derived, but with a fixed error on the limit of $(x^k,\\lambda^k,z^k)$. The relationship between the allowed error in Assumption 1 and the propagated eventual error in Proposition 1 would sufficiently describe the performance of IPM-LSTM.\n\nQUESTIONS:\nRefer to \"Weaknesses\".\n\nLIMITATIONS:\nThis theoretical paper appears to have no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed to replace the linear system solver used in the inner loop of interior point method (IPM) with an LSTM for solving general non-linear programs. The LSTM is trained in a unsupervised manner to minimize a unconstrained least square objective derived from the KKT conditions. The proposed framework, IPM-LSTM, can be used in an end-to-end way, or to warm-start IPM so that the number of overall outer IPM iterations can be reduces and thus the solving time decreases. The authors theoretically proved that, as long as the trained LSTM achieves certain level of accuracy, the IPM will converge. The authors conducted empirical experiments on a variety of non-linear programs to verify the effectiveness of IPM-LSTM and that it approximately satisfied the assumption in the theoretical analysis.\n\nSTRENGTHS:\n1. The idea of plugging in L2O methods to approximate a single step in a bigger optimization framework is interesting and promising.\n\n2. The proposed architecture is concise and effective to some extent. The authors also provided theoretical analysis to support their method.\n\n3. To some extent, the presented empirical resutls are promising.s\n\nWEAKNESSES:\nThere are three major issues that undermine the robustness of this work:\n\n1. The linear system solver used for IPOPT is unclear, which significantly impacts the solver's overall performance. While MUMPS is the default, HSL typically performs better, achieving 2-3 times faster results. This performance surpasses that of IPOPT when warm-started with the IPM-LSTM solution.\n\n2. There is no analysis of how the performance changes with varying numbers of iterations in the inner LSTM, which decides the performance-efficiency trade-off of IPM-LSTM. Longer inner iterations (deeper LSTM) may improve the quality of the solutions but increases computation costs. However, improved solution quality may also decrease the number of outer iterations needed and thus reduce the overall solving time. Moreover, increased LSTM depth can make the training process more difficult. Overall, thise is tricky part and should be empirically investigated more carefully.\n\n3. Although LSTM benefits from batched processing—a potential strength of IPM-LSTM accelerated by GPU—this work fails to provide empirical evidence supporting this property.\n\nQUESTIONS:\nSee the weaknesses section.\n\nLIMITATIONS:\nSee the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}