IPM-LSTM: A Learning-Based Interior Point Method for Solving Nonlinear Programs
Authors: Xi Gao
Jinxin Xiong
Akang Wang
Qihong Duan
Jiang Xue
Qingjiang Shi
Venue: NeurIPS 2024 poster (2024)
Keywords: learning to optimize
interior point method
nonlinear constrained programs
Primary area: optimization

Abstract: Solving constrained nonlinear programs (NLPs) is of great importance in various domains such as power systems, robotics, and wireless communication networks. One widely used approach for addressing NLPs is the interior point method (IPM). The most computationally expensive procedure in IPMs is to solve systems of linear equations via matrix factorization. Recently, machine learning techniques have been adopted to expedite classic optimization algorithms. In this work, we propose using Long Short-Term Memory (LSTM) neural networks to approximate the solution of linear systems and integrate this approximating step into an IPM. The resulting approximate NLP solution is then utilized to warm-start an interior point solver. Experiments on various types of NLPs, including Quadratic Programs and Quadratically Constrained Quadratic Programs, show that our approach can significantly accelerate NLP solving, reducing iterations by up to 60% and solution time by up to 70% compared to the default solver.

IPM-LSTM: A Learning-Based Interior Point Method for Solving Nonlinear Programs
Xi Gao Jinxin Xiong?., Akang Wang23", Qihong Duan', Jiang Xue and Qingjiang Shi
"School of Mathematics and Statistics, Xi'an Jiaotong University, Xi Shennzhnn Research Institute of Big Data, China
Xi'an, an, China
3 School of Data Science, The Chinese University of Hong Kong, Shenzhen, China Scchool of Software Engineering, Tongji University, Shanghai, China
Abstract
Solving constrained nonlinear programs (NLPs) is of great importance in various domains such as power systems, robotics, and wireless communication networks. One widely used approach for addressing NLPs the interior point method (IPM). The most computationally expensive procedure in IPMs is to solve systems of linear equations via matrix factorization. Recently, machine learning techniques have been adopted to expedite classic optimization algorithms. In this work, we propose using Long Short-Tern Memory (LSTM) neural networks to approximate the solution of linear systems and integrate this approximating step into an IPM. The resulting approximate NLP solution then utilized to warm-start an interior point solver. Experiments on various types of NLPs, including Quadratic Programs and Quadratically Constrained Quadratic Programs, show that our approach can significantly accelerate NLP solving, reducing iterations by up to 60% and solution time by up to 70% compared to the default solver.
1 Introduction
Constrained Nonlinear Programs (NLPs) represent a category of mathematical optimization problems in which the objective function, constraints, both, exhibit nonlinearity. Popular NLP variants encompass Quadratic Programs (QPs), Quadratically Constrained Quadratic Programs (QCQPs), semi-dcfinite programs, among others. These programs are commonly classified convex or non-convex, contingent upon the characteristics of their objective function and constraints. The versatility of NLPs allows for their application across wide array of domains, including power sys- tems (Conejo and Baringo 2018), robotics (Schaal and Atkeson 2010), and wireless communication networks (Chiang 2009).
The primal-dual Interior Point Method (IPM) stands as a preeminent algorithm for addressing NLPS (Nesterov and Nemirovskij 1994 Nocedal and Wright 1999). It initiates with infeasible solution positioned sufficiently far from the boundary. Subsequently. at each iteration, the method refines the solution by solving system of linear equations, thereby directing it towards the optimal solution. Throughout this iterative process, the algorithm progresses towards feasibility and optimality while keeping the iterate well-centered, ultimately converging the optimal solution. However, notable computational bottleneck arises during the process of solving linear systems, necessitating matrix decomposition with runtime complexity of O(n²).
Recently, the Learning to Optimize (L20) (Bengio et al., 2021 Chen et al., 2024 Gasse et al.) 2022) paradigm has emerged as promising methodology for tackling various optimization problems,
Corresponding authors: Akang Wang Srrangakang@ssibd.enn,,, Jiang Xue <x.jiang@ @xjiengnuuueeu.ccn>
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

IPM-LSTM
min f(z) rER
s.t. h(xx)=
LSTMcell
LSTMcell
x 2 0,
Opt. sot
IPOPT
min J'y p412
LSTM
new Itorate
Figure 1: An illustration of the IPM-LSTM approach.
spanning unconstrained optimization (Chen et al. 2022a linear optimization (Chen al.
2022b Li
et al. 2024), and combinatorial optimization (Baker 2019 Gasse et al. 2022 Han et al. 2023 h. Is
ability encapsulate common optimization patterns renders it particularly appealing. Noteworthy
the application of learning techniques to augment traditional algorithms such as the gradient descent
method (Andrychowicz et al. 2016). simplex method (Liu et al. 2024). and IPM (Qian et al.) 2024).
We observe that existing works on tearning-based IPMs primarily concentrate on solving LPs (Qian eet al. 2024).. Motivated by the robustness and efficiency of IPMs for general NLPs, pose the following question:
Can we leverage 1.20 techniques to expedite IPMs for NLPs?
In this study, we propose the integration of Long Short- Tern Memory (LSTM) neural networks to address the crucial task of solving systems of linear equations within IPMs, introducing novel approach named IPM-LSTM. An illustration of the IPM-LSTM approach is depicted in Figure T Specifically, we substitute the conventional method of solving linear systems with an unconstrained optimization problem, leveraging LSTM networks to identify near-optimal solutions for the latter. We integrate a fixed number of IPM iterations into the LSTM loss function and train these networks within the self-supervised learning framework. The substitution is embedded within classic IPM generate search directions. Ideally, the primal-dual solution provided by IPM-LSTM should be well-centered with respect to the boundary and associated with a small duality gap. Finally, we utilize such approximate primal-dual solution pairs to warm-start an interior point solver. IPM-LSTM has several attractive features: (i) it can be applied to general NLPs; (if) it strikes good balance between feasibility and optimality in the returned solutions; (iii can warm-stan and thereby accelerate interior point solvers.
The distinct contributions of our work be summarized as follows:
Approximating Solutions to Linear Systems via LSTM: This study marks the first attempt employ learning techniques for approximating solutions of linear systems in IPMs, achieving significant speedup compared to traditional linear algebra approaches.
We introduce a two-stage L20 framework. In the first stage, IPM- LSTM generates high-quality primal-dual solutions. In the second stage. these solutions are used to warm-start an interior point solver. This framework effectively accelcrates the solving process of IPMs while yielding optimal solutions.
Two-Stage Framework:
Empirical Results: Compared with existing L20 algorithms. IPM-LSTM demonstrates fa- vorable performance in terms solution feasibility and optimality across various NLP types, including QPs and QCQPs. Utilizing these solutions as initial points in the state-of-the-art NL. solver IPOPT (Wächter and Biegler 2006) reduces iterations by up to 60% and solution time by up to 70%.

---

2 Related Works
Constrained L20. Approaches utilizzng L20 for constrained optimization can be broadly categorized into two directions: (i) direct learning of the mapping from optimization inputs to full solutions. and (ii) integration of learning techniques alongside within optimization algorithms (Bengio
ict al. 202I Donti et al. 202T). Previous works (Fioretto et al.. 2020 Huang et al.) 202T] Pan et al..
2023 adopted the former approach, employing supervised learning scheme to train the mapping.
However, this method necessitates a large number of (near-) Joptimal solutions as training samples, making it eesource-intensive From self-supervised learning perspective, an intuitive approach is to incorporate the objective and penalization for constraint violation directly into the loss function (Kim) al. 2023 Park and Van Hentenryck] (023). Nevertheless, such approach may not guarantee the feasibility of the returned solutions. To address the feasibility issue, notable works such as Donti al. (202T) first predict partial solution via neural networks and then complete the full solution by utilizing equality constraints, iteratively correcting the solution towards the satisfaction of inequalities by applying gradient-based methods. However, for general nonlinear inequalities, this correction step may not ensure feasibility (Liang et al., 2023). Other approaches like Li ct al. (2023) utilized gauge mappings to enforce feasibility for linear inequalities, while Liang et al. (2023) proposed the homeomorphic projection scheme to guarantee feasibility. However, these methods have limitations; the former only applicable to linearly constrained problems, and the latter works for problems with feasibility regions homeomorphic to unit ball. Another critical issue with the approach in DDnti jet al. (021) is that the completion step may fail during training when the equality system with some fixed variables becomes infeasible, as highlighted in Han et al. (2024) and Zeng et al. (2024). Consequently, such an approach will not succeed during the training stage. To mitigate this issue, Han et al. (2024) proposed solving projection problem if the completion step fails. However, the computationally expensive projection step may still be necessary during inference, which hinders its practical value.
Learning-Baze -Bate IPMs. Primal-dual IPMs are polynomial-time algorithms used for solving con- strained optimization problems such as LPs and NLPs. The work of Qian et al.) (2024) demonstrated that properly designed Graph Neural Networks (GNNs) can theoretically align with IPMs for LPs, enabling GNNs to function lightweight proxies for solving LPs. However, extending this alignment to NLPs is challenging as representation learning for general NLPs remains unknown. Another avenue of research in learning-based IPMs involves warm-starting implementation. Previous works like Baker (2019), Diehl (2019) and Zhang and Zhang (2022) addressed alternative current optimal power flow (ACOPF) applications and proposed using learning models, such as GNNs, to learn the mapping between ACOPF and its optimal solutions. These predicted solutions are then utilized as initial points to warm-start an interior point optimizer. However, even if these solutions the optimal ones, they may not be well-centered with respect to the trajectory in IPMs, causing the optimizer to struggle in progressing towards feasibility and optimality (Forsgren 2006).
close to
3 Approach
3.1 The Classic IPM
We focus on solving the following NLP (I)
min
f(x)
zER"
s.l.
h(x)= 0
(I)
x 2 0
where the functions f IR" + IR and h Ra + RM are all assumed to be twice continuously differentiable. Problems with general nonlinear inequality constraints can be reformulated in the above form by introducing slack variables. We note that for simplicity, we assume all variables in (I) are nen-negative, though NLPs with arbitrary variable bounds can also be handled effectively. Readers are referred Appendix for details.
The primal-dual IPM stands as one of the most widely utilized approaches for addressing NIPss. entails iteratively solving the perturbed Karush-Kuul Tucker (KKT) conditions (2) for a decreasing
3

---

sequence of parameters converging to zero.
()()) Vh(x)
h(x)) 2,
(2)
diag(z)diagl (r) =
where À E IR and = E I denote the corresponding dual variables, diag(-) represents diagonal
matrix, and is a vector ones. Let (xx X, 2) denote the system of nonlinear equations in 2 We then employ one-step Newton's 's method solve such system, aiming to solve systems of
linear equations
3
fir()) A (x)
-1
A
Vh(x)
AA Az
= -F(x,,,2
diag(z)
diag(2)
(3)
The IPM commences with an initial solution (z 20, 20)20) such that x², 20 > 0. At iteration k the linear system (3) defined by the current iterate (a* x² 2*) is solved, with μ == [<**)==*) /n being the perturbation parameter and constant (0, 1). A line-search filter step along the direction (Ar*, A Az*) is then performed ensure boundary condition satisfaction well as sufficient progress towards objective value improvement or constraint violation reduction. This process iterates until convergence criteria, such achieving optimality and feasibility within specified tolerances, met. The IPM is guaranteed to converge to a KKT point with a superlinear rate. pseudocode of the IPM is presented Algorithm
Algorithm The classic IPM
Inputs: An initial solution (wx Ao 2²), (0,1), Outputs: The optimal solution (a" A 2²)
while not converged do
2: Update μ²²
Solve the system Jk [( (AA*)), (A=*) =
4: Choose a via line-search filter method
5:
(x*+1, A++1 22+1)) + (x A M,,²*) + a (Az AA* Azt)
6:
k + +++1
7:
end while
We note that, in classic IPMs, one typically reformulates the system (3) and then solves reduced system of equations (i.e., augmented system) for greater efficiency. However, in this work. are interested in the full systems since they are associated with smaller condition numbers (Greif al. 2014) that critical the performance of our proposed approach. Additionally, various techniques have been proposed to enhance the robustness and efficiency of IPMs, including second- order correction, inertial correction, and feasibility restoration. Interested readers are referred to Wächter and Biegler (2006) for further details about IPMs.
3.2 Approximating Solutions to Linear Systems
The small number of iterations in IPMs does not always guarantee efficiency because, at times, IPMs encounter a high per-iteration cost of linear algebra operations. In the worst-case scenario, the cost of solving dense optimization problem using direct linear algebra method solve the Newton equation system 33) may reach On2²) flops per iteration. This motivates us to avoid computing exact solutions to linear systems and instead focus their approximations. Toward this goal, we consider the following least squares problem 44):
|| + F*1]
min 3
2
where ||=|| denotes the Euclidean norm. If is also the exact solution [(A+*)T, (AA*)T [)=+)T]
3 is solvable, then an optimal solution to problem
TJ to system (3). Otherwise, we resort to an
approximation of the latter. This perspective similar to the inexacT7PM (Bellavia) 1998 Dexter al. 2022).

---

Assumption 1. At iteration k, we could identify some y such that
|| + Fa S (= //n 11y S (1 + Fo(x", X² 211
(5)
(6)
where 1) E (0, 1) and Fo(x", X x denotes F( A x²) with 0.
To satisfy Assumption the approximate solution sfth has to be bounded and accurate enough, regardless of whether J invertible.
Proposition (Bellavia (1998)). (x*, ,,,,,,",, "²* is generated such that Assumption is satisfied, let (x", X", =²) denote limit point of the sequence {(x", A ,²,**) )), then {(* >², x*)] converges to (x",, A 2²) and Folx", x*; =0 =0.
Proposition I limplies that if solutions with specified accuracy for linear systems in Step 3 are found, the IPM would converge.
3.3 The IPM-LSTM Approach
The problem 4 is unconstrained convex optimization problem. Various L20 methods have been proposed to solve such problems (Chen et al. [202aa] Gregor and LeCun 2010 Liu et al.] (2023). We will employ the LSTM networks in our L20 method for addressing problem hence our approach called "IPM-LSTM"..
Model Architecture, LSTM is type of recurrent neural network designed to effectively capture and maintain long-term dependencies in sequential data (Yu et al. 2019). LSTM networks are commonly considered suitable for solving unconstrained optimization problems due to the resemblance between LSTM recurrent calculations and iterative algorithms (Andrychowiez et al. 2016 Liu al. 2023
Lv al. 2017).
The LSTM network consists of T cells parameterized by the same learnable parameters 8. Each cell can be viewed as one iteration of a traditional iterative method, as illustrated in Figure Let (vv) J|Jby pk] for convenience. The 1-th cell takes the previous estimate yt-1 and the gradient (J+) (J + pk) as the input and outputs the current estimate ye:
y1 = LSTMa ([vc-1, (J%) yJ F*)])
The T-th cell yields yr as an approximate solution to problem 4 As suggested by Andrychowicz
et al. (2016) and Liu et al. (2023), utilize coordinate- wise LSTM that shares parameters not only across different LSTM cells but also for all coordinates of
grss
grad
grad..
LSTM Cell
LSTM Cell
Volac-s))
Vola--il
(u.. Vol(x)
Figure 2: The LSTM architecture for solving min d(y).
Model Training, We train the proposed optimizer by finding the optimal 0 in dataset M of NLPs. Each sample in M an instance of the optimization problem. During training, we apply the optimizer to each instance M M, performing K IPM iterations in the outer loop and I LSTM
5

---

time steps in the inner loop, generating a sequence of iterates [vi,.... "o.i,...y/.... vt),... ..... ".f""".".."............."""..... :.... y } where the superscript & denotes the IPM iteration number. We then optimize by minimizing the following loss function:
- MEM PLTZS
1 + ()
min
M
where the subscript M indicates that the corresponding term is associated with instance M.. Clearly, model training falls into the category of self-supervised learning. To mitigate memory issues caused by excessively large computational graphs, we employ truncated backpropagation through time after each IPM iteration during training, done in Chen et al. (2022a) and Liu et al. (2023).
Preconditioning. The Hessian matrix of ((y) is (JK)TJ², Jh, whose condition number, x((J")TT*)
is the square of that of J Consequently, s((Jk) Jd) can easily become very large. Since solving system (4) via LSTM networks emulates iterative first-order methods, the value of x((J*) J*)
strongly affects the performance of LSTM networks. To address this issue, we employ a simple diagonal preconditioning technique that rescales the Hessian matrix (J*)* Jk using the Ruiz scaling method Ruiz 2001 to decrease its condition number.
3.4 Two-Stage Framework
To further enhance the solution quality, we propose a two-stage framework that initially obtains a near- optimal and well-centered primal-dual solution via IPM-LSTM and then utilizes this approximate solution to warm-start an interior point solver. In this study, we select IPOPT (Wächter and Biegler 2006), an IPM-based solver renowned for its robustness and efficiency in optimizing NLPs.
Our two-stage framework works as follows: Given an NLP ( we generate an initial point (I.), 2 20 and formulate the least squares problem 4 Subsequently, the trained LSTM network with T cells solves this problem and returns a search direction. We then employ the simple fractional-to-ouundary method (Wächter and Biegler 2006) to determine the step size and reach the new iterate. This procedure is iterated K times, resulting in primal-dur dual solution (x A 2²x). Finally, the obtained solution serves the warm-start solution for IPOPT, leading to the optimal solution x" upon IPOPT convergence.
Experiments
4.1
Experimental Settings
We evaluate our approach and compare its performance against traditional methods as well as L20 algorithms for solving various types of NLPs. Furthermore, we also quantify the warm-starting effect of our proposed two-stage approach. Our code is available at hutns://giitubcccom//eesss//pppp
LSTM
Baseline Algorithms. In our experiments, we denote our algorithm by IPM-LSTM and com- pare it against both traditional optimizers and L20 algorithms. The traditional optimizers con- sidered are: (i) OSQP (Stellato et al., 2020 an ADMMd solver designed for convex QPs. (ii) IPOPT 3.14.8 (Wächter and Biegler 2006): a state-of-the art IPM-based solver for NLPs with the default linear solver MUMPS (Amestoy al. 2000) and convergence tolerance of 10-4, We also assess several L20 algorithms, including: (i) NN (Donti et al. 2021): straightforward deep learning approach that integrates the objective function and penalty constraint violations into the loss function. (ii) DC3 (Donti et al. (02T)) an end-to-end method that uses "completion" steps to maintain equality constraints and "correction" steps for incquality fcasibility. (iii) DeepLDE (Kim et al. 2023): algorithm that trains neural networks using primal-dual approach to impose incquality constraints and employs "completion" for equality constraints. (iv) PDL (Park and Van Hentenryck b023): self-supervised learning method that jointly trains two networks to approximate primal and dual solutions. (v) LOOP-LC (Li et al. 2023): a neural approximator that maps inputs of linearly constrained models to high-quality feasible solutions using gauge maps. (vi) H-Proj <Liang et al.) 2023): a method that applies a homeomorphic projection scheme to post-process solutions resulting from the completion step in DC3.
Datasets. The dataset used in this paper includes randomly generated benchmarks obtained from Chen and Burer 2012)) Donti al. (202 and Liang al. (2023), as well real-world instances

---

from Globaliib (see http://www.mmpiibborg. These benchmarks encompass QPs, QCQPs, and simplex non-convex programs. For each case, we generate 10, 000 samples and divide them into a 10: 1 ratio for training, validation, and testing, respectively. All numerical results are reported for the test set.
Model Settings. All LSTM networks have single layer and are trained using the Adam opti- mizer (Kingma 2 During IPM-LSTM training, an early stopping strategy with a patience of 50 is employed, halting training if no improvement is observed for 50 iterations, while satisfying inequality and equality constraints violation less than 0.005 and 0.01. The learning rate is 0.0001, and the batch size is 128 for each task. Additional IPM-LSTM parameters for each task are provided in Appendix(C
Evaluation Configuration. All our experiments were conducted on an NVIDIA RTX A6000 GPU, an Intel Xeon 2.10GHz CPU, using Python 3.10.0 and PyTorch 1.13.1.
4.2 Computational Results
Convex QPs. We consider convex QPs with both equality and inequality constraints:
min R
= x Qn + Po
2
s.t.
Pj S 4j
j = 1,.., ,1 j= = 1+ 1,11. 1,...,
(8)
p, 9)
a x; S 1.!
i= 1,, ,n
where Qo E ST. Pj E R", 9J E at E (-0) and x. {RU We conduct experiments on two groups of QPs, each instance with 200 variables, 100 inequalities and 100 equalities. The first group is generated in the same way as Donti et al. (202T), where only the right hand sides of equality constraints are perturbed while the second one considers perturbation for all model parameters. Let "Convex QP (RHS)" denote the former and "Convex QPs (ALL)" denote the latter. It is noteworthy that, in line with Donti et al.] (202T), we also investigate the performance of IPM-LSTM and baseline algorithms on smaller-scale convex QPs. Interested readers are directed Appendix D for details.
Table 1: Computational results on convex QPs.
Method
End-to-Enl End
IPOPT (warm start)
Total Time (4)
Gain
Obj... -
Max incq.
ineq.
cq. 1
Mean eq.
(s)
Itc.
(s)
Convex QPs (RHS)
aSqe
-29.176
0.000
0.000
0.000
0.000
01.009
IPOPT
-29.176
0.000
0.000
0.000
0.000
0.642
12.5
-26.787
0.000
0.000
0.631
0.235
<0.001
10.5
0.560
0.560
16.0%/12.8%
DC3
-26.720
0.002
0.000
0.000
0.000
<0.001
10.2
0.535
0.535
18.4%//16.7
DeepLDD
-3.697
0.000
0.000
0.000
0.000
<0.001
12.5
0.648
0.648
0.0%//-0.9%
PDL
-28.559
0.421
0.122
0.024
0.000
<0.001
9.7
0.514
0.514
22.49
LOGP-
-28.512
0.000
0.000
0.000
<0.0
10.8
0.565
13.6%/112.8 0%
H-Proj
-23.157
0.000
0.000
0.000
0,000
<0.001
11.2
0.605
0.605
10.4%/5 8%%
IPM-18TM
-29.550
0.000
0.000
0.002
0.001
0.175
7.2
0.370
0.545
42.4%/15.11%
Convex (ALL)
DSQP
-33.183
0.000
0.000
0.000
0,000
0.009
IPOPT
-33.183
0.000
0.000
0,000
0.000
0.671
12.9
IPM-18TM
-32,600
0.000
0.003
8.3
0.621
38.7%
We ran IPM-LSTM and all baseline algorithms on the test set "Convex QPs (RHS)", and we reported their computational results, which were averaged across 833 instances. The results are presented in Table I We denote this experiment by "End-to-End" for convenience. The columns labeled "Max Ineq." Mean Ineq.". "Max Eq.", and "Mean Eq." denote the maximum and mean violations for inequalities and equalities, respectively. The columns "Obj." and "Time (s)" represent the final primal objective and the runtime in seconds. Both OSQP and IPOPT solved these instances to guaranteed optimality, with OSQP being significantly more efficient due to its specialization as QP-specific solver. All L20 baseline algorithms returned solutions very quickly. However, solutions generated by NN and PDL exhibited significant constraint violations. While the solutions from DC3 and DeepLDE were nearly feasible, they corresponded to inferior objective values. On the other hand, LOOP-LC produced feasible and near-optimal solutions, whereas H H-Proj generated feasible solutions but with
7

---

larger objective values. Solutions identified by IPM-LSTM showed mild constraint violations but yielded superior objective values very close the optimal values. Clearly, IPM-LSTM effectively balances feasibility and optimality in the returned solutions. This comes at the cost of longer runtime compared to the 1.20 baseline algorithms, as the baselines typically employ simple multi-layer perceptrons, whereas IPM-LSTM utilizes neural network with several dozen LSTM cells.
Since IPM-LSTM is designed to provide interior point optimizers with high-quality initial points, we fed the returned primal-dual solution pair to IPOPT and reported the performance in Table For comparison, we also provided IPOPT with initial points generated from other L.20 algorithms. The columns labeled "Ite." and "Time (s)" under "IPOPT (warm-start)" indicate the number of iterations and solver time, respectively, while the column "Total Time (s)" represents the cumulative time for running both the L20 algorithms and IPOPT. The final column. "Gain (Ite/Time)" shows the reduction in iteration number and solution time, with the default IPOPT iteration number listed in the "Ite." column for reference. When initial solutions from IPM-LSTM and most 1,20 baseline algorithms (except DeepLDE) were used, IPOPT converged with fewer iterations and reduced solution time. Notably, IPM-LSTM achieved the most significant reduction in iterations, from 12.5 to 7.2, and decreased the average solver time from 0.642 seconds to 0.37 seconds. Including the computational time for IPM-LSTM, the total runtime was 0.545 seconds, reflecting 15.1 % reduction in time. It is worth noting that while IPM-LSTM did not yield the maximum solution time reduction, this was due to its relatively high computational expense.
To our knowledge, there is no existing representation learning approach for general convex QPs. Since the aforementioned 1.20 baseline algorithms depend on specific representations of QPs, they are not applicable to "Convex QPs (ALL.). Therefore, we only provide results for OSQP, IPOPT, and IPM-LSTM. We also report computational results averaged across 833 instances in the test set "Convex QPs (ALL)", presented in Table []] The results demonstrate that IPM-LSTM can identify high-quality solutions for general convex QPs, and using these solutions initial points can reduce iterations by 35.7% and solution time by 7.5%.
Convex QCQPs. We now turn to convex QCQPs with both equaltity and inequality constraints:
min *ER"
2 x Qu + Po
s.t.
x Q x + p3 x 9
j 1, .... ,1 j = + mm i 1, ,n..,
P2 x x=
o 5 S x.
where Q2 E ST. E ER" E R. x Ruf-oo and x. E [oo)). Similar to our experiments on convex QPs, we also consider two groups of convex QCQPs, each with 200 variables, 100 inequality constraints, and 100 equality constraints. The first group (denoted as "Convex QCQPs (RHS)") is generated as descrihed in Liang al. (002)), with perturbations only to the right-hand sides of the equality constraints. The second group (denoted as "Convex QCQPs (ALL)") considers perturbations all parameters. We also refer readers Appendix D for computational experiments on smaller-sized convex QCQPs.
We omit OSQP and LOOP since the former cannot handle QCQPs, while the latter is only applicable to linearly constrained problems. We evaluate IPM-LSTM and compare it against the remaining baseline algorithms. The computational results reported in Table Again, solutions produced by NN and PDL exhibit significant constraint violations, while those from DC3 and H-Proj are of high quality in terms of feasibility and optimality. Once more, the solutions produced by DeepLDE were deemed feasible but exhibited inferior objective values. Conversely, our approach, IPM-LSTM, produced solutions with superior objective values albeit with minor infeasibility. Compared to the baseline algorithms, utilizing solutions from IPM-LSTM to warm-start IPOPT resulted in the most substantial reduction in iterations.
As the aforementioned L20 baseline algorithms are not applicable to "Convex QCQPs (ALL)", we only report computational results for IPOPT and IPM-LSTM in Table The IPM-LSTM approach produced high-quality approximate solutions to convex QCQPs, and warm-starting IPOPT with these solutions accelerated IPOPT by 11.4%, with 33.1 1 1 reduction in iterations.
Non-convex QPs. We now consider non-convex QPs of exactly the same form as K but with Qo being indefinite. We take 8 representative non-convex QPs from the datasets Globallib and
8

---

Table 2: Computational results on convex QCQPs.
Method
End-to-End
IPOPT (warm start)
Tocal Time (3)
Gain
nie/Time)
Ob
Max ince...
Mean ineq.
cq. L
eq.
Time (8)
Itc.
Time
Convex QCQPs (RIIS)
IPOPT
-39.162
0.000
0.000
0,000
0.000
1,098
12.5
-2,105
0.0000
0.552
0.169
<0.001
12.1
1,311
1.24%/-1944%
DC3
-35.741
0.000
0.000
0.000
0,000
0.005
9.6
1,051
1,051
20.7%% 8%
DeepLDE
-15.132
0,000
0.000
0,000
0.000
<0.001
11.5
1.222
1.222
8.0%%/-1..2%
POL.
0.005
0.05
8.9
1.013
28.88%/7.77%
E-Proj
-36.062
0.000
0.000
0.000
0,000
<0.0011
9.8
1,070
1,090
21 66%66 66%
IPM-19TX
-38.540
0,000
0.000
0.004
0.001
0,205
8.0
0.825
1,030
%6/6./
Convex QCQPs (ALL)
IPOPT
-39.868
0,000
0.000
0,000
0.000
0.801
12.4
1PM-L9TX
-38.405
0.004
0.001
0,000
8.3
0.710
33.1%/114%%
RandQP (Chen and Burer 2012), each with up 50 variables and 20 constraints, and perturb the relevant parameters for instance generation. Details can be found in Appendix D.2
Among all the baselines, only IPOPT is applicable to these general non-convex QPs. Hence, we report computational results for IPOPT and IPM-LSTM in Table IPOPT solved these instances to local optimality, while IPM-LSTM identified high-quality approximate solutions very efficiently. Using these primal-dual approximations to warm-start IPOPT resulted in an iteration reduction of up to 63.9% and solution time reduction of up to 70.5%.
Table 3: Computational results on non-convex QPs.
IPOPT
IPM-LSTH
IPOPT (warm start)
Total
Gain
Instance
Obj.
Itc.
Time (6)
Obj.
Max Vio.
Time (5)
Obj.
Itc.
Time (6)
Time
(Ite./ (Tme)
qpl
0.001
52.0
0,707
0.045
0.008
0.017
0.001
42.0
0.559
0.576
19: 19.2%
qp2
0.001
69.0
0.034
0.008
0.001
40.0
0.347
0.376
42.0%%
MLrv
-S.4400
215.0
0.955
-34.663
0.000
0,009
-58.867
168.0
0.626
0.635
21.9 91/99/33355
-67.083
-30.955
0,000
-67.083
120.5
0.482
0.494
36.8%/38..19
sLN3
0.000
55.0
0,781
0.818
0.000
0.017
0.000
47.0
0.616
0.634
14.5%%/18.8%
-132.019
449.0
2.445
-61.428
0.000
0,016
-131.756
162.0
0.705
0.721
63.9%
Stn9
-126.945
655.0 0
5.457
-58.415
0.000
0.036
-127.652
408.0
1,830
1.856
37.7%//4..88
9p30_15_11 1__15_111
37.767
16.0
0.198
37,787
0.002
0.021
37.767
9.0
0.083
0. 104
43.7%/47.55
Max Vio, denotes the muximum consimint vidlation.
(202 ). we consider a set of simple non-convex programs where the linear objective term in (8) is substituted with pā sin (x). These instances were generated using the same methodology as Donti al. (202T). We subject these
Simple non-convex programs. Following the approach outlined in Donti et al.
problems to evaluation using both IPM-LSTM and baseline algorithms. Once again, the computational results underscore the high-quality solutions obtained by IPM-LSTM and its superior warm-starting capability. Detailed results are provided in Appendix D.4
4.3 Performance Analysis of IPM-LSTM
In Assumption T we posit that the linear systems 3 can be solved with an acceptable residual (Condition (5) and that the solutions properly bounded (Condition (6)) Although the LSTM network cannot guarantee the satisfaction of these conditions, we empirically assess the validity of Assumption in IPM-LSTM. We plot the progress of J + F= (z*)T=² /n. || and [Fo(** x =*)
the IPM iterations increase in Figure ((a) and (b) Condition (5) is mostly satisfied except during the first few iterations, while Condition (6) is strictly satisfied across all IPM
iterations. To assess the precision of the approximate solutions as the LSTM time steps increase, we plot the progress of the residual for linear systems (3) in Figure 3() At IPM iteration k, the residual decreases monotonically towards the LSTM time steps increase, indicating that LSTM networks can produce high-qualing approximate solutions to system 33. The approximation quality improves with the number of IPM iterations. As shown in Figure 3(d) the primal objective decreases monotonically towards the optimal value with increasing IPM iterations, empirically demonstrating the convergence of IPM-LSTM. Note that the condition number x((J²) J*) becomes quite large in the later IPM iterations, which can adversely affect the performance of LSTM networks in obtaining approximations to systems (3). Consequently, we terminate IPM-LSTM after a finite number of iterations. Further analysis regarding the number of IPM iterations and LSTM time steps is

---

presented in Appendix D.5 Additionally, we include the performance of IPM-LSTM under various
hyperparameter settings Appendix D.5
(a) Condition 55)
(b) Condition (6)
(c) Residual
(d) Objective value
Figure 3: The performance analysis of IPM-LSTM on convex QP (RHS).
5 Limitations and Conclusions
In this paper, we present a learning-based IPM called IPM-LSTM. Specifically, we propose ap- proximating solutions of linear systems in IPMs by solving least square problems using trained LSTM networks. We demonstrate that IPMs with this approximation procedure still converge. The solutions returned by IPM-LSTM experiments on various types of NLPs, including general QPs and QCQPs, showcase the effectiveness of IPM-LSTM and its ability accelerate IPOPT. Although IPM-LSTM generates high-quality primal-dual solutions, it is relatively computationally expensive due to the utilization multi-cell ISTM networks. In future endeavors, aim investigate the efficacy of employing low-complexity neural networks to approximate solutions of linear systems within IPMs.
used to warm-start interior point optimizers. Our computational
Acknowledgments
This work was supported by the National Key R&D Program of China under grant 2022YFAL003900 Jinxin Xiong and Akang Wang also gratefully acknowledge support from the National Natural Science Foundation of China (Grant No. 12301416), the Shenzhen Science and Technology Program (Grant No. RCBS20211000933300211 the Guangdong Basic and Applied Basic Research Foundation (Grant No. 2024A1515010306) and Longgang District Special Funds for Science and Technology
Innovation (LGKCSDPT2023002)
References
Patrick R Amestoy, lain S Duff, Jean-Yves L'Excellent, Jacko Koster. Mumps: a general purpose distributed memory sparse solver. In International Workshop on Applied Parallel Computing, pages 121-130. Springer, 2000.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to Iearn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
Kyri Baker.
Learning warm-start points for ac optimal power flow. In 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1-6. IEEE, 2019.
Stefania Bellavia. Inexact interior-point method. Journal of Optimization Theory and Applications, 96:109-12,, 1998.
Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: a methodological tour d'horizon. European Journal of Operational Research, 290(2):405-421, 2021.
Jieqiu Chen and Samuel Burer. Globally solving nonconvex quadratic programming problems via completely positive programming. Mathematical Programming Computation, 4(1):33-52, 2012.
Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang. and Wotao Yin. Learning to optimize: A primer and benchmark. Journal of Machine Learning Research, 23(189):1559 2022a.
10

---

Xiaohan Chen, Jialin Liu, and Wotao Yin. Learning to optimize: A tutorial for continuous and mixed-integer optimization. Science China Mathematics, pages 1-72, 2024.
Ziang Chen, Jialin Liu, Xinshang Wang, and Wotao Yin. On representing linear programs by graph neural networks. In The Eleventh International Conference Learning Representations, 2022b.
Mung Chiang. Nonconvex optimization for communication networks. Advances in Applied Mathematics and Global Optimization: In Honor of Gilbert Strang, pages 137-196, 2009.
Antonio J Conejo and Luis Baringo. Power system operations, volume 14. Springer, 2018.
Gregory Dexter, Agniva Chowdhury, Haim Avron, and Petros Drineas. On the convergence of inexact predictor- corrector methods for linear programming. In International Conference Machine Learning, pages 5007-5508.. PMLR, 2022.
Frederik Dichl. Warm-starting ac optimal power flow with graph neural networks. In 33rd Conference Neural Information Processing Systems (NeurIPS 2019), pages 1-6, 2019.
Priya I. Donti, David Rolnick, and J Zico Kolter. De3: A learning method for optimization with hard constraints. In International Conference on Learning Representations, 2021.
Stanley C Eisenstat and Homer F Walker. Globally convergent inexact newton methods. SIAM Journal on Optimization, 4(2):393 422, 1994.
Ferdinando Fioretto, Terrence WK Mak. and Pascal Van Hentenryck. Predicting ac optimal power flows: Combining deep learning and lagrangian dual methods. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1330-637,20 2020.
Anders Forsgren. On warm starts for interior methods. System Modeling and Optimization: Proceedings of the 22nd IFIP TC7 Conference held from July 18-22, 2005, Turin, Italy 22, pages 51-66. Springer, 2006.
Maxime Gasse, Simon Bowly, Quentin Cappart, Jonas Charfreitag, Laurent Charlin, Didier Chételat, Antonia Chmicla, Justin Dumouchelle, Ambros Gleixner, Aleksandr M Kazachkov, et al. The machine learning for combinatorial optimization competition (ml4co): Results and insights. In NewrIPS 2021 competitions and demonstrations track, pages 220-231. PMLR, 2022.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference international conference machine learning, pages 399-406, 2010.
Chen Greif, Erin Moulding, and Dominique Orban. Bounds on eigenvalues of matrices arising from interior interior-point methods. SIAM Journal on Optimization, 24(1):49-83 83,, 2014.
Jiayu Han, Wei Wang, Chao Yang, Mengyang Niu, Cheng Yang, Lei Yan, and Zuyi Li. FRMNet: A Feasibility Restoration Mapping Deep Neural Network for AC Optimal Power Flow. IEEE Transactions Power Systems, pages 1-11, 2024. ISSN 0885-8950, 1558-061 5678. doi: 10.11099PNRSS.22233347735 URL https://ieeexplore ieee. wtgg/document/1411110
Qingyu Han, Linxin Yang, Qian Chen, Xiang Zhou, Dong Zhang, Akang Wang, Ruoyu Sun, and Xiaodong Luo. A gnn-guided predict-and-scarch framework for mixed-integer lincar programming. In The Eleventh International Conference on Learning Representations, 2023.
Wanjun Huang, Xiang Pan, Minghua Chen, and Steven Low. Deepopf-v: Solving ac-opf problems efficiently.
JEEE Transactions on Power Systems, 37( (1(1)88008803 2021.
Hongscok Kim et al. Self-supervised equality embedded deep lagrange dual for approximate constrained optimization. arXiv preprint arXiv:2306.06674 2023.
DP Kingma. Adam: method for stochastic optimization. In Int Conf Learn Represent, 2014.
Bingheng Li, Linxin Yang. Yupeng Chen, Senmiao Wang, Qian Chen, Haitao Mao, Yao Ma, Akang Wang, Tian Ding, Jiliang Tang, et al. Pdhg-unrolles learning-to- opttimiee method for large-scale linear programming. In Forty-first International Conference on Machine Learning, 2024.
Meiyi Li, Soheil Kolouri, and Javad Mohammadi, Learning to solve optimization problems with hard linear constraints. IEEE Access, 2023.
Enming Liang. Minghua Chen, and Steven Low. Low complexity homeomorphic projection to ensure neural- network solution feasibility for optimization over (non-) convex set. In Conference on Parsimony and Learning (Recent Sporlight Track), 2023.

---

Jialin Liu, Xiaohan Chen, Zhangyang Wang, Wotao Yin, and Qin Cai. Towards constituting mathematical structures for learning optimize. In International Conference Machine Learning, pages 21426-21449. PMLR, 2023.
Tianhao Liu, Shanwen Pu. Dongdong Ge, and Yinyu Ye. Learning pivot smart expert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 8073-808) 2024.
Kaifeng Lv, Shunhua Jiang, and Jian Li. Learning gradient descent: Better generalization and longer horizons. In International Conference on Machine Learning, pages 2247-2255. PMLR, 2017.
Yurii Nesterov and Arkadii Nemirovskii. Interior-point polynomial algorithms in convex programming. SIAM. 1994.
Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999.
Xiang Pan, Minghua Chen, Tianyu Zhao, and Steven H. Low. Deepopf: A feassbility-opptmmzzed deep neural network approach for optimal power flow problems. IEEE Systems Journal, 17(1):673-683, 2023. doi:
N0.1100///SST..2222220000
Seonho Park and Pascal Van Hentenryck. Self-supervised primal-dual learning for constrained optimization. In
Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 4052 4052-4060, 2023.
Chendi Qian, Didier Chételat, and Christopher Morris. Exploring the power of graph neural networks in solving linear optimization problems. International Conference on Artificial Intelligence and Statistics, pages 1432-1440. PMLR, 2024.
Daniel Ruiz. scaling algorithm to equilibrate both rows and columns norms in matrices. Technical report. CM-P00040415, 2001.
Stefan Schaal and Christopher Atkeson. Learning control in robotics. IEEE Robotics & Automation Magazine, 17(2):20-29, 2010.
B. Stellato, G. Banjac, P. Goulart, A. Bemporad, and S. Boyd. OSQP: an operator splitting solver for quadratic pro- grams. Mathematical Programming Computation, 12(4))637-672, 2020. doi: 10. 10.10077112333-00000001117 URL httph://doo....... 20. 0007/=122532-00--0017-999
Andreas Wächter and Lorenz T Biegler. On the implementation of an interior-point filter line-scarch algorithm for large-scale nonlinear programming. Muthematical programming, 106:25-57, 2006.
Yong Yu, Xiaosheng Si, Changhua Hu, and Jianxun Zhang. A review of recurrent neural networks: Lstm cells and network architectures. Neural computation, 3137):1235-1270, 2019.
Sihan Zeng, Youngdae Kim, Yuxuan Ren, and Kiback Kim. QCQP-Net: Reliably Learning Feasible Alternating Current Optimal Power Flow Solutions Under Constraints, January 2024. http:///aaa org/abs. 2401. 06820 arXiv:2401 .06820 [cs, math] version: 1.
Ling Zhang and Baosen Zhang. Learning to solve the optimal power flow via lagrangian approach. In 2022 North American Power Symposium (NAPS), pages 1-6, 2022. doi: 00..100NAPP5611500022000001101
12

---

Appendix
A Implementation Details
The general NLP considered in this paper can be formulated as:
min re
f(z),
s.t.
( + 8 = 0
h(x) =
(9)
8 $20
3; 2 a
i 1L E T
02 43
where f RRn - Th, 9 I - Ri and h R > T are twice continuously differentiable functions; mi is the slack variable corresponding to the inequality constraints; T of # -0) and 1 {i: [i::0 x # oo). The Lagrangian function defined as:
L n, 2, 20 20) = +( g(z)+
Σ (..-))))))
Σ
2 (a)
x2) (10)
EE
EE
where 1) TR mine and T. are the corresponding dual variables. The perturbed KKT conditions are given by:
Vf(x) + + + - 2 + =²==0
g(z) + +())+==00
diag(n) - pe 20,((z) =0 (())==()===1 2 2 2220 20
720,8 0,8
(11)
The nonlinear system extracted from the KKT conditions is represented as:
F(x, th M,,,,,,,, ")=
(12)
where
V/(x) + Vg(x) A VV(((x) z 2
g(r) +
diag (1) diag s)c μe
,,,
0
h(x)
diag(= Jiiag x 14)e μee.
diag( ddiag x)e p.c.
The Jacobian matrix of F(r, 4, A, 8, at 2) is:
La n. *4,")
vh(x)
Vg(z),
0.
Ha, (t)
0.
ding s).
0.
ding ())
0.
Vh(m).
diag(= 52.
0.
0.
diag (o
x
diag(=" ".
0.
0,
0,
0.
ding(a
x)
where
Mirro
migg
Σ
T
V
(x)
Σ
M (
(13)
For convenience, we define the updates of primal and dual variables as:
Ar
An
AA
(14)
As
Az4
A
13

---

Then the linear system obtained from one-step Newton's method is shown as:
J (x. 2,,,,,,,222 = y= = - (I,, 77, A,,,,,,, 8,2",²²)
(15)
Initial points. If dual variables n, s, zh. and z exist, their initial values are set to 1. If the dual variable À exists, its initial value is set to 0. The initial value of the primal variable can be formulated as:
2 11, 1,
i E Te ie TL iE 14 n otherwise.
32
(at + 24 1/2, 0,
(16)
The step length. If the dual variables 8, xt and z0 exist, their step lengths are chosen as follows:
a" sup {a (0, 1] "/1 aÂm 2 0, 1, Mineg
(17)
a = sup fa E (0, 1] 8; + s; 2 0, = 1, .... Miney
(18)
a² = sup (a E (0, 1] + + aه 2 2
(19)
a= = sup fa (0, 2 + t 0 14}
(20)
If x is bounded, the step sizes for x and À are chosen to be equal as:
oa = sup { a E (0. 1] + adz, 2 o 1- adN, 2h o (. }")
(21)
If x is unbounded but inequality constraints exist, the step lengths for x and X are chosen the same as the step length for If is unbounded and there are no inequality constraints, the step lengths for and À are chosen to be 1. In our experiments, additional line search procedures for step lengths, as described in Bellavia (1998). were not implemented in order to save computational time.
B Proof of Proposition i
Proof. Step 1. Let 8 >> 0, suppose that (x* A E NN(x", ,²,,"" {(x, A, )1(,,,,=))
(x", , x*)l < 8}. Define a := min o²) and == min(rr",, }, where o² and r7" are chosen to satisfy
the update rule (11) in Bellavia (1998). Then from the Assumption T
|| < (1 t + +1 FFFo ^ =
(22)
By using the steplength & obtained from the line search method of Bellavia (1998) and defining:
g = 1 - a (1 -1-c0 -1²²)
(23)
we obtain:
||
11 y
(1- Da"
1 (1-1-a aI-o
I- - 11y*
(24)
S (1 jt) I-ot
Lto
Fo(* >²,=*)]|
(1 Ff
where 14:lam I-mmxe
(o* + 1)*)
(0, 7mmxx and T/max (0, Then, there exists constant r
independent of k such that (24) holds, whenever y is bounded by (1 + o + + (ooo A 2
Hence, from Theorem 3.5 of Eisenstat and Walker (1994). follows that (ath, AA, 2²,2*) - (x",
Step 2. In this step, we aim to prove that the step size a of the inexact IPM is bounded away from 0. Given that Assumption is satisfied, it follows that k| + |= and ||v* are both bounded. Then we assume the first two equations of (o, A, 2) are Lipschitz continuous gradient with constant L. The remaining proofs are consistent with Theorem 3.2 in Bellavia (1998).
Step 3. The line search method of Bellavia
1998 enables at to satisfy:
(x k+1
k+1
x+1 ||| < (1 3(1 - =o x x
(25)
14

---

where B E (0, 1). Therefore FllFo( AM,2%)13 } is decreasing and bounded, hence, it is convergent.
Based on Assumption (24 holds. Furthermore, we assume || Fo(x* , > o 0 and 5 chosen sufficiently small so that
Fo A2, 22) Fo(21, A1, 21) Jæ1 \1, 21)(222 -21) (A2
$ ((1 [) x1) .02 A1) -1)) (2 21)
(26)
is satisfied, whenever (I1, X1, ) (22, 22, 22) E N28 >") Deñin
S
sup ( (*)=7))
[Fo( X, 2)||,
(27)
then based on Lemma 5.1 of Eisenstat and Walker 1994), the loop of backtracking line search of Bellavia (1998) will terminate with
1 1- mi - -1)
(28)
20
where 0 E 10, is a control parameter. This implies that the series
Σ (1 - r/t) is divergent. By k=
applying 25 iteratively, we have
|lFo x 21) I| S B(1 1F 1 =
- Fo 201 II
(1 - B(1 ffr))
0Si<k
(29)
$ [o 2o 2 /exppp - B Σ - (1-)
ošjck
Since 3 > and 1 1-1 f 2 0, the divergence of series
Σ (1- وم' implics ||F( A *] + 0.
==
C Datasets and Parameter setting
The key hyperparameters for each task, including K, T, and the hidden dimension of LSTM networks, are listed in Table[4] Generally, IPM-LSTM demonstrates improved performance with larger values of K and T, although this comes at the of increased computational time. Increasing T could enhance the quality of solutions to the linear system. In this study, K is maintained at the same value for both training and testing. Further analysis can be found in Appendix D.S
Table 4: Instance information and hyperparameter settings.
Information
Hyperparameters
Instance
Source
7
Wiipg
meg
[4]
K
T
Hidden dimension
100
50
50
0
100
50
50
Convex QPs (RHS)
200
100
100
0
100
50
75
50
50
100
50
50
Convex QPs (ALL)
200
100
100
0
100
50
100
100
50
50
0
100
50
50
Convex QCQPs (RHS)
Synthetic
200
100
100
0
100
50
100
50
50
100
50
50
Convex QCQPs (ALL)
200
100
100
100
50
100
100
S0
50
100
50
50
Non-convex Programs (RHS)
200
100
100
0
100
50
75
100
50
50
0
100
50
50
Non-convex Programs (ALI.)
200
100
100
100
100
qpl
50
1
50
100
50
30
qp2
50
100
30
st.rvl
5
100
50
50
st_
Globallib
20
10
0
20
100
50
50
st_r33
50
1
0
50
100
50
50
nv7
30
20
0
30
100
50
50
rv9
20
0
50
100
50
50
9p30_15_0 ______
Rand QP
30
15
6
30
30
100
50
50
15

---

D Experimental Results
D.1 Convex QPs
The performance on convex QPs, including "Convex QPs (RHS)" and "Convex QPs (ALL)", each instance with 100 variables, 50 inequality constraints, and equality constraints, is shown in Table From Table IPM-LSTM provided the best objective value with acceptable constraint violations.
Table 5: Computational results on convex QPs
Method
End-to-End
IPOPT (warm start
Tocal Time (6)
(te./
Gain
Obj
Max
Meun ineq...
Max eq.
eq.
Time
Itc.
Time
Convex QPs (RHS)
OSQP
-15.047
0.000
0.000
0.000
0.000
0.002
-15.047
0.000
0.000
DC3
-13,460
0.000
0.000
0.000
0.000
<0.001
10.5
0.227
0.227
13.9%//5.66%
NN
-12.570
0.000
0.000
0.350
0.130
<0.001
10.7
0.254
0.234
12.3%
DeepLDE
0.000
-5.7%
PDL
-14,999
0.011
0.003
0.002
0,000
c0.001
9.5
0.199
0.199
22.16//26.088
L00P-LC C
-13.628
0.000
0.000
0.000
0.000
<0.001
11.2
0.246
0.246
8.2% 6%
K-Proj
-11,778
10.7
IPM-18TM
-14.955
0.000
0.000
0.001
0,000
0.045
6.5
0.115
0.160
46.7 /40.%
Convex QPs (ALL)
OSQP
-16.670
0.000
0.000
0.000
0,000
0.002
IPOPT
-16.670
0.000
0.000
0.000
0.000
0.279
12.4
0.000
0.279
0.000
0.003
8.5
0.201
31.2%
Also, IPM-LSTM achieved the most significant reduction in iterations when the returned primal-dual solution pair utilized for warm-starting IPOPT.
D.2 Non-convex QPs
For non-convex QPs 8 all the non-zero elements without any special physical meaning, such as all zeros or all ones, are multiplied by value generated from uniform distribution in the range [0.8, 1.2]. If the original elements are integers, we will perform an additional rounding operation. We use "p" and "r" to represent these operations, respectively, and use "c" to denote the case without perturbation. The detailed perturbation rules for each instance are shown
Table 6: Perturbation rules.
Instance
Qo
Po
Pineq
Qineq
Poq
4eg
x
qpl qp2
P
C
P
C
P
P
P
C
st_rv
P
P
st_rv2
P
P
I
C
st_v3
P
P
r
rv7
P
P
st_rv9
P
P
r
r
1__0_11_111
P
P
P
p
p
p
D.3 Convex QCQPs
The performance on convex QPs, including "Convex QCQPs (RHS)" and "Convex QCQPs (ALL)", each instance with 100 variables, 50 inequality constraints, and 50 equality constraints, is shown in Table From Table 7 IPM-LSTM provided the best objective value with acceptable constraint violations. Also, IPM-LSTK achieved the most significant reduction in iterations when the returned primal-dual solution pair is utilized for warm-starting IPOPT.
16

---

Table 7: Computational results on convex QCQPs
Method
End-to-End
IPOPT (warm start)
Tocal Time (3)
Gain
Ob
Max incq...
Mean ineq.
cq
eq.
Time
Itc.
Time (s)
convex QCQPs (RIIS)
IPOPT
-18.761
0.000
0.000
0.000
0.000
0.267
12.3
-1,931
03.000
12.2
0.8%-00.5
DC3
-14,111
0.000
0.000
0.000
0,000
<0.001
10.6
0.244
0.244
13.8% 8.88
DeepLDE
-10.331
0,000
0.000
0,000
0.000
<0.001
11.2
0.282
0.282
8.9%/1.79
POL
-15.311
10.1
0.227
17.99/220.99
E-Proj
-15.450
0.000
0.000
0.000
0.000
<0.0011
10.8
0.247
0.247
12.2%% 2./2/111... 9%
IPM-19TX
-18.654
0,000
0.000
0,000
0.000
0.051
7.6
0.163
0.213
36.2%%/25.8
convex QCQPs (ALL)
IPOPT
-21.849
0.000
0.000
0,000
0.000
0.253
11.8
IPM-19TX
-21.200
0.000
0.001
0.001
7.4
0.124
0.173
37.3%/33.6%
D.4 A Simple Non-convex Program
1
min #ER"
Qox + Po sin(x)
2
T Pj x qj
1, 1, ,,..,,
(30)
pj x 9j
x S 24 S x
i == 1, 1....
including "Non-convex Programs (RHS)" and "Non-convex Programs (ALL)", each instance with T00/200 variables, 50/100 inequality constraints, and 50/100 equality constraints, is shown Table From this table, IPM-LSTM provided the best objective value with acceptable constraint violations. Also, IPM-LSTM achieved the most significant reduction in iterations when the returned primal-dual solution pair is utilized for warm-stari stamt-t IPOPT.
The performance on simple non-convex programs 30
Table 8: Computational results non-convex programs
Method
Obj.
Max inog.
Mean ineg.
End-tn-Fnd End
Max oq.
Mean cq.
Time (s) J
It.. J
IPOPT
Time (s)
Total Time (s)*
(Ite Time)
Gain
Non-convex Programs (RIIS):
200,
100,
100
IPOPT
DC3
-20.671
0.000
0.000
0.000
0.000
<0.001
10.9
0.601
0.603
16.8%5/155 9%
NN
-20.736
(.00)
0.000
0.632
0.235
<0.001
11.0
0.607
0.607
16.0%//20.7
DeepIDE
-20.074
0.000
19.8%//1977
PDL
-21.859
0.589
0.167
0.026
0.000
<0.001
10.9
0.600
0.600
16.8%/
100P -LC
-21.932
0.000
0.000
0.000
0.000
<0,001
10.2
0.558
0.558
22.122/2,
R-Proj -19.097
0.000
12.2%%/11..69
IPM-LSIN
-22.213
0.000
0.000
0.002
0.001
0.175
9.5
0.533
0.708
27.5%
Non-convex Programs (AI.I):
200,
100,
100
IPOPT
0.000
0.768
IPM-LSTN
-20.288
0.000
0.000
0.006
0.002
0.195
12.1
0.639
0.834
15.4%//88668
Non-convex Prugrums (RIIS):
100, Mire
50, may
50
IPOPT
-11.590
0.000
0.000
0.000
0.000
0.289
12.9
DC3
-10.660
0.000
0.000
0.000
0.000
<0.001
11.6
0.259
0.259
11.6%
NN
-10.020
0.000
0.000
0.350
0.130
<0.001
11.4
0.253
0.253
11.5%%
DeepID8
4,870
0.000
0.000
0.0088
0.000
<0.001
13.1
0.294
0.294
-1.698/-1
PDL
-11.385
0.006
0.002
0.001
0.000
-0.001
9.5
0.207
0.207
25. 25.6%%/28.4%
LOOP-L0
-11.296
0.000
0.000
0,000
0.000
<0.001
10.1
0.217
0.217
21.7%
H-Proj
-9.616
0.000
0.000
0.000
<0.001
0.252
0.252
12.44%//12.88 2.8%
IPN-L9TN
-11.421
0.000
0.000
0.002
0.001
0.044
8.9
0.181
0.225
31.0%/22.
Non-convex Programs (ALL)):
100, Mige
50,
50
IPOPT
-12.508
0.000
0.000
0.000
0.000
0.305
13.2
IPM-L.3NN
-12.360
(.000)
0.000
0.001
0,000
0.044
80
0,149
0.193
39.4%
D.5 Performance Analysis
D.5.1
The Number of LSTM Time Steps
The number of L.STM time steps a key hyperparameter which decides the performance-effhiienyy trade-off of IPM-LSTM. To illustrate this, we conduct experiments convex QP (RHS) problem
17

---

with 100 variables, 50 inequality constraints, and 50 equality constraints, investigating the quality of approximate solutions under different LSTM time step settings. As shown in Table 9 the IPM-LSTM with deeper L.TTM architectures generally yields better approximate solutions (with lower objective values, smaller constraint violation and better warm-start performance) but with longer computational time. Specifically, the IPM-LSTM with 60 and 70 time steps achieves most significant reductions in warm-starting iteration count and total runtime, respectively. However, taking into account the end-to-end solution time, we have opted set T to 50 in experiments. At each IPM iteration, as the LSTM network depth increases, || y² + F*] decreases (see Figure ((a)). This indicates an improvement in the quality of solutions to the linear systems. Furthermore, the corresponding IPM-LSTM converges faster (e.g., fewer IPM iterations) when the LSTM network becomes deeper (see Figure 4.
Table 9: Computational results on convex QPs (RHS) under different LSTM time steps.
End-to-End
IPOPT (warm start)
Total Time (8)
Gain
(Ite./ Time)
Obj.
Max ineq.
ineq
eq. |
eq.
(s)
10
-12.740
0.000
0.000
0.006
0.002
0.014
9.7
0,203
0.217
20.5% 39.
-14.615
0.900
0.000
0.003
0.001
0.021
0.171
0.192
30.3%%/28.6%%
30
-14,753
0.000
0.000
0.002
0.001
0.029
8.0
0.155
0.184
34.4% 66%
41.8%//37 9%
50
11.9955
0.000
0.000
0.001
0.000
0.045
6.5
0.115
0.160
46.7%
70
11.0266
0.000
0.000
0.000
0.000
0.064
6.0
0.100
0.164
50.8% /39.0//
49.2%%/33.4%
90
-14.600
0.000
0.000
0.000
0.000
0.081
6.6
0.117
0.198
43.9%
-8- 2' 20
7 30
F- 50
Ohijottee value
7 60
F'
7 90
Oytind
-39
Figure 4: The objective values returned by IPM-LSTM each IPM iteration convex QP (RHS).
D.5.2 Linear System Solutions
In order to provide more specific presentation of how accurately the LSTT performs, we report the detailed values of Figure 3(a) in Table 10 We can conclude that, [J " + F*] is roughly in the 1 n at each IPM iteration. To reveal the relationship between the error of the linear system solution [ pk and the LSTM time steps, hidden dimensions, training sizes and test sizes, we conduct experiments on representative convex QP (RHS) problems with 100 variables, 50 inequality constraints, and 50 equality constraints, and the results are included in Figure
same order of magnitude as (< x*)
Figure 5(a) with the number of L.STM time steps increasing, | F*|| decreases.
In Figure 5(b) we consider L.TTMs with 25, 50, 75, and 100 hidden dimensions and find that an LSTM with a hidden dimension of 50, as used in our manuscript, generally performs the best (e.g., the smallest y F*)).
In Figure 5(c) larger training set is more beneficial for model training. The training set size in our experiment is 8,334, and the error in solving the linear system y| - F* is
smaller compared with the case of 4,000 6,000 training samples.
18

---

As shown by Figure ((d) the number of samples in the test set does not affect the perfor- mance of LSTM for solving linear systems.
Table 10: The detailed values of Figure ((a)
IPM Ite.
g + F
1
2.396
0.900
10
0.154
0.255
20
0.104
0.124
30
0.073
0.073
40
0.052
0.047
50
0,040
0.031
60
0.032
0,021
70
0.027
0.013
80
0.024
0.008
0.022
0.006
100
0.020
0.005
(a) LSTM time steps
(b) Hidden dimension
(c) Training size
(d) Test size
The relationship between the error the linear system solution with different parameter settings of LSTM.
Figure 5:
We take the log of the y-axis in Figure 3(a) and plot it in Figure 6(a) Roughty speaking, || F* is smaller than vl(==* ( <=*1/n in the first 40 IPM iterations, while || + F*1 surpasses nl(=*) ==1/n I/n in the later IPM iterations. We increase the number of LSTM time steps and report the computational results Figure 6(b) From Figure 6 we can claim that with the number LSTM
time steps increasing, [k F*] becomes smaller.
kall PD)
Mg FDD bgglo(
kglv
-12
IPV teudios
IPV Inveres
(a) IPM-LSTM (100. 50)
(b) IPM-LSTM (100, 90)
Figure 6: Equation under different LSTM time steps. The first number in the parentheses denotes the number of IPM iterations, while the second one represents the number of LSTM time steps.
D.5.3 Condition Numbers
The LSTT approach for solving linear systems is negatively affected by their large condition numbers. To demonstrate this, we consider the least squares problem
min (vv) = || 1rM vER!
(31)
19

---

We utilize first-order method (e.g., steepest descent method) to minimize o(v) and achieve linear convergence (Nocedal and Wright 1999.. i.e.
(1
(1² 2 - 2
(1'')
(32)
(N (JK))² +1
As we discussed in Section 3.3 since solving linear systems via LSTM networks emulates iterative first-order methods, thus the value of (J*) affects the performance of L.TTM networks. However, LSTM networks can empirically achieve a faster convergence rate than traditional first-order algo- rithms when solving the same least squares problems as shown in the computational studies (Section 3.1) of [ndrycchowicz et al. (2016). In order to alleviate the effect of large condition numbers, as discussed in Section 3.3 we have employed preconditioning techniques. To illustrate its effect, we conduct experiments on the simple non-convex programs, and report λ. (J*) and their values after preconditioning (in parantheses) across several IPM iterations (c.g., 1st 10th, 20 50th 100th) in We can conclude that the condition numbers & (JL) remain within reasonable magnitudes during the later IPM iterations, and significantly reduced after applying the preconditioning technique.
Table 11
Table 11: The condition numbers of simple non-convex programs in IPM iteration process.
Instance
1"
10th
20th
50th
100
Non-convex Programs (RHS) (100, 50, 50)
53.8 (59.8)
126.2 (580.7)
153.1 (711.2)
208.7 (1004.8) 8)
348 (1860.1
Non-convex Programs (ALI.) (100.50 50)
55.4 (59.8)
(517.0)
139.8 (658.5)
(1199919909))
329.2 (1859.9)
Non-convex Programs (RHS) (200, 100, 100)
91.5 (99.8)
157.1 (1114.0)
205.8 (1441.1)
326.2 323.2
488.3 (3667.8)
Non-convex Programs (AI.I) (200, 100, 100)
72.1(7577)
(1143.4)
(1352.7)
(2016.6)
(3015.3)
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Abstract and Section 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
Please refer to Section 3.2 and Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section
3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification: Please refer to Section 4.1 for data. Our code will be available to public once our work accepted, as mentioned Section
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
public/guides/ Sowtss//oooduuubassssoooooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https http://ooooSuuiiss.ooeee
//nips.cc/ppp wwwppbblccgguiecuceccccccooooope
httSSSbbmiiiiooooiiiiiooooii for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Section 4.1 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research presented in this paper adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There are many potential societal consequences of our work, none of which we feel must specifically highlighted here.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not present any such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer Section
4.1
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "ITjJgzIC1Z", "reviewer": "Reviewer_Qdyo", "summary": "This paper proposed to replace the linear system solver used in the inner loop of interior point method (IPM) with an LSTM for solving general non-linear programs. The LSTM is trained in a unsupervised manner to minimize a unconstrained least square objective derived from the KKT conditions. The proposed framework, IPM-LSTM, can be used in an end-to-end way, or to warm-start IPM so that the number of overall outer IPM iterations can be reduces and thus the solving time decreases. The authors theoretically proved that, as long as the trained LSTM achieves certain level of accuracy, the IPM will converge. The authors conducted empirical experiments on a variety of non-linear programs to verify the effectiveness of IPM-LSTM and that it approximately satisfied the assumption in the theoretical analysis.", "questions": "See the weaknesses section.", "limitations": "See the weaknesses section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of plugging in L2O methods to approximate a single step in a bigger optimization framework is interesting and promising.\n\n2. The proposed architecture is concise and effective to some extent. The authors also provided theoretical analysis to support their method.\n\n3. To some extent, the presented empirical resutls are promising.s", "weaknesses": "There are three major issues that undermine the robustness of this work:\n\n1. The linear system solver used for IPOPT is unclear, which significantly impacts the solver's overall performance. While MUMPS is the default, HSL typically performs better, achieving 2-3 times faster results. This performance surpasses that of IPOPT when warm-started with the IPM-LSTM solution.\n\n2. There is no analysis of how the performance changes with varying numbers of iterations in the inner LSTM, which decides the performance-efficiency trade-off of IPM-LSTM. Longer inner iterations (deeper LSTM) may improve the quality of the solutions but increases computation costs. However, improved solution quality may also decrease the number of outer iterations needed and thus reduce the overall solving time. Moreover, increased LSTM depth can make the training process more difficult. Overall, thise is tricky part and should be empirically investigated more carefully.\n\n3. Although LSTM benefits from batched processing—a potential strength of IPM-LSTM accelerated by GPU—this work fails to provide empirical evidence supporting this property.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for reading our manuscript and providing constructive coments. Please refer to the Author Rebuttal for a clarification of the two-stage framework proposed in our work.\n\n>*1. The linear system solver used for IPOPT is unclear, which significantly impacts the solver's overall performance. While MUMPS is the default, HSL typically performs better, achieving 2-3 times faster results. This performance surpasses that of IPOPT when warm-started with the IPM-LSTM solution.*\n\nThanks for your suggestion. The default solver MUMPS was used in our experiments. To evaluate the effect of linear solvers on the performance of IPOPT, we consider 3 commonly used linear solvers (MA27, MA57, and MA86) from HSL and bundle IPOPT with each of them. We conduct experiments on 6 datasets used in our manuscript, each with 100 variables, 50 inequality constraints, and 50 equality constraints. We report the average computational time (in seconds) for each dataset in the table below. Results for IPOPT with different linear solvers are presented in the 2nd - 5th columns, while those for IPOPT warm-started by IPM-LSTM are listed in the last four columns.\n\n- The computational results demonstrate that **MUMPS is generally outperformed by MA57** but superior to MA27 and MA86. \n- **Regardless of linear solvers, warm-starting IPOPT** with primal-dual solutions provided by **IPM-LSTM enhances the performance** of IPOPT itself.\n\n\n\n||IPOPT||||IPM-LSTM+IPOPT||||\n|--|---|---|---|---|---|---|---|---|\n|**Dataset**|**MUMPS**|**MA27**|**MA57**|**MA86**|**MUMPS**|**MA27**|**MA57**|**MA86**|\n|**Convex QPs (RHS)**|0.269 |0.328 |0.191 |0.304 |0.170 |0.220 |**0.131** |0.195 |\n|**Non-convex Programs (RHS)**|0.289 |0.428 |0.215 |0.387 |0.225 |0.297 |**0.171** |0.299 |\n|**Convex QCQPs (RHS)**|0.287 |0.388 |0.251 |0.327 |**0.204** |0.270 |0.220 |0.226 |\n|**Convex QPs (ALL)**|0.279 |0.354 |0.199 |0.376 |0.201 |0.272 |**0.159** |0.245 |\n|**Non-convex Programs (ALL)**|0.305 |0.396 |0.213 |0.387 |0.193 |0.256 |**0.146** |0.237| \n|**Convex QCQPs (ALL)**|0.253 |0.328 |0.213 |0.311 |0.173 |0.202 |**0.160** |0.194 |\n\n\n>*2. There is no analysis of how the performance changes with varying numbers of iterations in the inner LSTM, which decides the performance-efficiency trade-off of IPM-LSTM. Longer inner iterations (deeper LSTM) may improve the quality of the solutions but increases computation costs. However, improved solution quality may also decrease the number of outer iterations needed and thus reduce the overall solving time. Moreover, increased LSTM depth can make the training process more difficult. Overall, thise is tricky part and should be empirically investigated more carefully.*\n\nThanks for your suggestion. We first remark that in our implementation, the number of IPM iteration (e.g., outer iteration) is fixed to $100$. The number of iterations in the inner LSTM indeed incurs the performance trade-off. To illustrate this, we conduct experiments on convex QP (RHS) problems with 100 variables, 50 inequality constraints, and 50 equality constraints, investigating the quality of approximate solutions under different LSTM inner iteration settings. We report the results in the table below and Fig. 1 in the supplementary material.\n- At each IPM iteration, **as the LSTM network depth increases**, $\\\\|J^k y^k + F^k\\\\|$ decreases (see Fig. 1(a)). This indicates **an improvement in the quality of solutions to the linear systems**. Furthermore, the corresponding **IPM-LSTM converges faster** (e.g., fewer outer iterations) when the LSTM network becomes deeper (see Fig.1(b)).\n- From the table below, generally speaking, **the IPM-LSTM with deeper LSTM architectures tends to produce better approximate solutions** (with lower objective values and smaller constraint violation) but **with longer computational time.**\n- Training deeper LSTM networks indeed becomes more challenging, such as **longer training time and more memory consumption** due to large computational graphs, **vanishing or exploding gradient** issues [1].\n\n\n| # Ite. in LSTM | Obj. | Max ineq. | Mean ineq. | Max eq. | Mean eq. | Time (s) |\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n|10   | -12.740 | 0.000  | 0.000 |  0.006  |  0.002   | 0.012 | \n|20   | -14.615 | 0.000  | 0.000 |  0.003  |  0.001   | 0.020 | \n|30   | -14.753 | 0.000  | 0.000 |  0.002  |  0.001   | 0.028 |\n|40   | -14.897 | 0.000  | 0.000 |  0.003  |   0.001  | 0.037  | \n|50   | -14.906 | 0.000  | 0.000 | 0.001   |  0.000   | 0.045 | \n|60   | -15.021 | 0.000  | 0.000 |  0.001  |  0.000   | 0.055  | \n|70   | -15.026 | 0.000  | 0.000 |  0.000  |  0.000   | 0.063 | \n|80   | -15.012 | 0.000  | 0.000 |  0.000  |  0.000   | 0.072 |\n|90   | -14.960 | 0.000  | 0.000 |  0.000  |  0.000   | 0.080 | \n\n>*3. Although LSTM benefits from batched processing—a potential strength of IPM-LSTM accelerated by GPU—this work fails to provide empirical evidence supporting this property.*\n\nThanks for your suggestion. Given a batch size of $k$ testing samples, we are able to feed all of them to the trained IPM-LSTM at the same time. Let $T$ denote the wall-clock time used when all samples are solved by IPM-LSTM. The average solution time for each instance is then $T/k$. Thus, the proposed IPM-LSTM approach can indeed leverage the advantage of GPU batch processing.\n\nTo demonstrate this, we conducted tests on convex QPs (RHS) with 100 variables, 50 inequalities, and 50 equalities, with batch sizes varying from 10 to 5,000. The computational results are reported in Fig. 1\\(c\\) of the supplementary material. As shown in Fig. 1\\(c\\), **the average solution time decreases as the batch size increases**. However, due to hardware limitations, once the batch size exceeds a certain threshold, the average computational time will level off.\n\n[1] Pascanu, Razvan, Tomas Mikolov, and Yoshua Bengio. \"On the difficulty of training recurrent neural networks.\" International conference on machine learning. PMLR, 2013.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e5bkQxPviN", "reviewer": "Reviewer_9Qph", "summary": "This paper introduces a method called IPM-LSTM, which integrates machine learning techniques into interior point methods (IPM). Specifically, the authors propose training a RNN model, LSTM, to quickly approximate the solution of linear systems within IPM. This approach is numerically validated on several convex and nonconvex QPs.", "questions": "Refer to \"Weaknesses\".", "limitations": "This theoretical paper appears to have no potential negative societal impact.", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The topic is important as IPM plays a crucial role in solving linear and nonlinear programs, which have extensive applications in scientific computing.\n- The idea of accelerating a subroutine of IPM, rather than applying deep neural nets end-to-end to solve optimization problems, is insightful. The convergence of IPM is typically fast (superlinear convergence), leaving little room for improvement. However, solving the linear system in IPM is usually a computational bottleneck, making it worthwhile to accelerate with machine learning.\n- The experimental results are promising. IPM-LSTM clearly outperforms other learning-based baseline methods in terms of the objective function.", "weaknesses": "My main concern about this paper is Assumption 1, which requires the accuracy of the linear system solution to increase as the number of iterations $k$ increases. Based on this assumption, exact convergence is derived, as shown in Proposition 1. However, the empirical results in Section 4 indicate that IPM-LSTM does not achieve exact optimality, revealing a gap between theory and practice. To address this gap, I suggest:\n- Reporting the error of LSTM at each iteration. This would provide readers with an understanding of how accurately the LSTM performs. Additionally, it would be beneficial to report the relationship between the error of the linear system solution with the size, training, and testing overhead of the LSTM.\n- Using a log scale y-axis for Figure 3a for better precision. For example, 0.01 is 10 times greater than 0.001, but this difference is not reflected in the linear scale of Figure 3a.\n- Modifying Assumption 1 to better align with practice. For example, assuming a fixed error on the right-hand sides of equations (5) and (6). Based on this relaxed assumption, a result similar to Proposition 1 could be derived, but with a fixed error on the limit of $(x^k,\\lambda^k,z^k)$. The relationship between the allowed error in Assumption 1 and the propagated eventual error in Proposition 1 would sufficiently describe the performance of IPM-LSTM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for reading our manuscript and providing constructive coments. Please refer to the Author Rebuttal for a clarification of the two-stage framework proposed in our work.\n\n> *My main concern about this paper is Assumption 1, which requires the accuracy of the linear system solution to increase as the number of iterations $k$ increases. Based on this assumption, exact convergence is derived, as shown in Proposition 1. However, the empirical results in Section 4 indicate that IPM-LSTM does not achieve exact optimality, revealing a gap between theory and practice.*\n\nThank you for pointing this out. The IPM-LSTM utilizes an LSTM network to return approximate solutions to linear systems. Proposition 1 implies that exact optimality could be achieved by IPM-LSTM in theory. However, in practice, solving linear systems to the tolerance specified in Assumption 1 would be too time-consuming. As a result, we choose to **strike a balance between the accuracy and efficiency of solving linear systems via LSTM**. Consequently, the satisfaction of Assumption 1 is not fully ensured in our implementation, leading to the mild suboptimality and infeasibility observed in Section 4.\n\n>*To address this gap, I suggest:*\n>*1. Reporting the error of LSTM at each iteration. This would provide readers with an understanding of how accurately the LSTM performs. Additionally, it would be beneficial to report the relationship between the error of the linear system solution with the size, training, and testing overhead of the LSTM*.\n\nThanks for your suggestion. The error of solving linear systems is indeed vital. We have plotted the progress of $\\\\|J^ky^k+F^k\\\\|$ as the IPM iteration increases in Fig. 3(a) of our manuscript. Now we report the detailed values in the following table. From this table, **$\\\\|J^ky^k+F^k\\\\|$ is roughly in the same order of magnitude as $\\eta[(z^k)^{\\top}x^k]/n$** at each IPM iteration.\n\n| IPM Ite. | $\\|\\|J^ky^k+F^k\\|\\|$ | $\\eta[(z^k)^{\\top}x^k]/n$ |\n|:------ | ----- | -----|\n| 1       |  2.396  | 0.900 |\n| 10      |  0.154  | 0.255 |\n| 20      |  0.104  | 0.124 |\n| 30      |  0.073  | 0.073 |\n| 40      |  0.052  | 0.047 |\n| 50      |  0.040  | 0.031 |\n| 60      |  0.032  | 0.021 |\n| 70      |  0.027  | 0.013 |\n| 80      |  0.024  | 0.008 |\n| 90      |  0.022  | 0.006 |\n| 100     |  0.020  | 0.005 |\n\nTo reveal the relationship between the error of the linear system solution $\\\\|J^ky^k+F^k\\\\|$ and the LSTM time steps, hidden dimensions, training sizes and test sizes, we conduct experiments on representative convex QP (RHS) problems with 100 variables, 50 inequality constraints, and 50 equality constraints, and the results are included in Fig. 2 of the supplementary material.\n- In Fig. 2(a), with the LSTM time step increasing, $\\\\|J^ky^k+F^k\\\\|$ decreases.\n- In Fig. 2(b), we consider LSTMs with 25, 50, 75, and 100 hidden dimensions and find that an LSTM with a hidden dimension of 50, as used in our manuscript, generally performs the best (e.g., resulting in the smallest $\\\\|J^ky^k + F^k\\\\|$).\n- In Fig. 2\\(c\\), a larger training set is more beneficial for model training. The training set size used in our manuscript is 8,334, and the error in solving the linear system $\\\\|J^ky^k + F^k\\\\|$ is smaller compared with the case of 4,000 or 6,000 training samples.\n- As shown by Fig. 2(d), the number of samples in the test set does not affect the performance of LSTM for solving linear systems. \n\n>*2. Using a log scale y-axis for Figure 3a for better precision. For example, 0.01 is 10 times greater than 0.001, but this difference is not reflected in the linear scale of Figure 3a.*\n\nThanks for your suggestion. We now take the log of the y-axis in Fig. 3(a) of our manuscript and plot it in Fig. 3(a) of the supplementary material. \n- Roughly speaking, $\\\\|J^k y^k + F^k\\\\|$ is smaller than $\\eta [(z^k)^\\top x^k] / n$ in the first $40$ IPM iterations, while $\\\\|J^k y^k + F^k\\\\|$ surpasses $\\eta [(z^k)^\\top x^k]/n$ in the later IPM iterations.\n- Following the comments from Reviewer Qdyo, we increase the LSTM time steps and report the computational results in Fig. 3(b) of the supplementary material. From Fig. 3(a) and 3(b), We can claim that **with the LSTM time steps increasing, $\\\\|J^k y^k + F^k\\\\|$ becomes smaller and closer to $\\eta [(z^k)^\\top x^k]/n$**.\n\n>*3. Modifying Assumption 1 to better align with practice. For example, assuming a fixed error on the right-hand sides of equations (5) and (6). Based on this relaxed assumption, a result similar to Proposition 1 could be derived, but with a fixed error on the limit of $(x^k, \\lambda^k, z^k)$. The relationship between the allowed error in Assumption 1 and the propagated eventual error in Proposition 1 would sufficiently describe the performance of IPM-LSTM.*\n\nThank you for your suggestion. **Equation (5) in our manuscript is designed to be consistent with the assumptions made in classic inexact IPMs or inexact Newton methods**, as seen in Equation (4) in [1], Equation (2.1) in [2], Equation (6) in [3], and Equation (12) in [4]. To the best of our knowledge, the theoretical bounds referenced in these works are dependent on the iteration count $k$, and do not involve fixed error terms. We would explore the use of fixed error bounds in our future studies.\n\n[1] Bellavia, Stefania. Inexact interior-point method. Journal of Optimization Theory and Applications 96 (1998): 109-121.\n\n[2] Eisenstat, Stanley C., and Homer F. Walker. Globally convergent inexact Newton methods. SIAM Journal on Optimization 4.2 (1994): 393-422.\n\n[3] Al-Jeiroudi, Ghussoun, and Jacek Gondzio. Convergence analysis of the inexact infeasible interior-point method for linear optimization. Journal of Optimization Theory and Applications 141 (2009): 231-247.\n\n[4] Gondzio, Jacek. Convergence analysis of an inexact feasible interior point method for convex quadratic programming. SIAM Journal on Optimization 23.3 (2013): 1510-1527.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iQwBmKYty6", "reviewer": "Reviewer_dSF8", "summary": "This paper introduces IPM-LSTM, an approach integrating Long Short-Term Memory (LSTM) neural networks with Interior Point Methods (IPMs) to solve Nonlinear Programs (NLPs). The key innovation lies in approximating solutions to linear systems within IPMs using LSTMs, aiming to accelerate the convergence of classic IPM solvers. This approach leverages the Learning to Optimize (L2O) paradigm, presenting a two-stage framework where LSTM-generated solutions are used to warm-start an IPM solver. The authors compare the IPM-LSTM against traditional solvers and recent L2O methods across various NLPs, including Quadratic Programs and Quadratically Constrained Quadratic Programs. The proposed method reportedly reduces iterations by up to 60% and solution time by up to 70%.", "questions": "1. In the experiments, why not directly use the simple NN prediction as a warm-start point for the IPM solver?\n2. In the simulation part, as the solutions obtained by the proposed IPM-LSTM approach are used as a warm-start point to IPM, why are the equality constraints not satisfied? The authors are suggested to explain the experimental setup clearly.\n3. In Table I, the proposed IPM-LSTM approach has a larger constraint violation with a longer solution time as compared to the OSQP algorithm. Does this mean the proposed approach can not exceed state-of-the-art methods for solving convex QP problems?", "limitations": "The authors have discussed the limitation in Sec. 5.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The integration of LSTM networks to approximate the solutions of linear systems in IPMs is novel.\n2. The paper provides theoretical insights into the convergence properties of the proposed method under specific assumptions, adding to its credibility and understanding.\n3. The paper provides a comprehensive empirical evaluation across several types of NLPs. The results demonstrate improvements over traditional methods in terms of iteration count and computational time, which supports the effectiveness of the proposed method.", "weaknesses": "1. The IPM-LSTM, which uses LSTM to iteratively solve linear systems at each IPM iteration, repeats K times and feeds it into the IPM solver as a warm-start point, needs more justification.\n   1. The decision to use the L2O approach for solving a least squares problem (problem (4)) is not adequately justified. According to Assumption 1, the approximated solution needs to be bounded and accurate enough to guarantee the convergence of the outer loop. From the theoretical side, it is unclear how hard it is to satisfy those conditions by L2O approaches. From the empirical side, as shown in Figure 3(a), the accuracy condition is not always satisfied. If assumption 1 can not be guaranteed, the convergence of approximated IPM can not be guaranteed according to Prop. 1. \n   2. The approach of using an approximated IPM solution instead of directly generating a warm-start point using either NN or L2O raises questions about efficiency and effectiveness. Previous works [1-4] have shown that direct prediction of warm-start points can be more straightforward and computationally efficient. Besides, the convergence of approximated IPM can not be guaranteed, which also raises concerns about the quality of such a warm-start point. The authors need to provide more discussion or experimental comparisons to justify their more complex, iterative approximation method.\n\n[1] R. Sambharya, G. Hall, B. Amos, and B. Stellato, \"End-to-End Learning to Warm-Start for Real-Time Quadratic Optimization\", arXiv preprint arXiv:2212.08260, 2022.\n\n[2] Sambharya R, Hall G, Amos B, et al. Learning to warm-start fixed-point optimization algorithms[J]. Journal of Machine Learning Research, 2024, 25(166): 1-46.\n\n[3] F. Diehl, \"Warm-Starting AC Optimal Power Flow with Graph Neural Networks\", in Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS) Workshop, Vancouver, BC, Canada, Dec. 8 - 14, 2019.\n\n[4] K. Baker, \"Learning Warm-Start Points for AC Optimal Power Flow\", in Proceedings of IEEE 29th Machine Learning for Signal Processing Conference, Pittsburgh, PA, USA, Oct. 13 - 16, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for reading our manuscript  and providing constructive coments. Please refer to the Author Rebuttal for a clarification of the two-stage framework proposed in our work.\n\n### Weakness\n>*1. The decision to use the L2O approach for solving a least squares.......*\n\nThanks for raising this point.\n- **Theoretical**: the least-squares problem (4) in our manuscript is an **unconstrained convex optimization** problem. Several works have proved that **L2O approaches could solve such problems to guaranteed tolerances**. For example, the work of [5] proved that solutions provided by properly parameterized LSTM networks converge to one of the minimizers of composite optimization problems. Another interesting case is the work of [6] in which the authors proposed an unrolling of the ISTA algorithm and proved that solutions returned by the ISTA-unrolled method converge to the optimal ones.\n- **Empirical**: The accuracy condition is not strictly enforced in our implementation since we aim to **balance the accuracy and efficiency** of solving linear systems via LSTM. As shown in Section 4, the IPM-LSTM algorithm is able to provide high-quality approximate solutions with mild infeasibility and sub-optimality (**Stage I**). **These solutions are suitable for many practical applications**. If solutions with increased feasibility and optimality are needed, one can quickly **polish and refine** the approximate solutions by feeding them to an IPM solver (**Stage II**).\n\n\n\n>*2. The approach of using an approximated IPM solution instead of.....[1][2][3][4]...*\n\n\nThanks for raising this point. We further clarify the motivation of our algorithmic design as follows.\n\n- In this work, we focus on developing learning-based methods to **warm-start an IPM solver for optimizing general nonlinear and convex/non-convex programs**. As the Reviewer 9Qph commented, \"*The topic is important as IPM plays a crucial role in solving linear and nonlinear programs, which have extensive applications in scientific computing.*\" The works of **[1][2] are focused on warm-starting fixed-point algorithms for efficiently solving convex QPs and conic programs**. \n- As we mentioned in Section 2, warm-starting an IPM solver is notoriously hard since it entails providing **well-centered primal-dual solutions** [7][8]. Recently, learning-based approaches have been proposed to address this. However, most of works such as **[3][4] did not even include dual solutions** for warm-starting. **These methods might work well for specific applications** but could not boost the performance of an IPM solver when optimizing general nonlinear programs. Some works such as [9] indeed provide primal-dual solutions but those pairs are **not intended to be well-centered**.\n- Our proposed IPM-LSTM approach works in a similar way as a classic IPM except that linear systems are approximately solved by LSTM networks. Since every iterate (e.g., primal-dual solution) follows a central path (e.g., the same parameter $\\mu$ is chosen for each dimension in the perturbed KKT system, as shown in Algorithm 1), the final solution pair returned by IPM-LSTM would then be well-centered and of high quality. **Such a primal-dual solution is thus well-suited for warm-starting an IPM solver**.\n\n\n### Questions \n>*1. ...why not directly use the simple NN prediction as a warm-start point...*\n\nThanks for raising this point. As pointed out by [10], a **simple NN prediction often leads to significant constraint violation**. To alleviate this issue, we integrated the objective function and penalty for constraint violation into the loss function and included such a method as a baseline (denoted as \"NN\") in our manuscript. From Table 1, 2, 5, 7, and 8, we can conclude that **warm-starting IPOPT with initial solutions predicted by NNs brings limited (sometimes even negative) performance improvement**.\n\n\n>*2. ...why are the equality constraints not satisfied...*\n\n\nWe apologize for any confusion. In the computational results, **the \"IPM-LSTM\" in the end-to-end fashion produces approximate solutions** that might be neither feasible nor optimal, which can result in small violations of the equality constraints. However, when **these solutions are used to warm-start an IPM solver** such as IPOPT (denoted as \"IPOPT (warm-start)\" in each table), IPOPT returns locally optimal solutions that **do not exhibit any constraint violations**, as shown in Table 1, 2, 5, 7, and 8.\n\n\n>*3....Does this mean the proposed approach can not exceed state-of-the-art...*\n\nThank you for raising this question. Indeed, the proposed IPM-LSTM approach is outperformed by OSQP when solving convex QPs.\n\n- **The IPM-LSTM method is designed to provide high-quality approximate solutions for general nonlinear and convex/non-convex programs**. It follows a classic IPM approach, with the distinction that linear systems are approximately solved using an LSTM network.\n- **OSQP** employs an operator-splitting algorithm **specifically tailored for convex QPs**. This solver has demonstrated superior performance even when compared to state-of-the-art general-purpose solvers like IPOPT and Gurobi in optimizing convex QPs.\n\nTherefore, **IPM-LSTM is capable of addressing a wide range of optimization problems, but it is not as competitive as OSQP when it comes to solving convex QPs**.\n\n[5] Liu, Jialin, et al. (2023) \"Towards constituting mathematical structures for learning to optimize.\" ICML.\n\n[6] Aberdam, A., et al. (2021). Ada-lista: Learned solvers adaptive to varying models. IEEE TPAMI.\n\n[7] Nocedal, J., & Wright, S. J. (Eds.). (1999). Numerical optimization.\n\n[8] Wright, S. J. (1997). Primal-dual interior-point methods.\n\n[9] Park, S., & Van Hentenryck, P. (2023). Self-supervised primal-dual learning for constrained optimization. AAAI.\n\n[10] Donti, P. L., et al. (2021). DC3: A learning method for optimization with hard constraints. ICLR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Before addressing the reviewers' comments, we would like to first clarify the two-stage framework proposed in our work.
- **Stage I**: we **utilize IPM-LSTM to produce a high-quality approximate solution** (which might neither be feasible nor optimal but presumably well-centered). The IPM-LSTM is designed like a classic IPM with linear systems being solved by an LSTM network rather than a linear system solver. 
- **Stage II**: we then use the approximate solution to **warm start an IPM solver**, such as IPOPT. 

To the best of our knowledge, **our work is the first attempt of enhancing IPMs with learning-based techniques for addressing general nonlinear programs**.

Meta-review: This paper proposed to replace the linear system solver used in the inner loop of the interior point method (IPM) with an LSTM for solving general non-linear programs. Most of the reviewers agreed that this paper is technically solid. Overall, this paper is well-structured with a logical flow, and the idea of integrating machine learning techniques into interior point methods is valuable for this community. All the reviews tend to accept the paper during the reviewer-author discussion phase. Therefore, I recommend this paper to the NeurIPS 2024 conference.
Decision: Accept (poster)