IPM-LSTM: A Learning-Based Interior Point Method for Solving Nonlinear Programs

Authors: xi  gao
jinxin  xiong
akang  wang
qihong  duan
jiang  xue
qingjiang  shi
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:18:14

Abstract: Solving constrained nonlinear programs (NLPs) is of great importance in various domains such as power systems, robotics, and wireless communication networks. One widely used approach for addressing NLPs is the interior point method (IPM). The most computationally expensive procedure in IPMs is to solve systems of linear equations via matrix factorization. Recently, machine learning techniques have been adopted to expedite classic optimization algorithms. In this work, we propose using Long Short-Term Memory (LSTM) neural networks to approximate the solution of linear systems and integrate this approximating step into an IPM. The resulting approximate NLP solution is then utilized to warm-start an interior point solver. Experiments on various types of NLPs, including Quadratic Programs and Quadratically Constrained Quadratic Programs, show that our approach can significantly accelerate NLP solving, reducing iterations by up to 60% and solution time by up to 70% compared to the default solver.

Main paper: 
 introduction: 
 Constrained Nonlinear Programs (NLPs) represent a category of mathematical optimization problems in which the objective function, constraints, or both, exhibit nonlinearity. Popular NLP variants encompass Quadratic Programs (QPs), Quadratically Constrained Quadratic Programs (QCQPs), semi-definite programs, among others. These programs are commonly classified as convex or non-convex, contingent upon the characteristics of their objective function and constraints. The versatility of NLPs allows for their application across a wide array of domains, including power systems (Conejo and Baringo, 2018) , robotics (Schaal and Atkeson, 2010) , and wireless communication networks (Chiang, 2009) .The primal-dual Interior Point Method (IPM) stands as a preeminent algorithm for addressing NLPs (Nesterov and Nemirovskii, 1994; Nocedal and Wright, 1999) . It initiates with an infeasible solution positioned sufficiently far from the boundary. Subsequently, at each iteration, the method refines the solution by solving a system of linear equations, thereby directing it towards the optimal solution. Throughout this iterative process, the algorithm progresses towards feasibility and optimality while keeping the iterate well-centered, ultimately converging to the optimal solution. However, a notable computational bottleneck arises during the process of solving linear systems, necessitating matrix decomposition with a runtime complexity of O(n 3 ). spanning unconstrained optimization (Chen et al., 2022a) , linear optimization (Chen et al., 2022b; Li et al., 2024) , and combinatorial optimization (Baker, 2019; Gasse et al., 2022; Han et al., 2023) . Its ability to encapsulate common optimization patterns renders it particularly appealing. Noteworthy is the application of learning techniques to augment traditional algorithms such as the gradient descent method (Andrychowicz et al., 2016) , simplex method (Liu et al., 2024) , and IPM (Qian et al., 2024) .We observe that existing works on learning-based IPMs primarily concentrate on solving LPs (Qian et al., 2024) . Motivated by the robustness and efficiency of IPMs for general NLPs, we pose the following question:Can we leverage L2O techniques to expedite IPMs for NLPs?In this study, we propose the integration of Long Short-Term Memory (LSTM) neural networks to address the crucial task of solving systems of linear equations within IPMs, introducing a novel approach named IPM-LSTM. An illustration of the IPM-LSTM approach is depicted in Figure 1 . Specifically, we substitute the conventional method of solving linear systems with an unconstrained optimization problem, leveraging LSTM networks to identify near-optimal solutions for the latter. We integrate a fixed number of IPM iterations into the LSTM loss function and train these networks within the self-supervised learning framework. The substitution is embedded within a classic IPM to generate search directions. Ideally, the primal-dual solution provided by IPM-LSTM should be well-centered with respect to the boundary and associated with a small duality gap. Finally, we utilize such approximate primal-dual solution pairs to warm-start an interior point solver. IPM-LSTM has several attractive features: (i) it can be applied to general NLPs; (ii) it strikes a good balance between feasibility and optimality in the returned solutions; (iii) it can warm-start and thereby accelerate interior point solvers.The distinct contributions of our work can be summarized as follows:• Approximating Solutions to Linear Systems via LSTM: This study marks the first attempt to employ learning techniques for approximating solutions of linear systems in IPMs, achieving significant speedup compared to traditional linear algebra approaches.• Two-Stage Framework: We introduce a two-stage L2O framework. In the first stage, IPM-LSTM generates high-quality primal-dual solutions. In the second stage, these solutions are used to warm-start an interior point solver. This framework effectively accelerates the solving process of IPMs while yielding optimal solutions.• Empirical Results: Compared with existing L2O algorithms, IPM-LSTM demonstrates favorable performance in terms of solution feasibility and optimality across various NLP types, including QPs and QCQPs. Utilizing these solutions as initial points in the state-of-the-art NLP solver IPOPT (Wächter and Biegler, 2006) reduces iterations by up to 60% and solution time by up to 70%. 
related works: 
 Constrained L2O. Approaches utilizing L2O for constrained optimization can be broadly categorized into two directions: (i) direct learning of the mapping from optimization inputs to full solutions, and (ii) integration of learning techniques alongside or within optimization algorithms (Bengio et al., 2021; Donti et al., 2021) . Previous works (Fioretto et al., 2020; Huang et al., 2021; Pan et al., 2023) adopted the former approach, employing a supervised learning scheme to train the mapping. However, this method necessitates a large number of (near-)optimal solutions as training samples, making it resource-intensive. From a self-supervised learning perspective, an intuitive approach is to incorporate the objective and penalization for constraint violation directly into the loss function (Kim et al., 2023; Park and Van Hentenryck, 2023) . Nevertheless, such an approach may not guarantee the feasibility of the returned solutions. To address the feasibility issue, notable works such as Donti et al. ( 2021) first predict a partial solution via neural networks and then complete the full solution by utilizing equality constraints, iteratively correcting the solution towards the satisfaction of inequalities by applying gradient-based methods. However, for general nonlinear inequalities, this correction step may not ensure feasibility (Liang et al., 2023) . Other approaches like Li et al. (2023) utilized gauge mappings to enforce feasibility for linear inequalities, while Liang et al. (2023) proposed the homeomorphic projection scheme to guarantee feasibility. However, these methods have limitations; the former is only applicable to linearly constrained problems, and the latter works for problems with feasibility regions homeomorphic to a unit ball. Another critical issue with the approach in Donti et al. ( 2021) is that the completion step may fail during training when the equality system with some fixed variables becomes infeasible, as highlighted in Han et al. (2024) and Zeng et al. (2024) . Consequently, such an approach will not succeed during the training stage. To mitigate this issue, Han et al. (2024) proposed solving a projection problem if the completion step fails. However, the computationally expensive projection step may still be necessary during inference, which hinders its practical value.Learning-Based IPMs. Primal-dual IPMs are polynomial-time algorithms used for solving constrained optimization problems such as LPs and NLPs. Diehl (2019) and Zhang and Zhang (2022) addressed alternative current optimal power flow (ACOPF) applications and proposed using learning models, such as GNNs, to learn the mapping between ACOPF and its optimal solutions. These predicted solutions are then utilized as initial points to warm-start an interior point optimizer. However, even if these solutions are close to the optimal ones, they may not be well-centered with respect to the trajectory in IPMs, causing the optimizer to struggle in progressing towards feasibility and optimality (Forsgren, 2006) .3 Approach 
the classic ipm: 
 We focus on solving the following NLP (1):min x∈R n f (x) s.t. h(x) = 0 x ≥ 0 (1)where the functions f : R n → R and h : R n → R m are all assumed to be twice continuously differentiable. Problems with general nonlinear inequality constraints can be reformulated in the above form by introducing slack variables. We note that for simplicity, we assume all variables in (1) are non-negative, though NLPs with arbitrary variable bounds can also be handled effectively.Readers are referred to Appendix A for details.The primal-dual IPM stands as one of the most widely utilized approaches for addressing NLPs. It entails iteratively solving the perturbed Karush-Kuhn-Tucker (KKT) conditions (2) for a decreasing sequence of parameters µ converging to zero.∇f (x) + λ ⊤ ∇h(x) -z = 0 h(x) = 0 diag(z)diag(x)e = µe x, z ≥ 0 (2)where λ ∈ R m and z ∈ R n + denote the corresponding dual variables, diag(•) represents a diagonal matrix, and e is a vector of ones. Let F (x, λ, z) = 0 denote the system of nonlinear equations in (2).We then employ a one-step Newton's method to solve such a system, aiming to solve systems of linear equations (3).EQUATIONThe IPM commences with an initial solution (x 0 , λ 0 , z 0 ) such that x 0 , z 0 > 0. At iteration k, the linear system (3) defined by the current iterate(x k , λ k , z k ) is solved, with µ := σ (z k ) ⊤ x k /nbeing the perturbation parameter and constant σ ∈ (0, 1). A line-search filter step along the direction (∆x k , ∆λ k , ∆z k ) is then performed to ensure boundary condition satisfaction as well as sufficient progress towards objective value improvement or constraint violation reduction. This process iterates until convergence criteria, such as achieving optimality and feasibility within specified tolerances, are met. The IPM is guaranteed to converge to a KKT point with a superlinear rate. A pseudocode of the IPM is presented as Algorithm 1.Algorithm 1 The classic IPM Inputs: An initial solution (x 0 , λ 0 , z 0 ), σ ∈ (0, 1), k ← 0 Outputs: The optimal solution (x * , λ * , z * ) 1: while not converged do 2:Update µ k 3:Solve the systemJ k (∆x k ) ⊤ , (∆λ k ) ⊤ , (∆z k ) ⊤ ⊤ = -F k 4:Choose α k via a line-search filter method 5:(x k+1 , λ k+1 , z k+1 ) ← (x k , λ k , z k ) + α k (∆x k , ∆λ k , ∆z k ) 6: k ← k + 1 7: end whileWe note that, in classic IPMs, one typically reformulates the system (3) and then solves a reduced system of equations (i.e., augmented system) for greater efficiency. However, in this work, we are interested in the full systems since they are associated with smaller condition numbers (Greif et al., 2014) that are critical to the performance of our proposed approach. Additionally, various techniques have been proposed to enhance the robustness and efficiency of IPMs, including secondorder correction, inertial correction, and feasibility restoration. Interested readers are referred to Wächter and Biegler (2006) for further details about IPMs.The small number of iterations in IPMs does not always guarantee efficiency because, at times, IPMs encounter a high per-iteration cost of linear algebra operations. In the worst-case scenario, the cost of solving a dense optimization problem using a direct linear algebra method to solve the Newton equation system (3) may reach O(n 3 ) flops per iteration. This motivates us to avoid computing exact solutions to linear systems and instead focus on their approximations. Toward this goal, we consider the following least squares problem (4):EQUATIONwhere ∥•∥ denotes the Euclidean norm. If (3) is solvable, then an optimal solution to problem (4) is also the exact solution (∆x k ) ⊤ , (∆λ k ) ⊤ , (∆z k ) ⊤ ⊤ to system (3). Otherwise, we resort to an approximation of the latter. This perspective is similar to the inexact IPM (Bellavia, 1998; Dexter et al., 2022) .Assumption 1. At iteration k, we could identify some y k such thatEQUATION)where η ∈ (0, 1) and F 0 (x k , λ k , z k ) denotes F (x k , λ k , z k ) with µ = 0.To satisfy Assumption 1, the approximate solution y k has to be bounded and accurate enough, regardless of whether J k is invertible.Proposition 1 (Bellavia (1998)) .If (x k , λ k , z k ) is generated such that Assumption 1 is satisfied, let (x * , λ * , z * ) denote a limit point of the sequence {(x k , λ k , z k )}, then {(x k , λ k , z k )} converges to (x * , λ * , z * ) and F 0 (x * , λ * , z * ) = 0.Proposition 1 implies that if solutions with specified accuracy for linear systems in Step 3 are found, the IPM would converge.The problem ( 4) is an unconstrained convex optimization problem. Various L2O methods have been proposed to solve such problems (Chen et al., 2022a; Gregor and LeCun, 2010; Liu et al., 2023) . We will employ the LSTM networks in our L2O method for addressing problem (4), hence our approach is called "IPM-LSTM".Model Architecture. LSTM is a type of recurrent neural network designed to effectively capture and maintain long-term dependencies in sequential data (Yu et al., 2019) . LSTM networks are commonly considered suitable for solving unconstrained optimization problems due to the resemblance between LSTM recurrent calculations and iterative algorithms (Andrychowicz et al., 2016; Liu et al., 2023; Lv et al., 2017) .The LSTM network consists of T cells parameterized by the same learnable parameters θ. Each cell can be viewed as one iteration of a traditional iterative method, as illustrated in Figure 2 . Let ϕ(y) := 1 2 J k y + F k 2 for convenience. The t-th cell takes the previous estimate y t-1 and the gradient (J k ) ⊤ (J k y t-1 + F k ) as the input and outputs the current estimate y t :EQUATION)The T -th cell yields y T as an approximate solution to problem (4). As suggested by Andrychowicz et al. (2016) and Liu et al. (2023) , we utilize a coordinate-wise LSTM that shares parameters not only across different LSTM cells but also for all coordinates of y. Model Training. We train the proposed optimizer by finding the optimal θ in (7) on a dataset M of NLPs. Each sample in M is an instance of the optimization problem. During training, we apply the optimizer to each instance M ∈ M, performing K IPM iterations in the outer loop and T LSTM time steps in the inner loop, generating a sequence of iterates {(y 1 1 , ..., y 1 T ), ..., (y K 1 , ..., y K T )} where the superscript k denotes the IPM iteration number. We then optimize θ by minimizing the following loss function:min θ 1 |M| M ∈M 1 K K k=1 1 T T t=1 1 2 J k y k t (θ) + F k 2 M ,where the subscript M indicates that the corresponding term is associated with instance M . Clearly, our model training falls into the category of self-supervised learning. To mitigate memory issues caused by excessively large computational graphs, we employ truncated backpropagation through time after each IPM iteration during training, as done in Chen et al. (2022a) and Liu et al. (2023) .Preconditioning. The Hessian matrix of ϕ(y) is(J k ) ⊤ J k , whose condition number, κ((J k ) ⊤ J k ),is the square of that of J k . Consequently, κ((J k ) ⊤ J k ) can easily become very large. Since solving system (4) via LSTM networks emulates iterative first-order methods, the value of κ((J k ) ⊤ J k ) strongly affects the performance of LSTM networks. To address this issue, we employ a simple diagonal preconditioning technique that rescales the Hessian matrix (J k ) ⊤ J k using the Ruiz scaling method (Ruiz, 2001) to decrease its condition number.To further enhance the solution quality, we propose a two-stage framework that initially obtains a nearoptimal and well-centered primal-dual solution via IPM-LSTM and then utilizes this approximate solution to warm-start an interior point solver. In this study, we select IPOPT (Wächter and Biegler, 2006) , an IPM-based solver renowned for its robustness and efficiency in optimizing NLPs.Our two-stage framework works as follows: Given an NLP (1), we generate an initial point (x 0 , λ 0 , z 0 ) and formulate the least squares problem (4). Subsequently, the trained LSTM network with T cells solves this problem and returns a search direction. We then employ the simple fractional-to-boundary method (Wächter and Biegler, 2006) to determine the step size and reach the new iterate. This procedure is iterated K times, resulting in a primal-dual solution (x K , λ K , z K ). Finally, the obtained solution serves as the warm-start solution for IPOPT, leading to the optimal solution x * upon IPOPT convergence. 
experimental settings: 
 We evaluate our approach and compare its performance against traditional methods as well as L2O algorithms for solving various types of NLPs. Furthermore, we also quantify the warm-starting effect of our proposed two-stage approach. Our code is available at https://github.com/NetSysOpt/IPM-LSTM.Baseline Algorithms. In our experiments, we denote our algorithm by IPM-LSTM and compare it against both traditional optimizers and L2O algorithms. The traditional optimizers considered are: (i) OSQP (Stellato et al., 2020) Convex QPs. We consider convex QPs with both equality and inequality constraints:EQUATIONwhereQ 0 ∈ S n + , p j ∈ R n , q j ∈ R, x L i ∈ R ∪ {-∞} and x U i ∈ R ∪ {+∞}.We conduct experiments on two groups of QPs, each instance with 200 variables, 100 inequalities and 100 equalities. The first group is generated in the same way as Donti et al. (2021) , where only the right hand sides of equality constraints are perturbed while the second one considers perturbation for all model parameters. Let "Convex QP (RHS)" denote the former and "Convex QPs (ALL)" denote the latter. It is noteworthy that, in line with Donti et al. (2021) , we also investigate the performance of IPM-LSTM and baseline algorithms on smaller-scale convex QPs. Interested readers are directed to Appendix D for details. larger objective values. Solutions identified by IPM-LSTM showed mild constraint violations but yielded superior objective values very close to the optimal values. Clearly, IPM-LSTM effectively balances feasibility and optimality in the returned solutions. This comes at the cost of longer runtime compared to the L2O baseline algorithms, as the baselines typically employ simple multi-layer perceptrons, whereas IPM-LSTM utilizes a neural network with several dozen LSTM cells.Since IPM-LSTM is designed to provide interior point optimizers with high-quality initial points, we fed the returned primal-dual solution pair to IPOPT and reported the performance in Table 1 . For comparison, we also provided IPOPT with initial points generated from other L2O algorithms. The columns labeled "Ite." and "Time (s)" under "IPOPT (warm-start)" indicate the number of iterations and solver time, respectively, while the column "Total Time (s)" represents the cumulative time for running both the L2O algorithms and IPOPT. The final column, "Gain (Ite/Time)", shows the reduction in iteration number and solution time, with the default IPOPT iteration number listed in the "Ite." column for reference. When initial solutions from IPM-LSTM and most L2O baseline algorithms (except DeepLDE) were used, IPOPT converged with fewer iterations and reduced solution time. Notably, IPM-LSTM achieved the most significant reduction in iterations, from 12.5 to 7.2, and decreased the average solver time from 0.642 seconds to 0.37 seconds. Including the computational time for IPM-LSTM, the total runtime was 0.545 seconds, reflecting a 15.1% reduction in time. It is worth noting that while IPM-LSTM did not yield the maximum solution time reduction, this was due to its relatively high computational expense.To our knowledge, there is no existing representation learning approach for general convex QPs. Since the aforementioned L2O baseline algorithms depend on specific representations of QPs, they are not applicable to "Convex QPs (ALL)". Therefore, we only provide results for OSQP, IPOPT, and IPM-LSTM. We also report computational results averaged across 833 instances in the test set "Convex QPs (ALL)", presented in Table 1 . The results demonstrate that IPM-LSTM can identify high-quality solutions for general convex QPs, and using these solutions as initial points can reduce iterations by 35.7% and solution time by 7.5%.Convex QCQPs. We now turn to convex QCQPs with both equaltity and inequality constraints:min x∈R n 1 2 x ⊤ Q 0 x + p ⊤ 0 x s.t. x ⊤ Q j x + p ⊤ j x ≤ q j j = 1, • • • , l p ⊤ j x = q j j = l + 1, • • • , m x L i ≤ x i ≤ x U i i = 1, • • • , nwhereQ j ∈ S n + , p j ∈ R n , q j ∈ R, x L i ∈ R ∪ {-∞} and x U i ∈ R ∪ {∞}.Similar to our experiments on convex QPs, we also consider two groups of convex QCQPs, each with 200 variables, 100 inequality constraints, and 100 equality constraints. The first group (denoted as "Convex QCQPs (RHS)") is generated as described in Liang et al. (2023) , with perturbations only to the right-hand sides of the equality constraints. The second group (denoted as "Convex QCQPs (ALL)") considers perturbations to all parameters. We also refer readers to Appendix D for computational experiments on smaller-sized convex QCQPs.We omit OSQP and LOOP-LC since the former cannot handle QCQPs, while the latter is only applicable to linearly constrained problems. We evaluate IPM-LSTM and compare it against the remaining baseline algorithms. The computational results are reported in Table 2 . Again, solutions produced by NN and PDL exhibit significant constraint violations, while those from DC3 and H-Proj are of high quality in terms of feasibility and optimality. Once more, the solutions produced by DeepLDE were deemed feasible but exhibited inferior objective values. Conversely, our approach, IPM-LSTM, produced solutions with superior objective values albeit with minor infeasibility. Compared to the baseline algorithms, utilizing solutions from IPM-LSTM to warm-start IPOPT resulted in the most substantial reduction in iterations.As the aforementioned L2O baseline algorithms are not applicable to "Convex QCQPs (ALL)", we only report computational results for IPOPT and IPM-LSTM in Table 2 . The IPM-LSTM approach produced high-quality approximate solutions to convex QCQPs, and warm-starting IPOPT with these solutions accelerated IPOPT by 11.4%, with a 33.1% reduction in iterations.Non-convex QPs. We now consider non-convex QPs of exactly the same form as (8) but with Q 0 being indefinite. We take 8 representative non-convex QPs from the datasets Globallib and RandQP (Chen and Burer, 2012) , each with up to 50 variables and 20 constraints, and perturb the relevant parameters for instance generation. Details can be found in Appendix D.2. Among all the baselines, only IPOPT is applicable to these general non-convex QPs. Hence, we report computational results for IPOPT and IPM-LSTM in Table 3 . IPOPT solved these instances to local optimality, while IPM-LSTM identified high-quality approximate solutions very efficiently. Using these primal-dual approximations to warm-start IPOPT resulted in an iteration reduction of up to 63.9% and a solution time reduction of up to 70.5%. Simple non-convex programs. Following the approach outlined in Donti et al. ( 2021), we consider a set of simple non-convex programs where the linear objective term in ( 8) is substituted with p ⊤ 0 sin(x). These instances were generated using the same methodology as Donti et al. (2021) . We subject these problems to evaluation using both IPM-LSTM and baseline algorithms. Once again, the computational results underscore the high-quality solutions obtained by IPM-LSTM and its superior warm-starting capability. Detailed results are provided in Appendix D.4.In Assumption 1, we posit that the linear systems (3) can be solved with an acceptable residual (Condition (5)) and that the solutions are properly bounded (Condition ( 6)). Although the LSTM network cannot guarantee the satisfaction of these conditions, we empirically assess the validity of Assumption 1 in IPM-LSTM. We plot the progress of J k y k + F k , η (z k ) ⊤ x k /n, y k , and F 0 (x k , λ k , z k ) as the IPM iterations increase in Figure 3 (a) and 3(b). Condition ( 5) is mostly satisfied except during the first few iterations, while Condition (6) is strictly satisfied across all IPM iterations. To assess the precision of the approximate solutions as the LSTM time steps increase, we plot the progress of the residual for linear systems (3) in Figure 3(c) . At IPM iteration k, the residual decreases monotonically towards 0 as the LSTM time steps increase, indicating that LSTM networks can produce high-quality approximate solutions to system (3). The approximation quality improves with the number of IPM iterations. As shown in Figure 3 (d), the primal objective decreases monotonically towards the optimal value with increasing IPM iterations, empirically demonstrating the convergence of IPM-LSTM. Note that the condition number κ((J k ) ⊤ J k ) becomes quite large in the later IPM iterations, which can adversely affect the performance of LSTM networks in obtaining approximations to systems (3). Consequently, we terminate IPM-LSTM after a finite number of iterations. Further analysis regarding the number of IPM iterations and LSTM time steps is presented in Appendix D.5. Additionally, we include the performance of IPM-LSTM under various hyperparameter settings in Appendix D.5.  
limitations and conclusions: 
 In this paper, we present a learning-based IPM called IPM-LSTM. Specifically, we propose approximating solutions of linear systems in IPMs by solving least square problems using trained LSTM networks. We demonstrate that IPMs with this approximation procedure still converge. The solutions returned by IPM-LSTM are used to warm-start interior point optimizers. Our computational experiments on various types of NLPs, including general QPs and QCQPs, showcase the effectiveness of IPM-LSTM and its ability to accelerate IPOPT. Although IPM-LSTM generates high-quality primal-dual solutions, it is relatively computationally expensive due to the utilization of multi-cell LSTM networks. In future endeavors, we aim to investigate the efficacy of employing low-complexity neural networks to approximate solutions of linear systems within IPMs. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Please refer to Section 4.1 and Appendix C.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Please refer to Section 4.2.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Please refer to Section 4.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The research presented in this paper adheres to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper does not present any such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Please refer to Section 4.1.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: The general NLP considered in this paper can be formulated as:min x∈R n f (x), s.t. g(x) + s = 0 h(x) = 0 s ≥ 0 x i ≥ x L i , i ∈ I L x i ≤ x U i , i ∈ I U (9)where f : R n → R, g : R n → R mineq , and h : R n → R meq are twice continuously differentiable functions; s ∈ R mineq is the slack variable corresponding to the inequality constraints; I L = {i :x L i ̸ = -∞} and I U = {i : x U i ̸ = ∞}. The Lagrangian function is defined as: L x, η, λ, s, z L , z U := f (x) + η ⊤ g(x) + λ ⊤ h(x) - i∈I L z L i x i -x L i - i∈I U z U i x U i -x i (10)where η ∈ R mineq and λ ∈ R meq are the corresponding dual variables. The perturbed KKT conditions are given by:EQUATIONThe nonlinear system extracted from the KKT conditions is represented as:F (x, η, λ, s, z L , z U ) = 0 (12)whereF (x, η, λ, s, z L , z U ) =        ∇f (x) + η ⊤ ∇g(x) + λ ⊤ ∇h(x) -z L + z U g(x) + s diag(η)diag(s)e -µe h(x) diag(z L )diag(x -x L )e -µe diag(z U )diag(x U -x)e -µe        = 0The Jacobian matrix of F (x, η, λ, s, z L , z U ) is:J(x, η, λ, s, z L , z U ) :=        ∇ 2 L(x, η, λ, z L , z U ), ∇g(x) ⊤ , ∇h(x) ⊤ , 0, -I, I ∇g(x), 0, 0, I, 0, 0 0, diag(s), 0, diag(η), 0, 0 ∇h(x), 0, 0, 0, 0, 0 diag(z L ), 0, 0, 0, diag(x -x L ), 0 -diag(z U ), 0, 0, 0, 0, diag(x U -x)       whereEQUATIONFor convenience, we define the updates of primal and dual variables as:EQUATION)Then the linear system obtained from one-step Newton's method is shown as:J x, η, λ, s, z L , z U y = -F x, η, λ, s, z L , z U (15)Initial points. If dual variables η, s, z L , and z U exist, their initial values are set to 1. If the dual variable λ exists, its initial value is set to 0. The initial value of the primal variable x can be formulated as:EQUATION)The step length. If the dual variables η, s, z L and z U exist, their step lengths are chosen as follows:α η := sup {α ∈ (0, 1] | η i + α∆η i ≥ 0, i = 1, • • • , m ineq } (17) α s := sup {α ∈ (0, 1] | s i + α∆s i ≥ 0, i = 1, • • • , m ineq } (18) α z L := sup α ∈ (0, 1] | z L i + α∆z L i ≥ 0, i ∈ I L (19) α z U := sup α ∈ (0, 1] | z U i + α∆z U i ≥ 0, i ∈ I U (20)If x is bounded, the step sizes for x and λ are chosen to be equal as:α λ , α x := sup α ∈ (0, 1] | xi + α∆xi ≥ x L i , i ∈ I L ; xi + α∆xi ≤ x U i , i ∈ I U (21)If x is unbounded but inequality constraints exist, the step lengths for x and λ are chosen the same as the step length for s. If x is unbounded and there are no inequality constraints, the step lengths for x and λ are chosen to be 1. In our experiments, additional line search procedures for step lengths, as described in Bellavia (1998) , were not implemented in order to save computational time.Proof.Step 1. Let δ > 0, suppose that (x k , λ k , z k ) ∈ N δ (x * , λ * , z * ) := {(x, λ, z) | ∥(x, λ, z) -(x * , λ * , z * )∥ < δ}. Define σ := min{σ k } and η := min{η k }, where σ k and η k are chosen to satisfy the update rule (11) in Bellavia (1998) . Then from the Assumption 1:EQUATION)By using the steplength α k obtained from the line search method of Bellavia (1998) and defining:ηk = 1 -α k 1 -σ k -η k (23)we obtain:∥p k ∥ = ∥α k y k ∥ = ∥ (1-η k )α k y k 1-(1-α k (1-σ k -η k )) ∥ = 1-η k 1-σ k -η k ∥y k ∥ ≤ (1 -ηk ) 1+σ k +η k 1-σ k -η k ∥F 0 (x k , λ k , z k )∥ ≤ (1 -ηk )Γ∥F 0 (x k , λ k , z k )∥ (24)where Γ = 1+ηmax 1-ηmax , (σ k + η k ) ∈ (0, η max ) and η max ∈ (0, 1). Then, there exists a constant Γ independent of k such that (24) holds, whenever y k is bounded by(1 + σ k + η k )∥F 0 (x k , λ k , z k )∥.Hence, from Theorem 3.5 of Eisenstat and Walker (1994) , it follows that (x k , λ k , z k ) → (x * , λ * , z * ).Step 2. In this step, we aim to prove that the step size α k of the inexact IPM is bounded away from 0. Given that Assumption 1 is satisfied, it follows that ∥J k y k + F k ∥ and ∥y k ∥ are both bounded. Then we assume the first two equations of F 0 (x, λ, z) are Lipschitz continuous gradient with constant L. The remaining proofs are consistent with Theorem 3.2 in Bellavia (1998) .Step 3. The line search method of Bellavia (1998) enables α k to satisfy:EQUATIONwhere β ∈ (0, 1). Therefore {∥F 0 (x k , λ k , z k )∥} is decreasing and bounded, hence, it is convergent. Based on Assumption 1, (24) holds. Furthermore, we assume ∥F 0 (x k , λ k , z k )∥ ̸ = 0 and δ is chosen sufficiently small so thatEQUATIONthen based on Lemma 5.1 of Eisenstat and Walker (1994) , the loop of backtracking line search of Bellavia (1998) will terminate withEQUATIONwhere θ ∈ (0, 1) is a control parameter. This implies that the series ∞ k=0(1ηk ) is divergent. By applying (25) iteratively, we haveEQUATIONSince β > 0 and 1ηj ≥ 0, the divergence of series∞ j=0 (1 -ηj ) implies ∥F 0 (x k , λ k , z k )∥ → 0.The key hyperparameters for each task, including K, T , and the hidden dimension of LSTM networks, are listed in Table 4 . Generally, IPM-LSTM demonstrates improved performance with larger values of K and T , although this comes at the cost of increased computational time. Increasing T could enhance the quality of solutions to the linear system. In this study, K is maintained at the same value for both training and testing. Further analysis can be found in Appendix D.5. • As shown by Figure 5 (d), the number of samples in the test set does not affect the performance of LSTM for solving linear systems.Table 10 : The detailed values of Figure 3(a) . IPM Ite. ∥J k y k + F k ∥ η[(z k ) ⊤ x k ]/The LSTM approach for solving linear systems is negatively affected by their large condition numbers.To demonstrate this, we consider the least squares problemmin y∈R l ϕ(y) := 1 2 J k y + F k 2 (31)We utilize a first-order method (e.g., steepest descent method) to minimize ϕ(y) and achieve a linear convergence rate (Nocedal and Wright, 1999) , i.e.,EQUATIONAs we discussed in Section 3.3, since solving linear systems via LSTM networks emulates iterative first-order methods, thus the value of κ J k affects the performance of LSTM networks. However, LSTM networks can empirically achieve a faster convergence rate than traditional first-order algorithms when solving the same least squares problems as shown in the computational studies (Section 3.1) of Andrychowicz et al. (2016) . In order to alleviate the effect of large condition numbers, as discussed in Section 3.3, we have employed preconditioning techniques. To illustrate its effect, we conduct experiments on the simple non-convex programs, and report κ J k and their values after preconditioning (in parantheses) across several IPM iterations (e.g., 1 st , 10 th , 20 th , 50 th , 100 th ) in Table 11 . We can conclude that the condition numbers κ J k remain within reasonable magnitudes even during the later IPM iterations, and are significantly reduced after applying the preconditioning technique. (200, 100, 100) 91.5 (99.8) 157.1 (1114.0) 205.8 (1441.1) 326.2 (2398.3) 488.3 (3667.8) Non-convex Programs (ALL) (200, 100, 100) 72.1(75.7) 175.4 (1143.4) 184.5 (1352.7) 249.5 (2016.6) 368.4 (3015.3) Answer: [Yes] Justification: Please refer to Section 4.1 for data. Our code will be available to public once our work is accepted, as mentioned in Section 1.• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: This paper proposed to replace the linear system solver used in the inner loop of the interior point method (IPM) with an LSTM for solving general non-linear programs. Most of the reviewers agreed that this paper is technically solid. Overall, this paper is well-structured with a logical flow, and the idea of integrating machine learning techniques into interior point methods is valuable for this community. All the reviews tend to accept the paper during the reviewer-author discussion phase. Therefore, I recommend this paper to the NeurIPS 2024 conference.

Reviews: review: paper_summary: paper_summary: This paper proposed to replace the linear system solver used in the inner loop of interior point method (IPM) with an LSTM for solving general non-linear programs. The LSTM is trained in a unsupervised manner to minimize a unconstrained least square objective derived from the KKT conditions. The proposed framework, IPM-LSTM, can be used in an end-to-end way, or to warm-start IPM so that the number of overall outer IPM iterations can be reduces and thus the solving time decreases. The authors theoretically proved that, as long as the trained LSTM achieves certain level of accuracy, the IPM will converge. The authors conducted empirical experiments on a variety of non-linear programs to verify the effectiveness of IPM-LSTM and that it approximately satisfied the assumption in the theoretical analysis.
strength_weakness: strength_weakness: There are three major issues that undermine the robustness of this work:

1. The linear system solver used for IPOPT is unclear, which significantly impacts the solver's overall performance. While MUMPS is the default, HSL typically performs better, achieving 2-3 times faster results. This performance surpasses that of IPOPT when warm-started with the IPM-LSTM solution.

2. There is no analysis of how the performance changes with varying numbers of iterations in the inner LSTM, which decides the performance-efficiency trade-off of IPM-LSTM. Longer inner iterations (deeper LSTM) may improve the quality of the solutions but increases computation costs. However, improved solution quality may also decrease the number of outer iterations needed and thus reduce the overall solving time. Moreover, increased LSTM depth can make the training process more difficult. Overall, thise is tricky part and should be empirically investigated more carefully.

3. Although LSTM benefits from batched processing—a potential strength of IPM-LSTM accelerated by GPU—this work fails to provide empirical evidence supporting this property.
questions: questions: See the weaknesses section.
limitations: limitations: See the weaknesses section.
paper_summary: paper_summary: This paper introduces a method called IPM-LSTM, which integrates machine learning techniques into interior point methods (IPM). Specifically, the authors propose training a RNN model, LSTM, to quickly approximate the solution of linear systems within IPM. This approach is numerically validated on several convex and nonconvex QPs.
strength_weakness: strength_weakness: My main concern about this paper is Assumption 1, which requires the accuracy of the linear system solution to increase as the number of iterations $k$ increases. Based on this assumption, exact convergence is derived, as shown in Proposition 1. However, the empirical results in Section 4 indicate that IPM-LSTM does not achieve exact optimality, revealing a gap between theory and practice. To address this gap, I suggest:
- Reporting the error of LSTM at each iteration. This would provide readers with an understanding of how accurately the LSTM performs. Additionally, it would be beneficial to report the relationship between the error of the linear system solution with the size, training, and testing overhead of the LSTM.
- Using a log scale y-axis for Figure 3a for better precision. For example, 0.01 is 10 times greater than 0.001, but this difference is not reflected in the linear scale of Figure 3a.
- Modifying Assumption 1 to better align with practice. For example, assuming a fixed error on the right-hand sides of equations (5) and (6). Based on this relaxed assumption, a result similar to Proposition 1 could be derived, but with a fixed error on the limit of $(x^k,\lambda^k,z^k)$. The relationship between the allowed error in Assumption 1 and the propagated eventual error in Proposition 1 would sufficiently describe the performance of IPM-LSTM.
questions: questions: Refer to "Weaknesses".
limitations: limitations: This theoretical paper appears to have no potential negative societal impact.
paper_summary: paper_summary: This paper introduces IPM-LSTM, an approach integrating Long Short-Term Memory (LSTM) neural networks with Interior Point Methods (IPMs) to solve Nonlinear Programs (NLPs). The key innovation lies in approximating solutions to linear systems within IPMs using LSTMs, aiming to accelerate the convergence of classic IPM solvers. This approach leverages the Learning to Optimize (L2O) paradigm, presenting a two-stage framework where LSTM-generated solutions are used to warm-start an IPM solver. The authors compare the IPM-LSTM against traditional solvers and recent L2O methods across various NLPs, including Quadratic Programs and Quadratically Constrained Quadratic Programs. The proposed method reportedly reduces iterations by up to 60% and solution time by up to 70%.
strength_weakness: strength_weakness: 1. The IPM-LSTM, which uses LSTM to iteratively solve linear systems at each IPM iteration, repeats K times and feeds it into the IPM solver as a warm-start point, needs more justification.
   1. The decision to use the L2O approach for solving a least squares problem (problem (4)) is not adequately justified. According to Assumption 1, the approximated solution needs to be bounded and accurate enough to guarantee the convergence of the outer loop. From the theoretical side, it is unclear how hard it is to satisfy those conditions by L2O approaches. From the empirical side, as shown in Figure 3(a), the accuracy condition is not always satisfied. If assumption 1 can not be guaranteed, the convergence of approximated IPM can not be guaranteed according to Prop. 1. 
   2. The approach of using an approximated IPM solution instead of directly generating a warm-start point using either NN or L2O raises questions about efficiency and effectiveness. Previous works [1-4] have shown that direct prediction of warm-start points can be more straightforward and computationally efficient. Besides, the convergence of approximated IPM can not be guaranteed, which also raises concerns about the quality of such a warm-start point. The authors need to provide more discussion or experimental comparisons to justify their more complex, iterative approximation method.

[1] R. Sambharya, G. Hall, B. Amos, and B. Stellato, "End-to-End Learning to Warm-Start for Real-Time Quadratic Optimization", arXiv preprint arXiv:2212.08260, 2022.

[2] Sambharya R, Hall G, Amos B, et al. Learning to warm-start fixed-point optimization algorithms[J]. Journal of Machine Learning Research, 2024, 25(166): 1-46.

[3] F. Diehl, "Warm-Starting AC Optimal Power Flow with Graph Neural Networks", in Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS) Workshop, Vancouver, BC, Canada, Dec. 8 - 14, 2019.

[4] K. Baker, "Learning Warm-Start Points for AC Optimal Power Flow", in Proceedings of IEEE 29th Machine Learning for Signal Processing Conference, Pittsburgh, PA, USA, Oct. 13 - 16, 2019.
questions: questions: 1. In the experiments, why not directly use the simple NN prediction as a warm-start point for the IPM solver?
2. In the simulation part, as the solutions obtained by the proposed IPM-LSTM approach are used as a warm-start point to IPM, why are the equality constraints not satisfied? The authors are suggested to explain the experimental setup clearly.
3. In Table I, the proposed IPM-LSTM approach has a larger constraint violation with a longer solution time as compared to the OSQP algorithm. Does this mean the proposed approach can not exceed state-of-the-art methods for solving convex QP problems?
limitations: limitations: The authors have discussed the limitation in Sec. 5.

Comments: title: Please Engage in Discussion
Response to authors
Thanks for your rebuttal
Thanks for your rebuttal
comment: Dear Reviewer,

Thank you for your time and efforts throughout the review period. Please read the authors' rebuttal as soon as possible and indicate if their responses have addressed all your concerns. 

Best,

Your AC
We really appreciate your efforts for improving our work and are happy to know that all of your concerns have been addressed.
I greatly appreciate the detailed response and additional experiments. My concerns are fully addressed. The new experimental results regarding the accuracy of solving linear systems look pretty promising. I would like to upgrade my score.
Thanks for catching this. It is a typo in our manuscript. Indeed, all equality constraints in convex QCQPs only involve linear terms. We fix this in our revised manuscript.

We really appreciate your efforts for improving our work and are happy to know that your concerns have been addressed.
Thanks for your additional explanations and experiments, which address the raised concerns. I will adjust my score accordingly.

One additional comment: for convex QCQP, why are there quadratic equality constraints? They should be linear for convex problems.
>I have reviewed the rebuttal and acknowledge that accelerating linear system solving by NN within IPM iterations is a reasonable approach.

Thanks for your efforts and acknowledgement.

>While the use of NNs for solving linear systems is not new [1], integrating them within IPM solvers introduces specific challenges related to convergence and accuracy.

Thanks for sharing with us this work [1]. The authors of [1] focused on using NNs to solve **positive-definite linear systems of equations** while **linear systems in IPMs are indefinite**. We incorporate this work in our revised manuscript.

>I still have some concerns about the LSTM approach to solving linear systems with high accuracy.


>1. The provided theoretical guarantees for the Learning to Optimize (L2O) approach are limited to convex problems.

Thanks for raising this point. In this work, we focus on developing learning-based IPMs to address general nonlinear programs which might be non-convex. 
- The **IPM itself (i.e., the outer loop of IPM-LSTM) would guarantee that the returned solutions are locally optimal** as long as linear systems are solved to specified tolerances, as shown by Proposition 1 in our manuscript. 
- To solve linear systems via LSTM, we convert them to least squares problems of the form $\text{min}_y \frac{1}{2}|| J^ky+F^k ||^2$, which are **unconstrained convex programs**. As implied in Theorem 1 of [2], **solutions provided by** properly parameterized **LSTM networks would converge to one of the minimizers of convex optimization problems**.

Hence, from a theoretical point of view, **the proposed IPM-LSTM approach can solve both convex and non-convex programs to optimality**. We incorporate the above points in the revised manuscript.


>2. Many practical applications, especially those involving KKT systems for general non-convex problems, deal with ill-conditioned linear systems. Can the LSTM approach effectively handle such ill-conditioned systems.

Thank the reviewer for raising such an interesting point. Let $\kappa(\cdot)$ denote the condition number of a matrix.
- **The LSTM approach for solving linear systems is negatively affected by their large condition numbers.** To demonstrate this, we consider the least squares problem 
\begin{align}
\underset{y \in \mathbb{R}^m}{\text{min}} \; f(y) := \frac{1}{2}|| J^ky+F^k ||^2.
\end{align}
We utilize a first-order method (say the steepest descent method) to minimize $f(y)$ and achieve a linear convergence rate [3], i.e., 
\begin{align}
f\left(x_{k+1}\right) - f\left(x^{\star}\right)   \leq   \left(1-\frac{2}{(\kappa(J^k))^2+1}\right)^{2} \left( f(x_{k})-f(x^{\star})\right).
\end{align}
As we discussed in the **Preconditioning** part of our manuscript, since solving linear systems via LSTM networks emulates iterative first-order methods, thus the value of $\kappa(J^k)$ affects the performance of LSTM networks.
- As shown in the computational studies (Section 3.1) of [4], **LSTM networks can empirically achieve a faster convergence rate than traditional first-order algorithms** when solving the same least squares problems.
- To alleviate the effect of large condition numbers, as discussed in the **Preconditioning** part of our manuscript, **we have employed preconditioning techniques**.


>3. It is also unclear whether the used simple non-convex problem is ill-conditioned or not.

Thanks for your suggestion. For the simple non-convex programs used in our experiment, we report $\kappa(J^k)$ and their values after preconditioning (in parantheses) across several IPM iterations (say 1, 10, 20, 50, 100) in the following table.
- The condition numbers **$\kappa(J^k)$ remain within reasonable magnitudes**, even during the later IPM iterations.
- Applying the **preconditioning** technique indeed significantly **reduces the condition numbers** for those non-convex problems.



| Instance     | $1^{\text{st}}$ Iter. | $10^{\text{th}}$  | $20^{\text{th}}$  | $50^{\text{th}}$  | $100^{\text{th}}$ Iter. |
| ------------------------- | ------- | --------- | --------- | --------- | ---------- |
|**Non-convex Programs (RHS) (100, 50, 50)**|53.8(59.8)|126.2(580.7)|153.1(711.2)|208.7(1004.8)|348.4(1860.1)|
|**Non-convex Programs (ALL) (100, 50, 50)**|55.4(59.8)|113.6(517.0)|139.8(658.5)|214.0(1190.9)|329.2(1859.9)|
|**Non-convex Programs (RHS) (200, 100, 100)**|91.5(99.8)|157.1(1114.0)|205.8(1441.1)|326.2(2398.3)|488.3(3667.8)|
|**Non-convex Programs (ALL) (200, 100, 100)**|72.1(75.7)|175.4(1143.4)|184.5(1352.7)|249.5(2016.6)|368.4(3015.3)
     
[2] Liu, Jialin, et al. (2023) "Towards constituting mathematical structures for learning to optimize." ICML.

[3] Nocedal, J., & Wright, S. J. (Eds.). (1999). Numerical optimization. New York, NY: Springer New York.

[4] Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., ... & De Freitas, N. (2016). Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29.
I have reviewed the rebuttal and acknowledge that accelerating linear system solving by NN within IPM iterations is a reasonable approach.

While the use of NNs for solving linear systems is not new [1], integrating them within IPM solvers introduces specific challenges related to convergence and accuracy. 

I still have some concerns about the LSTM approach to solving linear systems with high accuracy.

- The provided theoretical guarantees for the Learning to Optimize (L2O) approach are limited to convex problems.
- Many practical applications, especially those involving KKT systems for general non-convex problems, deal with ill-conditioned linear systems. Can the LSTM approach effectively handle such ill-conditioned systems.
- It is also unclear whether the used simple non-convex problem is ill-conditioned or not.

---

* [1] Kaneda, A., Akar, O., Chen, J., Kala, V. A. T., Hyde, D., & Teran, J. (2023, July). A deep conjugate direction method for iteratively solving linear systems. In International Conference on Machine Learning (pp. 15720-15736). PMLR.