FSNet: Feasibility-Seeking Neural Network for Constrained Optimization with Guarantees
Authors: Hoang T. Nguyen
Priya L. Donti
Venue: NeurIPS 2025 poster ()
Keywords: deep learning
optimization
constraint enforcement
Primary area: optimization
TL;DR: This work proposes a new framework to solve parametric optimization problems with guarantees on feasibility and convergence.

Abstract: Efficiently solving constrained optimization problems is crucial for numerous real-world applications, yet traditional solvers are often computationally prohibitive for real-time use. Machine learning-based approaches have emerged as a promising alternative to provide approximate solutions at faster speeds, but they struggle to strictly enforce constraints, leading to infeasible solutions in practice. To address this, we propose the Feasibility-Seeking Neural Network (FSNet), which integrates a feasibility-seeking step directly into its solution procedure to ensure constraint satisfaction. This feasibility-seeking step solves an unconstrained optimization problem that minimizes constraint violations in a differentiable manner, enabling end-to-end training and providing guarantees on feasibility and convergence. Our experiments across a range of different optimization problems, including both smooth/nonsmooth and convex/nonconvex problems, demonstrate that FSNet can provide feasible solutions with solution quality comparable to (or in some cases better than) traditional solvers, at significantly faster speeds.

FSNet: Feasibility-Seeking Neural Network for Constrained Optimization with Guarantees
Hoang Nguyen
Priya L. Donti
Massachusetts Institute of Technology Cambridge, MA, USA hoangh@mit.eduu
Massachusetts Institute of Technology Cambridge, MA, USA donti @mit.edm
Abstract
Efficiently solving constrained optimization problems is crucial for numerous real- world applications, yet traditional solvers are often computationally prohibitive for real-time use. Machine learning- basedd approaches have emerged as a promising alternative to provide approximate solutions at faster speeds, but they struggle to strictly enforce constraints, leading infeasible solutions in practice. To address this, we propose the Peasibility-Seekinn Neural Network (FSNet), which integrates feasibility-seeking step directly into its solution procedure ensure constraint satisfaction. This Grasibility-scekinn step solves an unconstrained optimization problem that minimizes constraint violations in differentiable manner, enabling end-to-end training and providing guarantees on feasibility and convergence. Our experiments across a range of different optimization problems, including both smooth/nonsmooth and convex/nonconvex problems, demonstrate that FSNet can provide feasible solutions with solution quality comparable to (or in some cases better than) traditional solvers, at significantly faster speeds.'
Introduction
Machine learning (ML) has emerged as promising technique to provide fast surrogate solvers for parametric optimization problems [1, 2]. Specifically, given a set of optimization problems that share fixed structure but whose parameters vary between instances, ML can be used to learn approximate, fast-to-evaluate mappings from parameters optimization solutions. With this approach, many expensive real-world optimization problems can now be solved using ML in fraction of second, with promising examples in electric power systems [3, 4], building energy management systems [5], communication networks [6], robotic planning and control [7. 8). and supply chain management [9]. Unfortunately, although fast at producing solutions, ML approaches often struggle to satisfy problem constraints, resulting in solutions that infeasible and potentially unusable, particularly in safcty-critics applications. While several censtraintt-nffremmeet approaches have been proposed mitigate this problem, including penalty methods [10], projection- methods [11, 12], and iterative methods [13, 14), these methods often suffer from poor solution quality, expensive computational costs, and/or lack of guarantees.
To address these challenges, we propose the Feasibility -Seeiity--eeiinggg Neural Network (FSNet), general framework for solving constrained optimization problems using neural networks (NNs). Our key idea is to incorporate a differentiable Feasibility-seeking step into the NN's training and inference processes to minimize constraint violations while still enabling end-to-end training. In particular, this sesiibility-sceking step uses the NN's prediction as the starting point for constraint violation minimization problem, which is solved using iterative optimization solver to ultimately produce
Thee code is available at ttttps:/githhi cotp:ll...oo//////////// www.//////.b.//////o./ooo.....o/
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

feasible solution. This solution is evaluated via an unsupervised loss function that captures the objective function of the target optimization problem and the distance between the pre- and post- feasibility points. The full procedure is trained end-to-end, including backpropagation through the unrolled iterations of the optimization solver used in the feasibility-seekinn step.
Our main contributions are summarized as follows:
General framework for constraint enforcement in NNs. We introduce a principled and general framework, FSNet, that enables NNs to predict solutions satisfying both equality and inequality constraints in continuous optimization problems. The framework accommodates both convex and nonconvex constraints, making applicable to many problem domains.
Theoretical guarantees on feasibility and convergence. Under boundedness, L- smoothness, and the Polyak-tojjsiewiczz (PL) condition, FSNet guarantees exponentially fast convergence to a feasible point and provably reaches a stationary point in the parameter space when being trained using stochastic gradient descent, even when backpropagation is truncated. We also specify conditions under which FSNet's predictions provably coincide with a true minimizer of the original constrained optimization problem.
Strong empirical performance across various classes of problems. Across convex and nonconvex problems, FSNet produces feasible solutions and achieves small optimality gaps (less than 0.2% in convex cases), while providing solutions orders-of-man -fagnnnitddu faster than traditional solvers. In nonconvex cases, we show that FSNet can even find better local solutions than traditional solvers, at significantly greater speed.
2 Related Work
ML for optimization. There has been large body of literature leveraging ML to provide fast solutions to optimization problems, split into two major classes of approaches (see also reviews [4, 15, 16). The first class of approaches entails using ML to learn end-to-end surrogate models, in which problem parameters that vary between optimization problem instances are treated as inputs to the ML model [1, 2]. This includes supervised learning methods that require a training dataset of solved instances generated via traditional solvers [10, 17, 18], as well as sedffssupevieed//ssuevvee approaches that directly embed the optimization objective and/or constraint functions into the ML loss function [3, 13, 19, 20). In this work, adopt the self-supervised paradigm, which has been favored in recent work as avoids the large amount of time and resources required to create supervised training datasets [21]. The second class of ML-for-optimiizatoon approaches involves using ML to improve the performance of other optimization solvers. For instance, ML has been used to approximate algorithm steps for convex optimization to achieve guaranteed fast convergence [22], provide warm start points to accelerate fixed-point optimization algorithms [23], or approximate or remove complex or redundant constraints to reduce computational complexity [24, 25]. The choice between whether to use an end-to-end vs. ML-for-othher-olverr approach has historically been philosophical, with a lack of rigorous comparison. While our work primarily situated within the end-to-end surrogates literature, we also draw inspiration from the ML-for-otherrsolveer approach, notably by incorporating an optimization warm-start component within our framework.
Feasibility enforcement in ol--ooroptimization. While end-to and-to-end ML approaches have been successful in providing fast solutions, these approaches have struggled to satisfy problem constraints. Several
approaches have been proposed to mitigate this, including penalty methods, projection- based methods, and iterative methods. Penalty methods incentivize constraint enforcement by penalizing constraint violations within the MI. loss function [10, 26]; however, these methods do not have feasibility guarantees and can generate highly infeasible solutions in practice [3, 13, 26]. Projection- based methods do provide feasibility guarantees, by projecting the output of the ML model onto the feasible set [11, 12]-in the context of NNs, this can be done via differentiable orthogonal projection methods that admit end-to-end training [11, 20, 27]. In special cases where the projection admits a computationally efficient special structure (e.g.. solvable in closed form), projection methods can work extremely well, providing fast, feasible, high-quality solutions [19, 28, 29]. However, in general cases, solving orthogonal projection problems can be expensive, making training and inference slow, particularly in high-dimensional problems. Iterative methods aim to be more computationally efficient by instead deploying iterative algorithms to feasibility the predictions of an ML algorithm. Notable examples include DC3 [13], which uses variable

---

elimination-based approach to satisfy equality constraints and a gradient-based based iterative approach to satisfy inequality constraints, and ENFORCE [14], which handles equality constraints by unrolling a sequential quadratic programming procedure. Our work presents an iterative method that handles both equality and inequality constraints and, unlike previous iterative methods, provides guarantees on constraint enforcement, which are particularly important in (e.g.) safety-critical settings.
Differentiable optimization in deep learning. Optimization problems can be directly integrated into neural networks as differentiable layers to embed relevant inductive biases or constraint satisfaction directly into the learning process. To enable end-to-end training, gradients must be computed through these optimization layers. Implicit differentiation approaches compute the gradient implicitly using optimality conditions or -oint equations, with approaches presented in the literature for, e.g.. quadratic programs [20, 30], disciplined convex problems [27], cone problems [31, 32], submodular minimization [33], and other problem classes [34]. Unrolled differentiation, by contrast, explicitly unrolls the optimization iterations and leverages automatic differentiation tools to track gradients through each step [35-38]. While implicit differentiation can be more computationally efficient and numerically stable, it relies on obtaining an optimal solution to the optimization problem (which may be unavailable if the optimization did not converge) and is only applicable in situations where the ML-generated inputs show up in the fixed-point equations (which is not true for, e.g., warm start points). For the latter reason, FSNet employs unrolled differentiation (with truncated gradients) to differentiate through the Seasibility-scekingg step during training. For review of methods integrating optimization within neural networks, as well as differentiation techniques, we refer readers to [39, 40].
3 FSNet: Feasibility- Neural Network
We now present FSNet, a novel NN-based framework for providing fast, feasible, high-quality solu- tions to constrained optimization problems. Specifically, we consider the task of solving parametric optimization problems of the form:
min f(Ux) s.t. g(y: x) S 0, h(v;x) = 0,
(1)
SERn
where y E IR=" are decision variables, a E Rd are parameters, s IR x Rd - R is the objective function, and g IR" Rd + R"inng, h: RD x Rd - TR are inequality and equality constraint functions, respectively. For illustrative examples of how I and y are defined in various practical problems, we refer the reader to the tutorial on amortized optimization [2]. Since the minimizer(s)) of problem (1) change as the parameters x vary, this implies an underlying mapping (or mappings) from the parameters optimization problem as a learning problem in which we aim to use an ML model to approximate an underlying mapping. Specifically, let jo IR + R" denote an NN-based function parameterized by 0 E Ree,, with do (x) being its prediction for optimization parameters x. We aim to find 0 by solving:
the minimizer(s). Thus, we can frame the problem of solving the parametric
(1)((()))) s.t. ()) < 0, h(ūo());)) PER"'
min
Vr~D.
(2)
where e follows an unknown distribution D. and we assume access only to a finite dataset sampled from this distribution. Importantly, the goal is to choose # to minimize the expected objective value while ensuring that predictions satisfy all constraints. However, due to sources of error such as optimization error, representation error, and generalization error, NNs alone often fail to produce feasible solutions. To address this, it is necessary to have an additional module that can strictly enforce the feasibility of the predicted minimizer for any x ~ D.
To address this problem, we introduce the FSNet framework, which integrates a fcasibility-secking step into the NN training and inference pipelines. This framework is illustrated in Figure 1, with corresponding pseudocode in Algorithm I. FSNet consists of three main parts: prediction, feasibility seeking, and projection-inspired loss. Specifically, the prediction step entails employing any standard NN yo Rdd + R. to map from input parameters x to a candidate minimizer yo yo()). As this candidate generally violates the constraints, we subsequently execute caasibility-seekinn procedure, which solves an
problem to transform (z) into feasible solution jo (r). We train this prediction + feasibility-seekk pipeline end-to-end to minimize a loss function that penalizes both the original optimization objective f and the distance between the original prediction yo (2) and the post-feasibility outputs ğe (z).
3

---

min f().a)) s.t. u( = 0, h(j..) =
Algorithm 1 FSNet Training Algorithm
1: init. NN weights 6, learning rate
2: repeat
sample x: ~
(ar (ar
(a
4: predict yo (ar) via NN
I
(x)) (x)))
5: compute ğe (x) FS( via (4)
6: update 0 with V. F. (ya(z), (e())) via (6)
until convergence
Prediction
Feasibiity Seeking
Figure 1: A schematic of FSNet, our framework for solving constrained optimization problems.
3.1 Feasibility-seeking step
We now discuss the idea and implementation of the feasibilityssekking step, which serves as the core mechanism in our framework for providing provable feasibility guarantees. Given a potentially infeasible NN prediction yo (z), one natural idea to ensure feasibility is to establish a process that starts at (z) and iteratively moves the point toward the feasible set. As this process progresses, the constraint violations are expected gradually decrease and eventually vanish when feasible point is reached. This idea can be formalized by leveraging yo (x) as a warm start point for the following minimization problem, and letting (1)) be the problem's 's obtained solution:
min (55;)) = [1((s; + 119 (s;;) SE
where d(s; æ)) quantifies the total violation of equality and inequality constraints for given prediction and parameter and g (s; z²) max(g(s; z), 0) is applied element-wiit In practice, the violation- minimization problem (3) can be solved using one of many iterative optimization methods with local and/or global convergence guarantees [41]. For the purposes of our later theoretical analysis, present the vanilla gradient descent method here, which minimizes ((;;)) over iterations l via:
8g yo(r), 8k+1 8k no V (0(();2) 2),
where the iterations start from the prediction yo (z) and move in the direction of steepest descent with properly-chosen step size ne- Let FS TR. Rd + Th be mapping that maps the initial point yo (1)) and parameters x to the final prediction ýo (x:), in particular jo (æ) = FS (ar) == limg- &k- In practice, we terminate our iterative procedure after finite number of iterations until the function value d(ss; x) or its gradient drops below small, pre-specified threshold.
To enable end-to-end training through the NN prediction and Prasibility-scekiine steps, we apply backpropagation through the unrolled iterates of the feasibility-seeking optimization process. (Since the prediction yo (z)) does not appear within the optimality conditions of (3), implicit differentiation is not applicable for computing this gradient.)
Unlike projection-based methods that solve expensive constrained optimization problems, the feasibility-seeking step solves an unconstrained problem, which is typically easier and faster to solve. Unlike DC3 [13], which handles equality and inequality constraints separately, the feasibility-seeking step handles both simultaneously. which facilitates establishing feasibility guarantees (Section 4).
3.2 Loss function
We train the NN end-to-end through the feasibility-seeking procedure by minimizing unsupervised empirical loss function over a dataset (z.(1),...,, ....,,((5)),, defined as follows:
1 8 S
(())
Σ (vo 80 (())
(5)
where (0(x)= = (y) x)) is the output of the seasing step, and we define the function F(yo (x), So (x)) as:
F (vv((z), (x)) = ((o());; + 2 (luo(()) - (1.
(6)

---

with penalty weight > O. The term (z); x) promotes low objective value, while the term Uyo Go (z) (2)1l2 encourages minimal corrections by the feasibilitysseeking step, thereby incentiviz- ing the NN to produce predictions that are closer to feasible. From a geometric perspective, the term [1)) ğo (x) [/ can be interpreted as soft approximation of projection, loosely corresponding to finding a feasible point (a: that remains close to the original prediction ye(z). This term also plays a critical role in the convergence analysis of our framework, which we describe the next section.
Convergence Analysis
In solving problem (1), FSNet is expected to predict solution that minimizes the objective function while satisfying all constraints. Toward showing this, we prove that the feasibility-sceking procedure finds a feasible point (if one exists) an exponentially fast rate (Section 4.1) and that the overall FSNet training procedure converges in the sense of the expected gradient norm vanishing asymptotically (Section 4.2). Finally, characterize condition under which FSNet's prediction (o(x) coincides with an optimal solution of the target problem (Section 4.3). All proofs are given in Appendix D.
4.1 Convergence of fcasibility step
As the feasibility-seekin step essentially solves an unconstrained optimization problem via gradi- ent descent, the convergence analysis of the feasibility seeking step reduces to an analysis of the performance of gradient descent in unconstrained optimization. We state the relevant theorem and associated standard assumption below, and refer the reader to [42, 43] for full proofs.
Assumption 1. For any x. function o(:;z) ; La-smooth, admits minimizer s* E R², and satisfies the PL condition with constant Haz i.e., [VVo(s; 2 2up(o(s:z)
((s*:x)))
Theorem ([42)). Let 7 1 /o/ Lap Under Assumption I, the sequence {8k} generated by the gradient descent (4) with step size n1 = 1/L converges to 8* with rate
||sk s01 8noL3
2 (0((0: x) o(s";z)),
2
) 6(s;; x) - ((5";,)) S (Ø(s0: 2)) - ;()))))).
(8)
that if the target optimization problem (1) admits at least one feasible point for a given æ. i.c. <(s";z)= = 0, then the feasibility-seekinn step drives the NN's prediction yo (z) to feasible point an exponentially fast rate. Indeed, gradient deseent needs most (llog(1/ca) iterations to reduce the violation value Φ(8;;z) below a small threshold cs > 0.
From Theorem 1,
4.2 Convergence of NN training
This section analyzes the convergence of FSNet training using stochastic gradient descent (SGD). Before delving into the analysis, we emphasize an important practical aspect that can impact the convergence of training: the fcasibility-seckin problem (3) is typically solved by unrolling finite number of gradient descent steps in (4) until a small pre-specified convergence tolerance is met. To formalize this, let FS (yo(r); denote the feasibility-seeking mapping obtained by unrolling gradient descent in (4) for K iterations. We then define the resulting approximate solution as o (z) FS (z); Under this finite unrolling scheme, now introduce modified loss function to reflect the approximate nature of the ceaibbllit-seckking step:
S
L(0) = Σ (2)0(x D (z())).
(9)
The NN training is performed using SGD with an unbiased gradient estimator TT(()))
0,+1 0; () with (VVL())) = (L)),),
(10)
where [.] is the conditional expectation given the randomness up step
We now analyze the convergence NN training under this finite unrolling scheme. Establishing SGD convergence for the modified loss L follows standard arguments [44]. However, to understand
5

---

how finite unrolling of the feassbility--seeing step affects optimization of the exact loss L. we must also bound the discrepancy (() V 12- Thus, we need make boundedness and T-smoothness assumptions to control this discrepancy; otherwise, it can be arbitrarily large. Below, we use (w)) = (u)(w) to denote the Jacobian of vector-valued function &b with respect to w.
Assumption (Boundedness). There exist constants By: Brs, Br > such that for all 0, x:
[1) & Bx. 1 (9o(x)))))l: S Brs, (yoo())))) (z)) S Bp.
Assumption 3 (Smoothness). The function F(y, 1) is LF-smooth with respect to D. and the function L(0) is LN Smoohh and bounded below by L". The mappings FS and FS* satisfy for LFS > 0 that
(,))5((o());; JyFS (vo(<);;))) & ( uy (2))
Vo E ERC
Theorem 2. Suppose that E. [ S G for all and that the feasibility step is
unrolled for K steps in every forward pass. Let y be as defined in Theorem I. Under Assumption 1, 2, and 3, after ranning the SGD for T iterations with step size 1j no/ VT for some tju > 0, there is at least one iteration E to..... T } satisfying
(VL(0))) s so (T 12).
E
(11)
E [I 50 (T(111++77)
(12)
This theorem shows that, when the finite unrolling is done in practice, SGD converges in expectation to a stationary point of the NN parameters. Moreover, the finite unrolling introduces a bias order O(7*) to the decay rate of V but the bias decays exponentially in K due y < 1. In other words, for a moderate number of unrolled steps K, the bias is negligible, and the gradient norms under finite and infinite unrolling diminish at virtually the same rate. Consequently, finite unrolling is not only computationally efficient but also sufficient to preserve convergence guarantees.
4.3 Convergence to minimizers
In the previous section, we analyzed parameter-level convergence of the FSNet training process. Here, we turn to output-level convergence: first, we derive conditions under which the network's 's prediction is a stationary point of the function F, and then we identify when that prediction exactly matches the minimizer of the target optimization problem (1).
By the chain rule, the gradient of the loss (5) is the mean of (Jaya(x()))) over all samples. This implies that SGD can stall for two reasons: either the gradient v,F (yo(x()), do (z))) vanishes, or it remains nonzero but lies in the nullspace of Jays (z The former means that the prediction of the NN is a stationary point of F (xx())) go (z())), which is desirable, since only a stationary point has chance to be minimizer of the target optimization problem (1). Therefore, we want to rule out the latter case. One potential approach employing a sufficiently expressive NN so that Jacobian matrices are full-rank and stationary points of F can be reached for all training inputs x. We formalize this observation the following lemma.
()) Do ())
Lemma 1. Suppose SGD converges to 0*, where J(O)) = ((Joye- (x) (Joye- (I())))T)
has full column rank. Then, the prediction of the NN yo(x(0) is the stationary point of F F(yo(z(1),, ), go (x)) with ()()()))
for all i {I....,S}.
This lemma means that after being successfully trained, the NN in the FSNet framework can predict stationary point (z) of F (x)(()) (x)). If F is convex, then the stationary point is global minimizer of F. However, if F is nonconvex, the stationary point could be a minimizer, maximizer, or saddle point. In such case, additional assumptions are necessary for further analysis. Thus, we now make a (fairly strong) assumption that the NN's prediction is a global minimizer of F. However, a minimizer of the target problem (1), because F differs
a minimizer of F may not necessarily from the original objective function f in including a penalty term. Inspired by the penalty method in constrained optimization [41], we can increase the penalty weight p such that the difference (z) ÿe (x) 1|2 vanishes, which drives the minimizers of F and problem (1) to coincide.

---

Theorem 3. Given specific parameters suppose that the NN predicts yr which is global minimizer of Fly, y: Pr) f(:: + 11y 011 with ğ FF((;; æ); that the mapping FS is continuous; and that O <. Pr < p1+1, VT and pr - xc. Then, every limit point (y',"") of the sequence {(Vr: ûr) admits y" = 5",,, and O' is global minimizer of the optimization problem (1).
We acknowledge that the assumption in Theorem 3 is rather stringent, but it nonetheless provides some useful insights for practical implementation. In particular, the penalty term & (o) - (o(xr) in (6) needs to be increasing yield high-quality solutions. From our experiments, choosing a sufficiently large value of p is sufficient, and the range of effective is wide, making easier to select appropriate value.
find that
5 Practical Efficiency Improvements
While the above FSNet framework offers theoretical guarantees, we also provide practical strategies to improve training efficiency and stability. We discuss the use of truncated gradient tracking to reduce computational overhead, and enhancement to the loss function to improve training stability,
Truncated unrolled differentiation. Inspired by [37], we implement truncated unrolled differentia- tion to improve the computational efficiency of the FSNet framework. Specifically, when computing gradient updates, we backpropagate through only the first K" out of the total K" iterations of the Gaasibility-seeking procedure, rather than all K iterations. In other words, only the computation of the first K" iterations is included in the computational graph for backpropagation, while the remaining iterations act as pass-through layers without gradient tracking. This approach significantly reduces computational graph size, memory, and the overhead of the backward pass [37]. We empirically observe negligible loss in solution quality even for modest K", and in Appendix D.2, we prove that, under local strong convexity of é, the Jacobian error and the truncation-induced bias the SGD updates decay exponentially with K'.
Stabilizing early-stage training. With randomly initialized parameters, the NN may produce large constraint violations early in training, forcing the feasibility-seekin step to use many iterations and slowing down overall training. To mitigate this, we introduce an additional penalty for constraint violation when it exceeds threshold:
(1)6 { f(j(()):)) + (11 (0(x) + (s ((o((): (z) de (z)
(x)) 2 Q, otherwise,
(13)
where the threshold Q can be big, c.g., Q = 10 When constraint violations are too large (ø(y: x) > Q). the extra term adds an additional penalty to the loss that pushes the NN to make more feasible predictions. Otherwise, the extra term is removed, and the loss reverts exactly to the original form.
Experiments
Problem Classes. We evaluate our method on three families of synthetic problems: smooth convex, smooth nonconvex, and nonsmooth nonconvex. The smooth convex problems include quadratic anttraticallyyccnsttaeeed programs (QPs), quadratically constraiied quadratic programs (QCQPs), and second-order cone programs (SOCPs). Smooth nonconvex problems are constructed by adding elementwise sine/cosine terms into the objective and constraint functions of the smooth convex problems, while the non- smooth nonconvex problems further add an 4g-nrrm regularization term to the objective function (Appendix A). Each problem has 100 decision variables subject to 50 equality and 50 incquality constraints. We generate dataset of 10000 samples for each problem following the procedure in [13, 45], and split it into 7000 training, 1000 validation, and 2000 test samples for all NN-based methods. For the smooth convex settings, we additionally consider larger-scale problems with 500 decision variables, 200 equality constraints, and 200 inequality constraints. We use 17000 training, 1000 validation, and 2000 test samples for this case. Finally, we consider the problem of AC optimal power flow (ACOPF), a nonconvex, NP-hard problem in electric power systems that aims to schedule power generation at least cost while satisfying grid and device constraints; for this problem, use the formulation and dataset from [46].
Baselines. We compare our FSNet against:
7

---

Optimality Gap (%)
Equality Violation
Inequality Violation
1.25
MSNet
Add. Pen.
0.32
0.24
0.75
0.24
Value
0.50
0.16
0.16
0.25
0.08
0.08
4
-0.25
0.00
0.00
QP
QCQP
SOCP
QP
QCQP
SOCP
QP
QCQP
SOCP
Figure 2: Test results 2000 instances of convex problems with 100 decision variables. FSNet consistently obtains near-zero constraint violations and small optimality gaps across the problem classes, whereas the baseline methods incur significant constraint violations.
Solver: A traditional solver. We use OSQP [47] and SCS [48] for convex problems, and use IPOPT [49] for nonconvex problems.
Reduced Solver: For a fairer runtime comparison, we reduce the solvers' tolerances to achieve level of constraint satisfaction comparable to FSNet.
Penalty: A popular approach for addressing constraints in training NNs that adds a fixed-weight penalty for constraint violations within the loss function.
Adaptive Penalty: An improved version of the fixed penalty method that adjusts the penalty weight based on current violations, as [10].
DC3: The Deep Constraint Completion and Correction framework from [13].
We also implement the Projection method in the convex QP setting, which orthogonally projects the NN's prediction onto the feasible set using the qpth solver [20]. While we attempted to implement Projection for other problem classes via evxpylayers [27], this introduced significant computational overhead, making training prohibitively slow. All network, training, and solver-related hyperparame- ters are in Appendix C. We also provide experiments analyzing FSNet's performance with varying numbers of tracked iterations truncated unrolled differentiation (Section 5), different values of the penalty weight p. and additional results for DC3 using loss function (5) in Appendix B.
Evaluation metrics. We evaluate the methods using the following metrics: constraint violations, optimality gap, and computational time. We compute |[ (1) for equality violations and I|g (ğo (z);; |||| for inequality violations. Since some trusted solvers can return the global minimum for convex problems, we take their solution /salver to define the relative optimality gap by:
optimality gap f(i f (Usolver: x))
If (Jsolver; x)]
For computational time, we compare two metrics: Batch and Sequential computational time. For Batch computational time, all problem instances were solved in setting of maximal parallelization within a particular budget compute hardware. For Sequential computational time, all instances were solved one-at-a-time in sequence. Traditional solvers were run on Intel(R) Xeon(R) CPU R5-2670 v2 and all NN methods were run on NVIDIA Quadro RTX 8000 GPU. All metries are computed over 3 runs. Results these metrics are summarized below. with full results in Appendix B.
Feasibility-seeking method. We employ L-BFGS with backtracking line search for feasibility seeking, as it nicely balances convergence rate and scalability to nigh-dimensional problems [41].
6.1 Smooth convex problems
For the convex settings with 100 decision variables, Figure 2 shows that FSNet outperforms the other baselines by obtaining near-zero constraint violations, whereas other methods incur significant constraint violations. For DC3, equality constraints are handled well, but inequality constraints are not satisfied across cases; moreover, we observe that the performance DC3 on the inequality constraints is sensitive to the step size of correction steps. In contrast, our feasibility -asiiblltt-seekking step based on L--BGS performs robustly across problem classes without needing to change the hyperparameters.
8

---

Parallel Time
Sequential Time
Parallel Time
Sequential Time
10
PSSct Sasteer
Time (seconds)
102
10
10
7
QP
OCQP
SOCP
QP
QCQP
SOCP
NC OP
NC OCOP NC SOCP NC OP NC QCOP NC SOCP
Figure 3: Computational time on 2000 instances of convex and nonconvex (NC) problems with 100 decision variables.
Equality Vio.
Inequality Vio. Mean (Max)
Optimality Gap (%)
Runtime Batch (sequential)
Method
Mean (Max)
Mean
Min (Max)
Convex QP:
500, Reg
200, Blacy
200
1.9--9429--10
L.e-6=9.88-8
112.962:40.375
Solver
(1.20-6-988-) 9.8e-)
11.3-4=1133--)
11394.024:194..1
FSNet
1.0c44 2.566
4.5e-6=1.5e-7
0.066:27664
0.041 4-66e-4
0.599-41.8e.
(4.4e 3-8.2-4)
(4.2e 14.2e-4-9.06-9)
(0.119444963)
(163.558=11027)
Convex QCQP: n
500, Reg
200, Pince
200
5.0e-2=9..64
5508.419::-44220
Solver
(3.10-4=7.6-5)
(2.10-1=2.3e:3)
(33362.147 112211177:11099995
4-744 (377-4-1166))
9.9e-7-455-8
0.087++2.5..
30.291
NS
1.143
(1.0e-5=496-7)
(0.328:10003)
(212.817=1.255)
Convex SOCP: = 500, Mlg
200, Ripey = 200
5.8e- 26111
2.2e- 10::111-11
523.045:6:284
Solver
(2:1e-1243)))13))
(8.1e-8:27e-8)
1300044.13
FSNet
4-744-14 (370-422.1e6))
0.0 (0.0200)
0.730=0027
0.469:20.246 (1.12420048)
14.313-20.073 1.9)))
Table
Results of FSNet on 2000 instances of smooth convex problems with 500 decision variables.
FSNet also obtains small optimality gaps, all less than 0.2% on average. The baseline methods also exhibit small optimality gaps, but their solutions violate the constraints significantly, which means these gaps are not meaningful. Specifically, we see that when constraint violations exist, NNs can easily find an infeasible point to achieve low objective values and optimality gaps- e.g., Penalty, Adaptive Penalty, and even DC3 attain negative optimality gaps for some instances (Figure 2).
Overall, FSNet achieves order-of-magnituuee speedups over traditional solvers-e.g, 30-1000x 30-00000000000000000000 faster in batch solving (Figure 3). For QP problems. FSNet outperforms OSQP when using batch solving but is slower under sequential solving; this difference arises because OSQP is specifically designed to exploit the problem structure of QPs, whereas FSNet does not yet incorporate such problem-spp -sseccfic optimizations. Despite constraint satisfaction, the projection method runs about 3.6x slower than the traditional solver, and about 113x slower than FSNeet in the batch setting (Figure 3). In contrast, for QCQP and SOCP problems, FSNet consistently outperforms traditional solvers in both batch and sequential settings. Due to the incorporation of the Teasibility-seeking step, FSNet has slower runtime compared to other NN-based methods; however, this extra overhead is necessary tradeoff to ensure the feasibility of the prediction. Conversely, faster methods fail to ensure feasible solutions, rendering their speed meaningless in applications where constraint satisfaction is non-negotiable.
For larger-scale convex problems (with 500 variables, 200 equality constraints, and 200 inequality constraints), the results are presented in Table 1. In the batch setting, FSNet achieves speedups of 188x (QP) and 147x (QCQP), which are notably higher than the speedups obtained in the smaller problems with 100 decision variables (namely, 31 x and 104x, respectively). For SOCP, FSNet achieves a smaller speedup of 36x compared to 1 1125x x in the 100-variable case. In the sequential setting, FSNet achieved speedups of 8x (QP),143x x (QCQP), and 2754x (SOCP), compared to 0.8x, 6x, and 18x in the corresponding 100-variable cases. In general, FSNet becomes increasingly advantageous in larger-scale settings. This can be explained by the fact that the feasibility-seekin step, which is an unconstrained feasibility problem and the dominant component of FSNet, inherently more scalable than solving the original constrained optimization problems.

---

Nonvex QP
Nanconvex QCQP
Nonconvex SOCP
0.6
0.6
Solver FSNet
Probability
0.4
0.4
0.2
0.2
-19.5
0.0_20
0.00
Objective Value
Objective Value
Objective Value
Figure 4: Distributions of objective values for solvers and FSNet in nonconvex problems. FSNet obtains sharper and lower-centered objective distributions for Nonconvex QCQP and Nonconvex SOCP, while matching the solver's optimality in Nonconvex QP with nearly identical distributions.
6.2 Smooth nonconvex problems
On smooth nonconvex problems, FSNet consistently achieves near-zero constraint violations across nonconvex problems and achieves 85-3000x batch speedups and 2.7-1 177117x sequential speedups over traditional solvers (Figure 3). While the computational time of FSNet remains relatively stable across problem types, nonconvex solvers typically require more time than their convex counterparts, resulting in even greater speedup gains for FSNet in nonconvex settings. As in the convex cases, the Penalty and Adaptive Penalty show poor performance on these problems with respect constraint satisfaction. Results for DC3 are not shown, as this method sometimes diverged in the presence of nonconvex constraints due to the divergence of Newton's 's method in the completion procedure.
Notably, FSNet actually outperforms the traditional solvers in terms of optimality for nonconvex problems. Specifically, for the Nonconvex QCQP and Nonconvex SOCP problem variants, the optimality gaps are negative (Table B.2), indicating that the solutions predicted by FSNet yield lower objective values than those obtained by the solvers. The difference is clearly illustrated in Figure 4, where the distribution of FSNet objective values is sharper than that of the solver in Nonconvex QCQP, and FSNet's distribution for Nonconvex SOCP is tightly centered at lower objective values. For the Nonconvex QP, the two distributions are almost perfectly aligned, suggesting that both methods achieve similar solution quality.
6.3 Nonsmooth nonconvex problems and AC optimal power flow
Results on nonsmooth nonconvex problems and ACOPF problems are in Appendix B. The findings are generally consistent with those for smooth nonconvex problems, In particular, FSNet achieves near-zero constraint violations across these problems. For nonsmooth nonconvex problems, FSNet achieves 152-3316x speedups in the batch setting and 7-80x speedups in the sequential setting compared the traditional solver. We similarly observe that FSNet can find better local solutions with lower objective values than the traditional solver in the nonsmooth nonconvex setting. For the ACOPF problem, FSNet is faster than IPOPT, achieving 3x speedups on the IEEE 30-bus system and 11 x on the IEEE 1 118-bus system in the sequential computation, with a less than 1 1% optimality gap.
7 Conclusion
We proposed a novel framework, FSNet, that integrates a cessibility-seekinn step into NN training to solve general constrained optimization problems, with theoretical guarantees on feasibility and convergence. The feasibility-seeking step solves an unconstrained optimization over a function of constraint violations, initialized at the NN's prediction, to produce a feasible solution. Our experi- ments on smooth/nonsmooth and convex/nonconvex problems demonstrate that FSNet consistently outperforms popular bascline methods in achieving negligible constraint violations. Regarding optimality, FSNet matches the performance of traditional solvers on convex problems and often finds better local solutions in nonconvex cases, all while being significantly faster.
We observe through the experiments that FSNet can perform well for problems that violate the assumptions used in the convergence analysis (e.g., L-smoothness, PL condition). Thus, extending the convergence analysis Section 4 to cover such cases a potential direction for future work. While FSNet shows strong performance across diverse problem classes, scalability to extremely large- scale settings also remains an open direction. Another potential future direction is further scaling up the performance of the feasibility-seeking step using more efficient optimization algorithms.
10

---

Acknowledgments
This work was supported by the U.S. National Science Foundation under award #2325956.
References
[1] Pascal Van Hentenryck. Optimization learning. arXiv preprint wwxxx::501.004444 2025.
[2] Brandon Amos et al. Tutorial on amortized optimization. Foundations and Trends in Machine Learning. 16(5):592 732, 2023.
[3] Seonho Park, Wenbo Chen, Terrence WK Mak, and Pascal Van Hentenryck. Compact optimization learning for ac optimal power flow. IEEE Transactions on Power Systems, 39(2)4350-4339 2023.
[4] Hooman Khaloie, Mihaly Dolanyi, Jean-Francois Toubeau, and François Vallée, Review of machine learning techniques for optimal power flow. Applied Energy, 388:125637, 2025.
[5] Ján Drgoña, Aaron Tuor, Vikas Chandan, and Draguna Vrabie. constis-onnttreiee deep learning of multi-zone building thermal dynamics. Energy and Buildings, 243:1 10992, 2021.
[6] Xiaoyan Hu, Christos Masouros, and Kai-Kit Wong. Reconfigurable intelligent surface aided mobile edge computing: From optiiiztionnbasee based to location-only learning- based solutions. IEEE Transactions on Communications, 69(6) 2021.
[7] Yuwei Wu, Xiatao Sun, Igor Spasojevic, and Vijay Kumar. Deep learning for optimization of trajectories for quadrotors. IEEE Robotics and Automation Letters, 9(3):2479 2486, 2024.
[8] Abhishek Cauligi, Preston Culbertson, Bartolomco Stellato, Dimitris Bertsimas, Mac Schwager, and Marco Pavone. Learning mixed-integer convex optimization strategies for robot planning and control. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 1698-170S IEEE, 2020.
[9] Vahid Eghbal Akhlaghi, Reza Zandehshahvar, and Pascal Van Hentenryck. PROPEL: Supervised and reinforcement learning for large-scale supply chain planning. arXiv preprint arXiv2250.007883 2025.
[10] Ferdinando Fioretto, Pascal Van Hentenryck, Terrence WK Mak, Cuong Tran, Federico Baldo, and Michele Lombardi. Lagrangian duality for constrained deep learning. In Machine learning and knowledge discovery in databases. applied data science and demo track: European conference, ECML pKDD 2020, Ghent, Belgium, September 14-18, 2020, proceedings, part v. pages 118-135. Springer, 2021.
[11] Youngjae Min, Anoopkumar Sonar, and Navid Azizan. cand-constrained neural networks with universal approximation guarantees. arXiv preprint arXiv:2410. 10807, 2024.
[12] Dou Huang, Haoran Zhang, Xuan Song, and Ryosuke Shibasaki. Differentiable projection for constrained deep learning. arXiv preprint arXiv:2111.10911 2021.
[13] Priya L. Donti, David Rolnick, and J Zico Kolter. DC3: A learning method for optimization with hard constraints. In International Conference on Learning Representations, 2021.
[14] Giacomo Lastrucci and Artur M Schweidmmann. ENFORCE: Exact nonlinear constrained learning with adaptive-depth neural projection. arXiv preprint 250.:25502 2025.
[15] Yoshua Bengio. Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: methodological tour d' d'horizon. European Journal of Operational Research, 2020.
[16] Fouad Hasan, Amin Kargarian, and Ali Mohammadi. A survey applications of machine learning for optimal power flow. In 2020 IEEE Texas Power and Energy Conference (TPEC), pages 1-6. JEEE, 2020.
[17] Wenbo Chen, Sconho Park, Mathieu Tanneau, and Pascal Van Hentenryck. Learning optimization proxies large-scale 2022.
economic dispatch. Electric Power Systems Research, 213:100h 108566.
[18] Ahmed S Zamzam and Kyri Baker. Learning optimal solutions for extremely fast ac optimal power flow. In 2020 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmarGGriCoomm), pages 1-6. IEEE, 2020.
[19] Mathieu Tanneau and Pascal Van Hentenryck. Dual Lagrangian Learning for Conic Optimization. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
11

---

[20] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization a layer in neural networks. In International conference muchine learning. pages 136-145. PMLR, 2017.
[21] Florian Thams, Andreas Venzke, Robert Eriksson, and Spyros Chatzivasilciadis. Efficient database generation for data-driven security assessment of power systems. IEEE Transactions on Power Systems, 35(1))30-41, 2019.
[22] Hong Ye Tan, Subhadip Mukherjee, Junqi Tang, and Carola- Bibbaan Schönlieb. Data-driven mirtor descent with input-convex neural networks. SIAM Journal on Mathematics of Data Science, 5(2):558-587, 2023.
[23] Rajiv Sambharya, Georgina Hall, Brandon Amos, and Bartolomeo Stellato. Learning to warm-start lixed-point optimization algorithms. Journal of Machine Learning Research, 25(166): 46, 2024.
[24] Dimitris Bertsimas and Georgios Margaritis. Global optimization: a machine learning approach. Journal of Global Optimization, 01(1):1-77, 2025.
[25] Sidhant Misra, Line Roald, and Yeesian Ng. Leaming for constrained optimization: Identifying optimal active constraint sets. INFORMS Journal on Computing. 34(1)=463-480 2022.
[26] Sconho Park and Pascal Van Hentenryck. Self-supervised primal-dual learning for constrained optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 4052-4060, 2023.
[27] A. Agrawal, B. Amos, Barratt, Boyd. Diamond.
Kolter. Differentiable convex optimization layers. In Advances in Neural Information Processing Systems, 2019.
[28] A new computationally simple approach for implementing neural networks with output hard constraints. In Doklady Mathematics, volume 108, pages S233-S244 Springer, 2023.
[29] Jesus Tordesillas, Jonathan P How, and Marco Hutter. Rayen: Imposition of hard convex constraints on neural networks. arXiv preprint arXiv:2307.08.. 2023.
[30] Connor W Magoon, Fengyu Yang, Noam Aigerman, and Shahar Z Kovalsky. Differentiation through black-box quadratic programming solvers. arXiv preprint urXiv:2410.06324 2024.
[31] Priya Donti, Metrose Roderick, Mahyar Fazlyab, and Zico Kolter. Enforcing robust control guarantees within neural network policies. In International Conference on Learning Representations, 2021.
[32] Akshay Agrawal, Shane Barratt, Stephen Boyd, Enzo Busseti, and Walaa Moursi. Differentiating through a cone program. Journal of Applied and Numerical Optimization, 1(2):10- 1107-115,77 2019.
[33] Josip Djolonga and Andreas Krause. Differentiable learning of submodular models. Advances Neural Information Processing Systems, 30, 2017.
[34] Stephen Gould, Richard Hartley, and Dylan Campbell. Deep declarative networks. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(8):3988-4004 2021.
[35] Vishal Monga, Yuelong Li. and Yonina C Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEFE Signal Processing Magazine, 38(2):18-44, 2021.
[36] James Kotary, My H Dinh, and Ferdinando Fioretto. Backpropagation of unrolled solvers with folded optimization. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 23, 2023.
[37] Amirreza Shaban. Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back- pack-propagation for bilevel optimization. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1723-1732. PMLR, 2019.
[38] Damien Scieur, Gauthier Gidel, Quentin Bertrand, and Fabian Pedregosa. The curse of unrolling: Rate of differentiating through optimization. Advances in Neural Information Processing Systems, 35:17133- 17145, 2022.
[39] Jayanta Mandi, James Kotary, Senne Berden, Maxime Mulamba, Victor Bucarey, Tias Guns, and Ferdi- nando Fioretto. Decision-focused learning: Foundations, state of the art, benchmark and future opportuni- ties. Journal of Artificial Intelligence Research, 10:1623-1701, 2024.
[40] Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig. Stephan Hoyer, Felipe Llinares- Lineres-López. Fabian Pedregosa, and Jean- Philippe Vert. Efficient and modular implicit differentiation. Advances in neural information processing systems, 35:5230-5244 2022.
12

---

[41] Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer. 1999.
[42] Boris Polyak. Gradient methods for the minimisation of functionals. USSR Computational Mathematics and Mathematical Physics, 3(4):864-878, 1963.
[43] Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradient methods under the polyak-tojasiewiez condition. In Joint European conference machine learning and knowledge discovery in databases, pages 795-811. Springer, 2016.
[44] Shai Shalev-Shwartz and Shai Ben-David. Understonding machine learning: From theory algorithms. Cambridge University Press, 2014.
[45] Enming Liang, Minghua Chen, and Steven H Low. Homeomorphie projection to ensure neural-network solution feasibility for constrained optimization. Journal Machine Learning Research, 25(329):1-55, 2024.
[46] Michael Klamkin, Mathieu Tanneau, and Pascal Van Hentenryck. PGLeam-an open-source learning toolkit for optimal power flow. arXiv preprint 2222: 2025.
[47] Bartolomeo Stellato, Goran Banjac, Paul Goulart, Alherto Bemporad, and Stephen Boyd. Osqp: An operator splitting solver for quadratic programs. Mathematical Progranming Computation, 12(4):637-672, 2020.
[48] Brendan O'Donoguue Operator splitting for homogeneous embedding of linear complementarity problem. SIAM Journal on Optimization, 31:1999-2023, August 2021.
[49] Andreas Wächter Lorenz T Biegler. the implementation an interior-point filter line-search algorithm for large-scale nonlinear programming. Mathematical programming, 106:25-57, 2006.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state the paper's focus on the design of new framework for solving constrained optimization using neural networks. The contributions are accurately reflected, and the key assumptions for the theoretical guarantees are explicitly stated in the contribution summary.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the strength of assumptions in Section 4, and the limitations of the proposed method in Section 7.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide all assumptions and proofs for our theoretical results in Section 4 and Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We report all experiment settings in Section 6 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the anonymized code and data with instructions to reproduce experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
Pobllic/guides/CddSttttii www.ammmissoooocccyyy..... for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //niss. cccccc//p/ www.ccpppcc//eeeecc.eeeeeeeeetle wtttiii/guie////eee Sotps//oooSeueiiioioonnooo ssioliolici for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We provide all hyperparameters of methods in Section 6 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We experiments with three independent runs and report the mean and standard deviation of evaluation metries (Section 6).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information on the CPU and GPU we used for our experiments in Section 6.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ httppp:////uiu../.eeee hptpblic/tthissuiieeeee...
Answer: [Yes]
Justification:
Our research adheres every aspect of the NeurIPS Code of Bthics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work focuses methodological advances for constrained optimization and does not involve direct deployment or societal-facing applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This study does not involve data or models that pose a risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite and credit all used assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcee thcode. cowccoo//aooo/o datasatssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: We provide anonymized code and (synthetic) data for the paper with instruc- tions for running the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not conduct crowdsourcing experiments nor research with human
subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
This study does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
This paper does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: http://pppp..cccoaaeearrrs////// wct//ooneeeencce//25//////// for what should or should not be described.
20

---

A Problem formulations
A.1 Convex problems
The following formulations share: E R", Q E S++P L,
QP:
min LSUSU
2 Qy v"y
s.t. Ay = x, Gy Sh,
where G Bias X E Pince
1 T Qy + +pu y
QCQP:
min LSUSU
2"
Ay E y H.y + 9. v h4,
where H, E 1 R², h E R for i
1, niney-
1 T Qy + vTy 2リ
SOCP:
min LSVSU
s.t. Ay =
||Gay + hill2 S of y + di,
where G; mx h4 RT o E R E R for 1, ..... Tineg.
A.2 Nonconvex problems
We modify the convex problems by adding element wiis sine and cosine functions.
Nonconvex QP:
min I.SSU
2 Qy + pp -Tppsins)) sin(y)
s.t. Ay z, Gsin(y) S h cos(x),
Nonconvex QCQP:
min LSWSU
2" Qu +p' sin(y)
s.t. Ay = H, y H + gh cos(y) s shi
Nonconvex SOCP:
min S.SUUU
" Qy + p² sin(y)
Ay T. [[s(())))) hall2 S cJ y + d.
A.3 Nonsmooth nonconvex problems
Compared to the (smooth) convex problems, we further add an &n-norm regularization term to the objective function.
Nonsmooth nonconvex QP:
min LS 2 Qy + p sin(y) + Al|v|la
Qy+
Ay x, Gsin(y) < cos(x))
Nonsmooth nonconvex QCOP:
min LS
2" Qu + p sin(y) + A|lv|la
s.t. Ay z, y H.U 9. cos(y) I has
Nonsmooth nonconvex SOCP:
min L LS
2 Qy + p sy) Ally|l2
s.t. Ay = x, ||G cos(y) hull2 S of y + d.
A.4 AC optimal power flow
For the AC optimal power flow problem, the formulation from [46], reproduced below:
21

---

ΣΣS opp
min of
iEN JEG,
s.t.
P Pi
d
Σ
Pe +
Σ Pe.
ViEW
CE
CEE,
C. 41 + D,0
Σ de
+
Σ de
VieW
NEE;
EEE!
Pe ge of + Je viúy cos(O; 0z) + be u.u sin - -0,)
2
Ve (i.j) 8
E D ' - b v,, cos(O; 02) + ge Uuu sin(0: - 0,)
Ve (1.j) E
Pc de oj
tt 2 + De Ui cos(O, - 02) - 0e Sewvysink
03)
Ve (i,j) 83
d' 00 o uu
u1 cos(6: 8,) o. w,oy sin(0; - 0,)
AA (i.5) 33
3324
3
(v))² (gf)² 5
333k
(0))² + (1.)² <
AQ. 0, 0f S A0.
aA
=
(6.))
=
p S pf S
Vi
of S o. क
vieg
2 $ U; S 7
Vi N
Se $ P/ < Se
S, < S.
33
3. <P. 33.
3
S. S d- <
3
where N, E, and denote the sets of buses, branches, and generators, respectively; ps and gs represent the active and reactive generation powers; pf and o (p" and g²) denote the active and reactive power flows in the forward (reverse) direction; and and denote the voltage magnitude and angle. The network parameters g* g ""g , gg and btt bft bt 5 are obtained from the admittance matrix of the system. In addition. S denotes the apparent power flow limit, while A and Ae specify the lower and upper bounds on the voltage angle difference. The parameters p and 7r (g and 7F) indicate the lower and upper limits of active (reactive) generation, and y and v define the permissible range of voltage magnitudes.
22

---

B More experiment results
B.1 Smooth convex problems
Table B.1 presents detailed results for smooth convex problems. Figure B.1 plots FSNet's runtime on 2000 test instances function of batch size. Thanks to GPU parallelism, the runtime decreases monotonically as batch size increases, so it is recommended to use the largest batch size possible to maximize speedup. Figure B.2 reports CPU utilization for the parallel solver runs. Across all problems, CPU usage remains near 100% throughout execution, showing that we fully exploit available CPU resources. This indicates that our runtime comparisons between NN- NN-based methods and the solver are fair.
Equality Vio.
Inequality Vio. Mean (Max)
Optimality Gap (%)
Runtime (s) Butch (Sequential)
Method
Mean (Max)
Mean
Min (Max)
Convex QP: v. 100,
100, Bineg
100
1777-16
2.4e-14 14122-444--6
3.8022:50:445
Solver
(1.9--13435--1
(5.4e-1424.100) 19-116))11
(140.328:24.585)
1.2c 6-774488
3,777=00.00
Reduced Soliver
1.
(8.5e-541 0e-5)
(7.le-5:100e) 0e-1)
(5.3e-3430--))
(127.105::5:37)
Projection (ugtt)
9.75-14433.--16
1.2e-144:4.e-16
1e2
13.71
(144-134444-115)
(4.1e-1342..e-15))
(0.02253363)
(225 (35.0661433385)
1.36-411..ee
1.60--:1444e-7
5.70-31197e44
0.121
FSNet
0.015
(7.70-44:199 4)
(7.20-3:4:32e--)
10.038:4:7.7e
(180.389
0.219:15.8::3
0.023:13.36-3
-0,074:20.013
0,043:229e-3
Ponalty
-0.02910013
(5.4e-5-12..ee-5)
(0.227:+0.039)
10.028:19.84)
(1.594:0.051)
0.221=77.e-3
0.022:
0.78240004
0.042-1176-3
Adaptive Penalty
0,974100009
(0.371::2:9e-)
(0.211=00011))
(1.213400004)
(1.549=0011))
4.9.-13447..6-6
0.133-20.011
0.087.20019
DC3
(5.8e-13266.8-11
(0.259=0.002))
0003550016
(0.029+am21)
(53. (511558))))88)
Convex QCQP:
100, flg
Rineq
50
8.4e-741968
1.5e:3420065
46.619=20557
Solver
(2.4e-5+536.6)
(7.7e:34 0.00)
(1118.149433555)
FSNet
(1.50-3:55.7o44)
1.2e:44:..2o55
(6.4o-5:1244-5)
1.0e:6::226-7
0,035-12--
1.8
(0.359:200.699)
0,448:20116
Penalty
0.191=:596-1
0.031=001
0.038
0.237-=0.022
0,048=100-3
(0.355:10014)
(0.02)=00.019)
(0.397:10))
(1.707:268:-3)
0.179:17.6:1-3
0.028:12.1e3
-0.119:10.067
0.047:22.8-3
Adaptive Peaalty
0,274
(0.333:=7.6e-3)
(0.497=
(0.797:t0.079)
(1.703++032)
0.120:10.032
0.009
0.163
DC3
(0.6002=0.072)
-0.018+0087
(0.259++00099)
(79.234=+6847)
Convex SOCP: 100, Dag
50, Mineq
50
3.1e-144-4.20-1
1.3e-11
332.109--21186
Solver
(2.2e 13::0000
(1.0e 8::8.
(3702.308+:9.50
Reduced Soliver
3,9e-74-13668
1.8e44 4-4.1e66
1.8e-52211e6
-1.20-3444.5e-
330.1434:1009
(4.8 66:33--7)
(2.0e 3:b.77)))
(1.4e-3+1111e4)
(1895.989:14.50
FSNet
1.3e-44-12ed
1.7e-8+-136-8
0.159
0.046-43..6-3
8.29
(5.5e 44999-5)
(4.9e
(1.39240376)
(30.399949.335)
Penalty
0.098:13:26-1
0.0152332663
0.024:20.018
-0.235=0.100
0.0492144-3
(0.139:23.6e:3)
(0.309-10.053)
(0.338:10.108)
(1.91920223)
Adaptive Penalty
(0.136:17.2e-3)
0.064222.663
(0,305:20,147)
0.01
0.25620.126
1.1.220.14
(1.891+22:299)
4.047
7.8e-14:-63e--1 (1.16-13:19..--11
3.00 (0.325):0.049)
9.313
0.114:55.43 (78.831:5254)
DC3
0.05320.011
30.00000000)
Table B.I: Test results of on 2000 instances of smooth convex problems across 3 random seeds.
23

---

Problems
Nonconvex Problems
Time (seconds)
OCOP
NC QCQP
S0CP
NC SOCP
10
10
100
10
10-1
128
256
512
1024
2000
128
256
512
1024
2000
Batch Size
Batch Size
Figure B.1: Computational times FSNet on 2000 instances of smooth convex and nonconvex (NC) problems with varying batch sizes.
Convex Problems
Nonconvex Problems
125
Convex OP
Convex SOCP
125
NC
NC SOCP
QCQP
QCOP
CPU CPU Usage (56)
75
75
50
50
25
0
50
100
150
200
0
200
400
600
Time Step
Time Step
Figure B.2: CPU utilization over time for parallel solver runs on smooth convex and nonconvex problems.
B.2 Smooth nonconvex problems
Figure B.3 and Table B.2 present the detailed results for nonconvex problems. Figure B. and B.2 also show the computational time of FSNet with different batch sizes and CPU utilization on nonconvex cases, respectively. The patterns are the same as those observed in the smooth convex problems.
Optimality Gap (%)
Equality Violation
Inequality Violation
104
FSNet
Ado. Pen.
0.45
10a
0.6
10
0.30
Value
0.4
0.15
-10
0.2
-102
-10
0.00
0.0
QP
QCQP
SOCP
QP
QCQP
SOCP
QP
QCQP
SOCP
Figure B.3: Test results 2000 instances of smooth nonconvex problems. FSNet consistently obtains near-zero constraint violations and small optimality gaps across the problem classes, whereas the baseline methods incur significant constraint violations.
24

---

Equality Vio. Mean (Max)
Inequality Vio. (Max)
Optimality Gap (%)
Runtime (s)
Method
(Max)
(sequential)
Nonconvex QP: n 100, Teg
100, Rines
100
L.5e-1421.ee166
188-91175--11
0.662
Solver
(8.3--14=3.1e-15)
8.0--10
(470.846:+03991)
FSNet
(1.40-342e4)
(4.40-521
7.2e: 7-8870
0.14620.012
0..85200500)
0.34
(171.226:11667))
0.146=0.088
Nonconvex QCQP: n 100, Blg
50, Pineg
1.86-14=1.66-16
8.9e-9#2.1e:10
216.772++2.680
Solver
1495-14433515)
(5.46-81166-9))
(11169.226::00222)
FSNet
9.76-5230-6
2.9e-61224e-7
-0668
49.436
0.228
(6.50-6129--))
(1.oo-4114e3)
(8 0.942)
1305:239125.722)
Nonconvex SOCP:
100, nog 50, Pameg
14
9:
1216.105
Solver
1330-132336-19
(7.6--177)--0)
(33765.58)
5.8o-513.5e-5 (e.
5.20-711.5e- (2.5e-4:11.6))
9.366.9900 (67.138
0,406=0.055
FSNet
-2.3c3410.63
(288.161:5) 161:161111999 161:199.998
Table B.2: Test results of FSNet on 2000 instances of smooth nonconvex problems.
B.3
Nonsmooth nonconvex problems
Table B.3 shows the detailed results of nonsmooth nonconvex problems (see Appendix A for formu- lations). As shown in this table, FSNet achieves near-zero constraint violations across nonsmooth nonconvex problems and achieves I 552-3116x speedups in the batch setting and 7-80x speedups in the sequential setting compared to the traditional solver, which is also visualized in Figure B.5.
Notably, FSNet can attain lower-cost local solutions than the solver, as evidenced by the negative optimality gaps in Table B.3 and by the distributions of objective values in Figure B.4.
Equality Vio.
Inequality Vio.
Optimality Gap (%)
Runtime (s)
Method
Mean (Max)
Mean (Max)
Mean
Min (Max)
Batch (sequential)
Nousmooth Nonconvex QP:
100, Beg 100, Ripeg 100
5.4g-14=
5.3e-10-:5.6e122
19.161+00.66
17.9:-14:6000 144669--16)
(1.2e 4e1211)))))
(1 (1120.035+88607)
FSNot
9.0e-523.8-6
6:3e-7130e-8
0.109
-1.081 202275
0.62:000004
(1.50-3
4420--16.7766
(8.12810059)
(136.553:144761)
Nonconvex QCQP: v 100, Dag = 50, Pinng = 50
6.8e 144:244-16
4.9c 91:37-111
288 768-
Solver
(1.0e-13dt 13: 1.46-55)
(4.5e-8115e-9))
(15092, 165:1)655)
ESNet
9. (6.5e-6#22966)
19 (e
-3.437
-27.299 (9.466+40119)
0.539:10.018
(247.819++37.21)
Nonsmooth Nonconves SOCP: 100, meg
50, Mineq
7.8e-14:1..e--1 3e-16
3.4--9
13614465-5.100
Solver
(1.20-13a2.5-115))
5.96-81227e9
(21642.764:81.56)
6.26-5 (2.5e 3
4.20-7 56006)))
-9.625
FSNet
41.26349.4522
(658.5 55553111))
067.3889++66997
Table B.3: Test results of FSNet on 2000 instances of nonsmooth nonconvex problems.
25

---

Nonsmeoth Nonconvex QP
Nancanvex Nonsmaoth OCOP
Nanconvex Nonsmeath SOCP
0.6
0.6
0.3
Solver
Probability
FSNet
0.4
0.4
0.2
0.2
0.1
0.0
20
-18
-17,5
-15.0
-12.5
-5
Objective Value
Objective Value
Objective Value
Figure B.4: Distribution of the objective values in nonsmooth nonconvex problems.
Parallel Time
Sequential Time
10
FSNet
2Z
Soovver
Time (seconds)
101
10"
10-1
10
NSNC QP
NSNC QCQP
NSNC SOCP
NSNC QP
NSNC QCQP
NSNC SOCP
Figure B.5: Computational time on 2000 instances of nonsmooth nonconvex (NSNC) problems.
B.4 FSNet with truncated unrolled differentiation
This section evaluates the impact of the truncation depth on the performance FSNet (see Section 5). We set the maximum number of feasibility-seeking iterations to 50, and vary the truncation depth from to 50. The results on the smooth convex problems are shown in Table B.4 and Figure B.6.
Training
Inference Time (second) seconds)
Batch Time
Sequential Time
Training Time (hours)
0.8
Inference Time (seconds
0.6
400
0.6
0.4
300
QP
QCQP
0.2
200
0.4
SOCP
0
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
Backoropagated Iterations
Backpropagated Iterations
Backpropagated Iterations
Figure B.6: Training time and batch inference time of FSNet with different truncation depths.
When K' 0, no iterations of the Ceasibility-seeking procedure included in the computational graph, so it acts as non-differemtiable pass-through layer. This yields the smallest graph and therefore the shortest training time, but incurs large optimality gap, indicating the network cannot learn the true solutions of Problem (1). As K' increases, the training is successful with small optimality gaps (and near-zero constraint violations), yet training time grows due the larger graph and additional backpropagation cost (Figure B.6). Importantly, for K' between 10 and 50, both total constraint violation and optimality gap remain essentially unchanged. This means that modest value of K" is sufficient to render the bias caused by the gradient truncation negligible, which agrees with our analysis in Appendix D.2. This suggests a practical tip that, rather than fully gradient-tracking all iterations of the feasibility-seeking procedure, we only need to track a few of them while maintaining high performance for FSNet and reducing the training time.
Since truncation affects only the backward pass during training, inference times are virtually identical for all choices of K' 4 O.
26

---

Tracked Iter. K'
Total vio.
Optimality Gap (%)
Runtime (s)
Training time (h)
(Max)
Mean (Max)
Batch (Sequentia))
Convex QP: n
100, Rec
100, Dince
100
4.40-51le6 (4.2e-44442e5)
14.143110024 (16.31440.488)
0.174100066
0.348
(296.889
10
(6, 1o-1e-4)))))
7.5e- 555--556.066
(0.05020.005)
0.01520000
(168.69677.896)
0.13440036
0.490=00015
6.8e-5483666
0.011:0000
20
(5.50-42:46e-5))
(0.034:0.002)
(162.488:377754)
0.068=00077
6.65-54555e6
0.015#0.002
0.1244:0053
30
19:2--41110-4)
(0.0462200088)
(166. (166.242::45.944
0.695=0:008
6.60-3:15.5e6
0.015:10.002
0.125:1:0:010
40
0.682
(9.22-4--19e4)
(0.046-+0.00))
(136.59933.09))
6.60-5:155e6
0.015:10.002
0.125:1:0033
50
0.691=0.019
(9.2e-4+-19e44)
(0.046+00005)
(176.246+449894)
QCQF:
100, meg
Fineq
7.2e-5444.5e-7
37.17444.148
0.648-+0003
0
0.461=00011
(79974)))))))
(45.58440.395)
(284.445=357722)
6.28-54-2886
0.032440.02
0.475-40008
10
48.1e-44600-55
(0.40S==0.022)
(193.722:22,770)
0.536
6.20-5:200-6
0.032:1..022
0.451
20
0.77
(13-1-4)))))
(0.411=00.02))
(233,545=22.455)
6.20-54588-6
0.035
0.457::0009
30
0.757=0.028
(2.10-03:15.7-44)
(0.432:10.070)
(213.981:
40
(2.1o-03:157e-4)
8.26-54-58666
(0.432:10.070)
0.035:20.001
(200.018:26.11))
0.440::0006
0.729=0.015
6.28-54-5.866
0.035
0.446:+0000
0.765
(2.1e
(0.432:00.00))
(174.800=:28766)
Convex SOCP: n. 100, neg
50, Piuce
50
4.8o-5122-5
181.38:10002
0.416:00066
(4.70-4:1744e5)
(194.957::0896)
0.406:0.032
6.6e:54773e6
0.136:20.010
0.288++0008
10
0.532=0.050
(6.2e-446636-5)
(1.006220.443)
(252.299=55.041)
20
6.2e-54t:22e66
0. 0.0622
0.289++0003
0.717
(1.914220:49))
(187.842=26.580)
6.3e-5466.e6
0.159:20.005
0.336-+0036
30
9:.00-5))))
(1.869)0.776)
(272.440=52.514)
0.780
6.3e-5::6 Je-6
0. 159:10.005
0.297:10.10
40
0.804:10.015
(6.70-4119955)
(1.889 0.3376)
(214.608:1 17.089)
50
6.3e-5:261ed (6.70-419.9e5))
0. 0.00
0.297:1:0008 (259.888:130.355)
0.77
(1.37)))03766
Table B.4: Numerical results on 2000 instances of smooth convex problems with varying numbers of tracked iterations.
B.5 FSNet with different values of
This section investigates the impact of the value of penalty weight the performance of FSNet (see Section 3.1). Table B.5 shows that the constraint violations are near-zero for choices of p. since the feasibility-scekin NN's prediction. This is an advantage of FSNet, as the feasibility is guaranteed by a well-designed feasibility-seeking step, rather than being left to the NN alone. While the optimality gaps similar across different p. both training and inference times decrease as p increases (Figure B.7). Actually, larger penalty weight strongly penalizes the distance between the solutions before and after the messbbilityseeking step, driving the NN to produce outputs that are closer to the feasible set. This reduces the number of iterations required by the feasibility-seeking step, leading to lower computational time for the entire process. For example, setting p 50 makes the training 1.4-1.6x faster and inference 1.3-1.7x faster compared to p However, excessively large values p may cause the optimization to overemphasize feasibility at the expense of optimality, potentially slowing convergence of the network parameters.
step always tries to enforce the solution's 's feasibility regardless of the
27

---

Total vio.
Optimality Gap (%)
Runtime
Training time (h)
Mean (Max)
Mean (Max)
Batch (Sequential)
Convex QP:
100,
100, Pineg
100
5.8e-05 26--0
0.008:00001
0.183+00055
1.07==0.227
(5.8e-04:1 1.0-04)
(0.029=;0003)
261.704413447)
6.5e (0.:000:000000
0.009:+0.01
0.150+
0.5
(9.54-0422.4o044)
(0.031=0000)
(236.6862210.166)
0.827=6076
0666-05-
0.011:100002
0.183:10.004
2,5
0,747=0.044
1..8o-4421.4o004 (0.041
(196. 120=247.620)
6.62-05::5.5e-66
0.015:10002
0.142:10.027
0.687
(0.046++0008))
(186.968::34119))
7.10-0526..9-00
0.014110001
0.117:1:0029
0.676:20:022
19.6e-04--22.0e0000
(0.042-+00000)
(152.014::22.92))
6.7e 4:24465
00001
0.133+00.000
20.0
0.649=6.013
(1.2:-03=+2..e00))
(0.042++0003)
(165.496-123.169))
0.1044:0.013
50.0
(9.3c 2.5042)))))
(0.074:0016)
(15.5699:315576
0.661
Convex QCQP:
100, Reg
Rioeg
50
7
0.02340002
0.576:::0044
(0:315:10.06))
(257.309:15.105)
1.039
1.76-0551..0-05
0.028:10.02
0.487:100055
(1.0a-03:11.e-00)
(0.412::00075)
(216.307:312115))
0.811000100
7.
0.030=20.00
0.4643:00004
2.5
0.77
(1.62-03::336--00)
(0.411:200033)
(187.807::11011)
5.0
(2.17433)337744444
6.2e-05::55886 068
(0.432+0001)
0.035:10.003
(180.2824432.99))
0.4514:0008
0.743=0.014
6.16-05::6..ee00
0.038-10.00
0.450:10.014
10.0
0.715=00006
(1.5e 1.5e-033:1.22-18 (0.417=+0088)
(222.192::36.58))
5.8e-05
0.000
0.437::00009
0.762+00052
(2.4e-03=669e-44))
(0.466=+0008))
(240.089:133.113))
3.3e-0521..0
0.049:20016
0.447:10015
50.0
(265--03))
(0.486110:08)
(225. (23:130))))661)))
0.716=60031
Convex SOCP: n 100, Begq
50, Raneq = 50
5:32-05=2.26-06
0.356:20097
0.360:10:01
0.976=00.04
(7.22444)))))444)
(1.490++0278)
(310.059=71.100)
0.5
7.20-05=8.3e-06
080:100012
0.325:10.017
0.921=0.014
06666-04 (2.347:10514)
(237.029:172.66)
6.62-0522926-66
1.1410001
0.294:10:007
0.797=0.028
(7.0c-04=1.ee00) (0.978:10.109)
(206.250:115333)
6.3e-05-66.1e-66
0.159-+-0005
0.307-+0.014
5.0
0.837=0.005
(6.7e (6.9963)) (1.889+0337))
(8117644118466
10.0
0.174:+u020
0.291
0.725=0.011
(7.7e 2072-344)
(1.365+0021)
(19866422155576)
6.1e-00--441e-66
0.184440077
0.287-40.013
20.0
(9.5e 14
(0.994-20.94)
(197.955=333.944)
0.714=00041
6,3 1e000
190:20066 (1.093:10.192)
0.269:10.015 (172.091::41.039)
50.0
0.694=0.027
(7.42-04-922--5))
Table B.5: Numerical results of FSNet on 2000 instances of smooth convex problems with varying
Training
Batch Time
Sequential Time
Inference Time (seconds)
Inference Time (seconds)
Training Time (hours)
QP
1.0
QCQP
300
SOCP
0.4
0.8
200
0.2
100
0
0.5
2.5
5
10
20
50
0.5
2.5
5
10
20
50
0
0.5
2.5
5
10
20
50
of
Value
p
p
Figure B.7: Training time and batch inference time of FSNet with different values of p.
28

---

B.6 AC Optimal Power Flow
We evaluate FSNet in solving the AC optimal power flow (ACOPF) problem in power systems. We use the formulation and dataset in [46]. In the following experiments, we use 10000 samples for each grid size, split into 8000 for training and 2000 for testing. The parameters x represent the load demands, and the prediction (o(z) of the neural network is the voltage magnitudes and angles of all power system buses. The active and reactive power generations can then be computed from the voltage magnitudes and phase angles via the power balance equations. In particular, the power output of the generator at bus is given by
pē = p +
g20 +
Σ
D
Σ
P.:
3EE
BEER
of = 1h b
Le
Σ ge-
3EE.
E
p4 ge dh + ge uey cos 0,) + byth 0. sin(0: 00),
Ve (i.j) E,
de D2u u/ beru cos(O; (0; 0,) + ge v.yy sin - -0));
Ve = (1.5) E E,
pe = " f + _ v,y cos(O, - 0z) - 0%d u,yy sin(0: - f,),
VE = (1,5) E,
d. - 03 6yd u.úy cOs(O; 02) 9hd 0. sin - 0,),
Ve (i,j) E E.
In addition, the Levenberg-Marquarte Marquardt method employed for the feasibility-seeking step.
The results for the IEEE 30-bus and 118-bus systems under three random seeds are presented in Table B.6. FSNet achieves small (numerically zero) equality and inequality constraint violations, and significantly outperforms the Penalty method. Moreover, the optimality gap remains below 1% in both test cases. In terms of runtime in the sequential setting, FSNet significantly outperforms IPOPT, achieving 3 speedups on the IEEE 30-bus system and 11x on the IEEE 1 118-bus system. The Penalty method, on the other hand, despite its fast inference. suffers from high constraint violations. particularly in terms of maximum violations on the IEEE 118-bus system. Overall, this result is consistent with our previous findings on convex and nonconvex problems.
Equality Vlo.
Incquality Vio.
Optimality Gap (%)
Runtime (s) Sequential
Method
Mean (Max)
Mean (Max)
Mean
Min (Max)
IEEE 30-bus: a = 60, Req
60, Bieg
248
6.70-7:000
5.4e-4000
IPOPT Solver
$1,53+00
6:000)
(1.1e-6+00)
FSNet
1.85-547..56 (12.1-3)))11144)
1.4e-844.779
0.81
0.07:4:1.7
16.43++0.13
(6.7e441176-4)
(3.61+092)
Penalty
8.2e-415:0655
3.7 (0.026:18.5e-3)
0..13
(0.013=1..e-3)
2.24
(7.09.10665)
0.52:0033
IEFE 118-bus:
236, Blg
236, Raneg
1196
1.20-62000
3.1o-8100
IPOPT Solver
(2.8e-5:n0))
(1.le-5500)
392.83-000
1.8e-5:11.ee5
2.40-6:22.ee-7
-1.67=00.2
FSNet
0.91000
33.35 10.6)
0.0017
(0.043)18.5e-3)
(4.91
Penalty
2.50-3:13.20-4 (0.124:0.014)
3.5 (0.28)+10041)
0.49
-4.67=0.11 (1.51:0225)
0.81
Table B.6: Test results of FSNet on 2000 instances of ACOPF problems.
B.7 More experiments for DC3
In [13], the authors mention that unrolling more correction steps may improve the feasibility of the prediction. To evaluate this, we ran DC3 with different numbers of correction steps on 2000 test instances of smooth convex SOCP. As shown in Table B.7, increasing the number of correction steps only yields a marginal reduction in inequality violations (due to the default small step size of 10 10-"), while significantly increasing runtime.
Next, we evaluate the performance of DC3 with different correction step sizes. We fixed the number of maximum correction steps and changed the stepsize obtain the results in Table B.8. Stepsizes
29

---

Max Correction Steps
Eq. Violation
Ineq. Violation
Optimality Gap (%)
Time (s)
10
2.30c-14 14
0.024
0.051
0.095
50
2.30c-14 14
0.024
0.051
0.474
100
2.27c-14
0.024
0.052
0,948
200
2.28e-14
0,023
0.052
1.896
500
2.28e-14
0,022
0.054
4.738
1000
2.34e-14
0.020
0.057
9.526
2000
2.28e-14
0.016
0.061
19.046
5000
2.27c-14
0,010
0.070
48.0 48.024
Table B.7: Sensitivity of DC3 performance to the maximum number of correction steps 2000 instances of smooth convex SOCP problem, with fixed correction stepsize of 10-?
Correction Stepsize
Eq. Violation
Ineq. Violation
Objective Gap (%)
Time (s)
le-07
2.30c- 14
0.024
0,051
0.338
lc-06
2.30c- 14
0,024
0.052
0.121
le-05
2.28c-14 14
0.020
0.056
0.095
le-04
2.28e-14
0.004
0.081
0.096
le-03
2.31c-14
0.000
0.130
0.095
le-02
-3.52e+12
2.89e+30
1e58
0.095
Table B.8: Sensitivity of DC3 performance to the correction stepsize on 2000 instances of smooth convex SOCP problem, with fixed maximum correction steps of 10.
up to 10-$ achieve limited violation reduction, whereas stepsizes of 10 and 10-3 substantially mitigate violations. However, overly big stepsize (e.g., 1) causes the correction procedure to diverge. Note that all experiments here are inference time. During training, a large correction stepsize can destabilize the optimization, as noted in [13]. This demonstrates the DC3's sensitivity the correction stepsize, whereas our reasibility-seeking step, implemented via L-BGFS with backtracking line search, mitigates such sensitivity and delivers robust performance across diverse problems using the same hyperparameters.
30

---

C Hyperparameters
For DC3 and Penalty methods, adopt the hyperparameters tuned in [13]. For the Adaptive Penalty method, we performed brief manual sweep to identify settings yielding strong empirical performance. For FSNet, we configured the L-BFGS solver with sufficiently large maximum- iteration budget and memory size to guarantee convergence on all the problem classes. We also evaluated several neural network sizes and learning rates to ensure sufficient expressive capacity and stable optimization for all problem classes. Table C.1 lists all hyperparameter values.
Hyperparameter
Valuc
Shared:
Learning
Se-4
Learning rate decay
0.5
Number of hidden layers
4
Nonlinearity
SiLU
Number of hidden neurons per layer
1024
Train/validation/esst ratio
0.7/0.1/0.2
Minihatch size
512
Random seeds
2025, 2027, 2029
FS:
Equality violation weight
10
Inequality violation weight
10
Max L-BFGS iterations
50
L-BFGS memory
30
Epoch
100
Learning rate decay steps
2000
Penalty:
Equality violation weight
50
Inequality violation weight
.50
Epoch
1000
Learning rate decay steps
4000
Adaptive Penalty:
Initial equality violation weight
30
Initial inequality violation weight
30
Max equality violation weight
500
Max inequality violation weight Increasing rate Epoch
500
2
1000
Learning rate decay steps
4000
DC3:
Equality violation weight
10
Inequality violation weight
Correction stepsize
1e-7
Max correction stepsize
10
Correction momentum
0.5
Epoch
1000
Leamiing rate decay steps
4000
Table C.1: Hyperparameters
31

---

D Analysis and Proofs
D.1 Proof of Theorem
Theorem 2. Suppose that E,, [ G for all and that the feasibility-seeking step is unrolled for K steps in every forward pass. Let be as defined in Theorem 1. Under Assumption I, 2, and 3, after running the SGD for T iterations with step size n7 no/ V for some Nju > 0, there is least one iteration E {O..... }} satisfying
|VL so (0(T-112)
(11)
[[V S 50 (T-1114+
(12)
Proof. Gradient bound of finite-unroling loss. For the first part of the theorem, follow the analysis of the SGD for L-smooth function. By the quadratic upper bound of the L-smooth function, have
L( < L(0.) + (VL(0)), 01) + IN
110++1 0.112-
Observing from the SGD update (10) that 0z+1 - 0r = - taking the conditional expectation given all randomness up to time t, and rearranging terms, we obtain
L - - (L())) - E,((((++1)) + 17
LN E, [(
2
||5L(0))|
Taking the full expectation for both sides and using the assumption E,
< G yields
E [ [ < (L(O.) - B[L(0.+))) LNnG
Finally, we take the average over O,..., and use the telescoping sum to get
Σ [IVLL()))13]
I(0b) L
LNnG
nT + 2
Since (0...,.1 ( 1/ <
obtain the first statement of the
theorem by replacing the stepsize 1] = no /TT.
Gradient bound of infinite-unrolling loss. difference:
For any 0, we consider the norm of the gradient
VV VL(0)]12
V (z) - V - TeF(vo(1(1)) 1000000(200) (z((1))
=
S
V o Fyo (vo (z) VoF(vo(z(0), ).0z (z)()))
(D.1)
which follows from the triangle inequality, We consider one element of the sum, and let g = V (oF(yo (o(x)) - ()o()) ui (x))]]- Then, via the chain rule, we have
[[(avo V ((o((), (o(x)) + (Jovo())) (T, x)) Fg (V, (x))
(V (z))ll
(z))
BBy (I V ())(()))) (o(=)) (J,FS* ()o()):))) (V, i0 (z))
where the inequality follows from the bounded Jacobian in Assumption 2.
32

---

The first term becomes (by expansion of the gradient terms):
| (( F(jo(())) 06 (z)) = p)(5o(x)) 10 (x)ll2-
The second term:
(1) (x); x)) V, (00) { V if (z))]l2
(I) ( - (J,FS* (v)(
(o(x))
+ ( [1)
(z))) 1u)) 10
(z))
+ +1J,FF (yo (100 Vy 10 (x))l12
Bp do (z) 10 (z)112
- - (,0 | 25 (x)1l2
=(BLFF + BBwLL)))(lllll
(z))12
where the second inequality follows from the smoothness of the feasibility-seeking mapping and function (zo(z)), 3o (z)) (Assumption 2).
Let V + BFLFs + BFsLp. Thus, g is upper bounded by
g v (a) Üs (x())) 1--
Using this upper bound and Theorem we can rewrite the gradient difference (D.1) as follows:
|VC(()) VL(0)l2 S
1
11l0 (a(0)) (x((11)))
5
Sun L3
(d(vb)); (0 (0)) ((00(1()):
(D.2)
5
2
where (0 = (S( (x(()); x())).
Let (00) E Sung 1. 7 K (Ø(yo (x(())((()))))))))))))))))))(( (((0))) -000))))0(0000)0000
2000
Using the inverse triangle
"g
inequality, the inequality (D.2) leads
|L((0)aa S (|VL(0))|2 - -
We consider the time t and use (11) to obtain
B|ll [()))(()) - [[
<
(D.3)
Finally, we need to show that (()().)) is upper bounded. We have
Sun 2
[[0(0,)) 5 1
E (v)o(a())); - ())(
(D.4)
Since Theorem shows that the gradient descent converges to the minimum, we have (1o xz 2 (zx((x))); (()) Thus, it suffices to show El (zz(0); (())) is upper bounded for any i. The argument proceeds as follows: We first show that the NN parameters remain bounded during training with SGD, which implies that the network output ()(())) is also bounded. From there, we conclude that the expectation E (z(())((())) x0) is bounded from above.
Bound of the NN parameters: We first establish the bounds of the NN parameters during training with SGD, which via (10) exhibits
||0 0g 2| L t t= O,..., - 1.
Taking the sum over j 0,...,t t and using triangle inequality and the telescoping sum, we have
|0 00ll2 - Σ
110 0,112
ml (())((,
1=0
j=0
33

---

Taking the square for both sides and using the Bauchy-Schware inequality yields
10, Bol
IV S TTT j=0
(VL(0))
J=0
J=0
Using the assumption of bounded variance of the stochastic gradients, obtain
[1 8o - 1n TT
By replacing 1 = 10///// and taking a regular expectation, we finally obtain
E - 8ol 21 < TGG.
(D.5)
Bound of the NN output:
For notational brevity, we denote No = ye, and yt ye, (2), which are outputs of NNs with initialized parameter 8g and parameter 8, at time t. By the mean value theorem and the bounded Jacobian ||Joyo (x)ll2 & By in Assumption we have
1/ye voll2
BN 110 00l12.
which, squaring both sides, leads to
Ellyy - vol121 < E[
0oll - B TG.
(D.6)
Bound of E[o(y;;
By the L-smoothness of (();:)), we have the quadratic upper bound: d(y;; x) I ((vb: x) + (Vy x), y1 yo) + L 11 -uul22
Taking the expectation on both sides and using (D.6) yields
Elo(ye x)] S o(yo:x) + BNTV V ( + Lo B² BTGT < oc.
2
Therefore, we have that E[c(vo (x(0)) [(0) is upper bounded for any i, which implies that E[ó(ūo (z) [ is similarly bounded given that ((jo(=()) x0()) < (x))))()); by The- orem Finally, by (D.4), there exists a constant & > 0 such that ([)(),)) S k for = =o....,
Using this in (D.3), we complete the second statement of the theorem.
D.1.1 Proof of Lemma
Lemma 1. Suppose SGD converges to 0*, where J(0")= = (JU))))))(()) (Joyn (c(51)) has full column rank. Then, the prediction of the NN ya(x(0) is the stationary point of F(yo(x(1)), No (0)) with (o(x(()) FS(ya (x;0);; for all i E SI.
Proof. the stationary parameter 0*, the gradient of the loss function vanishes:
1 0 = VC(0)) = s
Σ (x(0))
V
(z)
+ (J,FF( (a (0)) x (())) ( F 30(x(1))
Let ((t) VyF(zo(Z(1)), (0))))))))))((()))))))) + (5)) T (z(00) (o(x) The stationarity
results in
(1)
J(T))
0.
((8)
Since J(0*) has full column rank, this implies
(v(1)
(()))
0. Thus, the NN output you (2:())
and go (2) ())) satisfy
(1) = (ye ()), do (z) + ;);
T VV (z)) = 0,
which is the stationary condition of (z)o 3 (x))
34

---

D.1.2 Proof of Theorem 3
Theorem 3. Given specific parameters suppose that the NN predicts y- which is global minimizer of Fly, y: p+) ) = f(y: 1)::)))))))) Pr lly ell with ÿ FS x); that the mapping FS continuous; and that O < Pr < Pr41, VT and pr + xc. Then, every limit point ("'"") of the sequence {(rr: admils y = 1j", and 1* is global minimizer of the optimization problem (1).
Proof. Let y be global solution to problem (1), that is
fi; x) - f(;;
for all the feasible points D. We also have that y = FS(T; x) given that is already feasible. By the global optimality of (Ur: 0r), we have that F(yr, DriPt) S Ply. Wi p+) which results in the inequality
f()::)) + PT ll, - 2-113 f(T:)) + Pr 1 712 = f(T:x).
(D.7)
2
2
By rearranging this inequality, obtain
2 |1l- - 00112 (f(::)) - f()::)))
(D.8)
Pr
Suppose that y² is limit point of {vr}r2a. so that there is an infinite subsequence T such that
lim /r = y² TET
As the feasibility seeking mapping is continuous by assumption, we can define g = limyer z)). By the continuity, we have
lim ||l- 1-112 = lim I|l- FS(U+i x))2 110" 11 FET
FET
Taking the limit as r 00.7 E 'T on both sides of (D.8) yields
er 12 < lim 2 (f(T;)) f(riz)) 0.
2
1 5 2 ret 11
TET Pr
Therefore, we have that y² = 0"" Note that 1 is the output of the feasibility-seeking mapping, so y and i" are feasible. Moreover, by taking the limit - 0o, T in (D.7), we have
f = f(0"(()))))))))))
lim TET 2
Pr lll, 0-11 - f(::z).
Since 5 is a feasible point with an objective value no larger than that of the global solution T, must itself be global solution.
Using a similar argument, we can show that ÿ is a minimizer of Problem under the same assump- tions, even when p However, in practice, we often choose p for the reasons discussed in Section 3.1 and Appendix B.
D.2 Analysis of Truncated Unrolled Differentiation
In this section, we build intuition for why truncating backpropagation remains effective in practice. Specifically, if the violation function Φ(s, x) is locally strongly convex, the bias introduced by truncation decays exponentially with the truncation depth K'. Consequently, a relatively small K" suffices to yield a high-quality gradient estimate.
The violation function s > ((s; twice differentiable. For sk R., there exists a neighborhood N of sæ where s + 6(5;z) is p al-strongly convex. Moreover, for some Ko E N, the iterates satisfy 8s: E N for Ko-
Assumption
Although the violation function involves max operation and is not twice differentiable at the kink points of the max, we can approximate max practice using the softplus function, which infinitely differentiable and can approximate the max arbitrarily well. Therefore, this assumption remains reasonable practice due to the smooth approximation provided by the softplus function. This assumption enables us to bound the bias caused by the truncation as in the following lemma.
35

---

Lemma 2. Suppose Assumption 4 holds and that the feasibility-seekiin procedure is unrolled with the stepsize Te (0, L/) for a total of K iterations, but gradient tracking is performed only for the first K' E [Ko, K] iterations, with the remaining tterations treated pass-through during backpropagation. Define the true Jacobian when all K iterations are tracked as JyFS (yo(z); and the truncated Jacobian as y FS (2)
Osk
Osk'
JyFS (x));)) Osp
FS (vo(r);;))
8so
Then, the bias caused by the truncation satisfies
|| (BFF5((uo(z))))
JS (vo(t); S C(1 5k y8
where 8 nopu" 10, and C is a finite positive constant.
Since the truncation occurs during the backward pass, it also perturbs the stochastic gradient. We denote the resulting truncated gradient estimate by &urun L(O.). The next lemma quantifies this effect.
Lemma 3. For gradient estimate without truncation, suppose E, [[VL(0)) S and
[.( VI((.) for all t. Then, when the gradient truncation is applied, under Assump- tions in Lemma 2, the truncated gradient estimate satisfies
Ex
V trun L(0.)1I G HOK'
Bg trun L(0.)] VI(0,) +EK-
where a = O(8K') and EK E IR with ||ex> = 0
This lemma shows that the truncation introduces bounded biases to the variance and expectation of the gradient estimate, yet these biases decay exponentially fast in the truncation depth K'. This qualification allows us to understand the behavior of the SGD when the truncation is applied.
Theorem 4. Suppose Assumption
3, and the assumptions of Lemma 3 hold. Then, after running the SGD for T iterations with step size nuo VT with nj O, there is at least iteration t E {O..... T 1} satisfying
S 00 = [T(T11268*)) +
(D.9)
B[ O (T- + 2 + +8k))
(D.10)
where 7 and 8 are defined as in Theorem and Lemma 2.
This theorem characterizes how the bias caused by truncating the gradient tracking in the feasibility- seeking procedure propagates to the convergence of SGD. We see that the upper bounds of the expected gradient norms contain bias terms of order () and O(5"*). Since 0 < 6 < 1, even modest values of K and K' suffice to render these biases negligible in practice. We now present the proof's of the above lemmas and theorems.
D.2.1 Proof of Lemma 2
Lemma 2 an immediate result of the following lemma.
Lemma Let yp Rd + IR be twice differentiable, L-smooth, satisfy the PL condition with constant μ. Suppose there exists neighborhood N of sk on which ip is strongly convex, i.e. (2p(s) = =1I, E N with p' Define
0SK
K-1
Osk
K'-1
jerue-
(T-
Jerun
II k=0
Oso
Isa
k=O
If St E N for all k > Ko and stepsize of gradient descent is chosen 1) (0, then for any K" [oo, K), the error matrix E Jurue jtrun satisfies
| < C(1 - 8 - )8"
where 8 =1 =1-me 1-mp' E 10,1) 1) and C 48 Ko + Ko.
36

---

Proof. Define J
0sk-1 88k
= I - [VV2p(sxx and rewrite the norm of error matrix E as
[B1l |[ ................................ Jk -) Jo)ll2
1 |JK-1..... JK 111.)l/x'-11....00 Jol2
(D.11)
Step 1: 0 <
For Ko, the L-smoothness gives LL 3 ((s() 3 LI, so
||&ll22 || - ()4p (s))l ll s 11+10 1+42 nL.
For Ko S k < K', strong convexity implies μ 3 V p(s)s 3 LI, and then we obtain
| 12 ~ - mpp..
Let 5 = mil [0, 1). Then, the sub-muliippiicatiitty property of the spectral norm obtain
[[Jk-11... Jo)ll2 & Jx,
Jo)ll2 s + mL)Ko.
(D.12)
Step K' 5 K.
Use the telescoping sum for [[ Jk' 1112
K-1
K-1
|| ... JK' - 1112 =
Jj
(Js - 1)
K-1
K
Jj
1
-
k=
K
i=b+1
From the L-smoothness we have || - T = ? S nL. For & 2 K' > Kg. by assumption, in the neighborhood / where yp is locally strongly convex, and thus obtain
K-1 l II J
5
j=k+1
1
2
which results
K-1
|| - JK /1|2 S nl.
Σ
8k-k-1
k=k
We change the index of the summation i K
When K', K K' 1. When
A = K 1, i = 0. The sum becomes a finite geometric series with 0 < 8 < 1:
K K' -1
|| Jr. 1112 S 1
Σ
i=0
= -n 8K-K"
/ (1 - gk-K').
(D.13)
Step 3: Combining the bounds
Substituting (D.12) and (D.13) into the inequality error matrix (D.11) yields
| S 11'
1 - _Kk -K-K's
Ko -Ko +
Ko.
Define the constant
4 8- Ko 4 +
1 Ko, We can rewrite the bound as follows:
E 2 S C(1 - 8k K' )8
37

---

D.2.2 Proof of Lemma 3
Proof. For a minibatch with D samples, we have
VI((0)) - trum T(O0,)
1 D Σ (JoBo( VV, yy (1)((( (x(0))
(Joyo (x )) (J,FS* (yo (z) (z))0) ;z) V V (1)(((())) 15050(''0 (z()))
+
(J)) V ((yo(x())) ).98 (zx
(Joua(z(")) (J (S ();
P (vo(x! 2000 (xr
1
(vo(x)
D
FS (y0(z)())=²0)) V ( V (z(11)
BN Cp - 8k- oK-K'jaa )ok
(by Lemma 2)
1=1
Bp - )0"
(D.14)
Then, by triangle inequality, we have the following bound
R2
A trum T(0..)
[]]] < E2
(| ((0.)) +T.(0:)1= +|
I Bx - -K 2 +G+ ++++BBBBCVG11 8 K 38
G O(8*')
In addition, by (D.14), we have that
vI(())) trrum L(0.) s (L((0)) - [(0))
S Bn Bp 3K-K')88K UK'
which results in
((,) uK 1 5 Strun Strum L(O.) - (L.O.) 1 (element-wise).
There exists Ege TM such that ||= |12 S UK' vng that satisfies
tr T.(O.] E. [+T(0)) + VI.(0.) tK--
E
Since UK. = O8**), ). the proof is complete.
D.2.3 Proof of Theorem 4
Proof. Gradient bound for Emite-unrolling loss with truncated gradient. By the quadratic upper
bound of the L-smooth function and bounded variance of the stochastic gradient (Lemma 3):
[L(++0++)] < L(O.) - 7VVL(()), L(0.))) + LNT E, Stron L(0.)
LNn
= =(()) - - + Ex.) +
2 (G+ok).
(D.15)
38

---

We observe that
(VI(O.); VI(0.) + = [VL(0)) + (VI(O),, ex)) +
s 11
elx 12
2
2
2 IVV(()))
マ |lex: 112 + (IIV 2( + 11xxr11)
7 IVI(0)))
|| 11 VT((0)) +
2 2 IVL(0.))
| 112
Using this in (D.15) and rearranging terms, we obtain
(V.().) - (L((,) (.(L(0++1]) + | 112 LNu(G
Taking the expectation for both sides and the average over T steps, we obtain
2
L(.)) ((00))(())))
T/T
2
- 1/T
(L(0)) - L3) + 1|5x0112 LNn(G+
Since min (T1... ( 12 S - 2 we obtain the first statement of the theorem by replacing the stepsize 17 /TT.
Using the same argument the proof of Theorem 2, there exists positive constant K1; K2 such that
k2n"<1 + 1lEx112 + L no
C[) s ks 7/4 1/4 + Kg? k-1
VT ok.
39


Reviews:
[{"review_id": "IZ4PGjsARd", "reviewer": "Reviewer_C6cv", "summary": "This paper proposes FSNet, a neural network framework that integrates a differentiable feasibility-seeking mechanism to solve constrained optimization problems efficiently. A feasibility-seeking procedure is performed via vanilla gradient update. Unlike traditional solvers that are computationally intensive or learning-based methods that often produce infeasible solutions, FSNet explicitly minimizes constraint violations during inference, ensuring solution feasibility. The approach supports end-to-end training and offers theoretical guarantees on convergence and feasibility.", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: The authors present their ideas clearly, with well-written content and detailed mathematical derivations.\n\nWeaknesses: The proposed gradient step-based method is similar to optimizer-based constraint satisfiability layers, such as CvxpyLayers, OptNet, SATNet, GLinSAT, etc. Some of these layers use interior point method to solve the projection problem while some of them use gradient-based first-order methods. Actually, the proposed method FSNet can be regarded as a projection layer whose objective is to minimize the constraint violation and the resulting (convex/non-convex) optimization problem is solved by the gradient-based method. One potential gap between the theorem in this paper and the reality is that, it is usually hard to decide Lipschitz constant in the actual computation. Instead of everytime tuning the stepsize by hand, will it be better to automatically approximate the Lipschitz constant especially in convex cases? Besides, if there is some potential way to tackle the stepsize in non-convex cases, that will be much better.", "was_revised": true, "final_justification": "The authors' response has adequately addressed my concerns, and therefore I have increased my score.", "rebuttal": "Thank you to the reviewer for their support and thoughtful comments.\n\n> The proposed gradient step-based method is similar to optimizer-based constraint satisfiability layers, such as CvxpyLayers, OptNet, SATNet, GLinSAT, etc. Some of these layers use the interior point method to solve the projection problem while some of them use gradient-based first-order methods. Actually, the proposed method FSNet can be regarded as a projection layer whose objective is to minimize the constraint violation and the resulting (convex/non-convex) optimization problem is solved by the gradient-based method.\n\nWe would like to emphasize that FSNet is conceptually different from the mentioned frameworks. The projection-based approach solves a constrained distance-minimization problem to map an infeasible point back onto the feasible set. Unlike the projection-based approaches, **FSNet gets rid of the notion of distance minimization and instead directly minimizes the magnitude of constraint violations**. Our approach offers advantages related to scalable computation and broad applicability as follows:\n\n- The feasibility-seeking step is faster, more scalable, and easier to parallelize than the projection layer, primarily because it involves solving an unconstrained problem, whereas projection requires solving a constrained optimization problem. In our experiments, the projection layer using qpth (i.e., OptNet) is even slower than directly solving the original problem with OSQP. As shown in Table B.1, the projection approach takes 13.71 seconds to solve QP problems in parallel, compared to 3.802 seconds with OSQP and only 0.121 seconds with FSNet. This means FSNet is approximately 100$\\times$ faster than using the projection layer. In addition, we also attempted to implement CvxpyLayers for general convex problems, but found it significantly slower than qpth for QPs and too slow to be practical for other convex problems.\n    \n- The differentiable optimization layers the reviewer references only support linear or convex constraints, e.g., CvxpyLayers (disciplined convex constraints), qpth (linear constraints), SATNet (semidefinite constraints), and GLinSAT (linear constraints). While nonconvex constraints could in theory be handled by implementing custom differentiable nonconvex projection layers, we opted not to do this given the poor computational performance of projection methods in the convex settings we tested. In contrast, FSNet can readily and scalably handle both convex and nonconvex constraints, enabled by our key idea of directly seeking a feasible point via an easier-to-solve optimization subproblem.     \n- The DC3 framework is another alternative to projection in the implicit layers literature that the reviewer references. However, it performed significantly worse than FSNet in our experiments.\n\nAnother major point of difference is that previous works on this topic are typically validated and interpreted primarily through experiments, and lack theoretical insights (e.g., [1], [2], [3]).\nIn contrast, within FSNet, we provide both an empirical framework with strong computational performance and generality, and also a theoretical analysis of feasibility and convergence. Put otherwise, FSNet goes beyond the previous approaches by offering both empirical effectiveness and theoretical guarantees.\n\n[1] Tanneau, Mathieu, and Pascal Van Hentenryck. Dual Lagrangian learning for conic optimization. Advances in Neural Information Processing Systems, 2024.\n\n[2] Priya L. Donti, David Rolnick, and J Zico Kolter. DC3: A learning method for optimization with hard constraints. In International Conference on Learning Representations, 2021.\n\n[3] Park, Seonho, and Pascal Van Hentenryck. Self-supervised primal-dual learning for constrained optimization. Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 4. 2023.\n\n> One potential gap between the theorem in this paper and the reality is that, it is usually hard to decide Lipschitz constant in the actual computation. Instead of everytime tuning the stepsize by hand, will it be better to automatically approximate the Lipschitz constant especially in convex cases? Besides, if there is some potential way to tackle the stepsize in non-convex cases, that will be much better.\n\nWhile the Lipschitz constant is commonly used within theoretical analyses of first-order methods (e.g., [1]), in practice, when actually running these methods, the Lipschitz constant is not used directly. In our experiments, we follow the literature in using practical methods as follows:\n- For the feasibility-seeking step, we implement L-BFGS with a backtracking line search to determine the stepsize for sufficient decrease at each optimization iteration. This solver is applied across both convex and nonconvex problems without requiring additional tuning. \n- For the NN training, we use the AdamW optimizer and perform a few trials to choose a proper learning rate. The same learning rate is used across all problems. It is also standard practice in machine learning to use grid search to tune learning rates when needed.\n\nWe would also like to emphasize that we do not use the assumptions in the theorems (e.g., the L-smoothness and the PL condition) as a starting point in designing FSNet. While the main contribution of our paper is the introduction of a new method for constraint enforcement in deep learning, the theoretical analysis is intended to provide additional insight into performance and design choices, and definitely does not cover all settings where FSNet can work well. As such, FSNet is not limited to the specific assumptions within the theoretical analysis.\n\nIn our experiments, we have already evaluated FSNet in settings where the L-smoothness and the PL condition do not hold. For example, while the PL condition may be (locally) satisfied in some convex problems, it is generally violated in nonconvex problems, which can have saddle points with vanishing gradients and nonzero function values. Similarly, the L-smoothness condition is violated in both convex and nonconvex SOCP problems due to the presence of $\\ell_2$ norms in the constraints, which make the violation function $\\phi$ nondifferentiable and thus not L-smooth. In addition, we add the $\\ell_2$ norm regularization in the objective function in nonsmooth nonconvex cases (please see Table B.3), which also makes the objective functions not L-smooth. Despite these violations, FSNet performs robustly across all tested scenarios. \n\nIn summary, the Lipschitz constant is used solely in the theoretical analysis; in practice, FSNet operates effectively without requiring it. Moreover, the theoretical analysis in this paper is aimed at offering insight into how and why FSNet works under certain conditions, and is an **additional contribution** compared to prior works providing only empirical methods. Importantly, the assumptions in theorems do not restrict the applicability of FSNet, as our experimental results demonstrate that FSNet performs robustly well even when the theoretical assumptions are violated.\n\n[1] Garrigos, Guillaume, and Robert M. Gower. \"Handbook of convergence theorems for (stochastic) gradient methods.\" arXiv preprint arXiv:2301.11235 (2023)."}, {"review_id": "m3lWn9JN8K", "reviewer": "Reviewer_d4Uk", "summary": "This paper introduces FSNet, a Feasibility-Seeking Neural Network framework aimed at solving constrained optimization problems efficiently while providing guarantees on constraint satisfaction and convergence. The approach integrates a differentiable feasibility-seeking step into both the training and inference pipelines of standard neural networks, minimizing constraint violations via a separate unconstrained optimization problem, which is solved iteratively starting from the NN predictions. Theoretical analyses demonstrate guarantees on exponential convergence to feasible points under standard smoothness and PL conditions, and convergence of the NN parameter optimization under SGD.", "questions": "Please list up and carefully describe questions and suggestions for the authors, which should focus on key points (ideally around 3–5) that are actionable with clear guidance. Think of the things where a response from the author can change your opinion, clarify a confusion or address a limitation. You are strongly encouraged to state the clear criteria under which your evaluation score could increase or decrease. This can be very important for a productive rebuttal and discussion phase with the authors.\n\n1. Can the authors provide quantitative analysis or further comments on FSNet’s performance when the smoothness or PL conditions are significantly violated—are there clear failure cases, or does the method degrade gracefully?\n2. How sensitive are the results (constraint violation and optimality gap) to the penalty weight $\\rho$ and unrolled step count $K$ in both training and inference?\n3. Are there foreseeable bottlenecks or challenges when scaling FSNet to thousands or tens of thousands of decision variables? Is the feasibility-seeking optimization step a limiting factor?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. FSNet is not limited to a specific problem class; it is applicable to both convex and nonconvex, smooth and nonsmooth constrained optimization problems. The feasibility-seeking module is conceptually general and practically flexible.\n2. FSNet achieves large speedups over solvers (up to 1000x in some cases), showing practical value for real-time or large-scale deployments.\n\n**Weaknesses:**\n\n1. The benchmarks are synthetic, following protocols from previous works, but there is a lack of real, application-driven case studies (e.g., energy systems, supply chains, robotics), which would better validate the practical impact and generalizability.\n2. The current method does not exploit specific structures of certain problem classes, which leads to FSNet being outperformed by highly optimized solvers. There is limited discussion on how to adapt FSNet to leverage such structure for further efficiency gains.\n3. The method’s performance and guarantees depend on tuning parameters such as penalty $\\rho$, convergence thresholds, and number of unrolled iterations in the feasibility-seeking step.", "was_revised": true, "final_justification": "My score is already positive, and I am happy to maintain it.", "rebuttal": "Thank you to the reviewer for their support and thoughtful comments.\n> The benchmarks are synthetic, following protocols from previous works, but there is a lack of real, application-driven case studies (e.g., energy systems, supply chains, robotics), which would better validate the practical impact and generalizability.\n\nWe have now run experiments on FSNet for the AC optimal power flow (ACOPF) problem in power systems. For the detailed results table, please refer to the third comment of `Reviewer q8xn`. FSNet satisfies equality and inequality constraints (numerical violations on a similar order to that of the IPOPT solver). Moreover, the optimality gap remains below 1$\\%$ in both test cases. In terms of sequential runtime, FSNet significantly outperforms IPOPT, achieving **3**$\\times$ speedups on the IEEE 30-bus system and **11**$\\times$ on the IEEE 118-bus system. The penalty method, on the other hand, despite its fast inference, suffers from high constraint violations, particularly in terms of maximum violations on the IEEE 118-bus case. Overall, this result is consistent with our previous findings on convex and nonconvex problems in the original manuscript. \n> The current method does not exploit specific structures of certain problem classes, which leads to FSNet being outperformed by highly optimized solvers. There is limited discussion on how to adapt FSNet to leverage such a structure for further efficiency gains.\n\nThank you for pointing out the need for discussion on this point. We believe that FSNet's slower sequential performance on the convex QP problem compared to OSQP is partly due to its implementation in PyTorch using Python, without speed optimization. In contrast, OSQP and other solvers are implemented in fast programming languages with a lot of solver-specific performance optimizations. We expect that, if FSNet were implemented with similar performance optimizations and in the same programming languages as these solvers, it could outperform OSQP. This is because solving the unconstrained feasibility-seeking problem used within FSNet is inherently cheaper and more efficient than solving the original constrained optimization problem.\n\nWe aim to design the FSNet framework to have broad applicability and thus avoid incorporating problem-specific structures within this initial general framework. However, there are indeed several straightforward directions to integrate such properties into FSNet (and we are in fact working on some of these in follow-on work for specific real-world problems such as AC optimal power flow). For instance:\n- Problem structure can guide the design of the solver used for the feasibility-seeking step. By exploiting the form or sparsity of constraint functions, a faster solver can be tailored to the problem.\n- In some cases, a subset of decision variables determines the full solution. FSNet can leverage this by having the neural network predict only this subset. The feasibility-seeking step can then work only with this subset, and then the full solution can be recovered after that. This reduces the problem dimension and leads to additional speedups. This approach is related to the variable elimination used in DC3.\n\nWe will add a discussion on this point to the paper.\n> (W) The method’s performance and guarantees depend on tuning parameters such as penalty $\\rho$, convergence thresholds, and number of unrolled iterations in the feasibility-seeking step.  \n(Q) How sensitive are the results (constraint violation and optimality gap) to the penalty weight $\\rho$ and unrolled step count $K$ in both training and inference?\n\nWe agree that the sensitivity to hyperparameters is an important aspect to understand. In our appendix, we provide the experimental results to evaluate the performance of FSNet under different penalty weights and varying numbers of unrolled steps. Overall, we observe that FSNet is not highly sensitive to these choices and maintains consistent performance across settings. We summarize the findings below.\n- **Penalty weight** (Appendix B.5): The total constraint violation and optimality gap remain stable across different values of the penalty weight. However, FSNet tends to run faster with larger penalty weights. This is because higher penalties encourage the neural network to predict points closer to the feasible set, reducing the number of iterations needed in the feasibility-seeking step.\n- **Number of unrolled steps in forward pass**: In our experiments, the L-BFGS solver terminates either when it reaches a maximum of 50 iterations or when the constraint violation function $\\phi$ falls below a threshold (typically $1e{-8}$ to $1e{-10}$). The number of unrolled iterations actually varies depending on the difficulty of each problem instance. In our experiments, we observe that the L-BFGS solver often meets the stopping threshold in fewer than 50 iterations. This threshold is a key parameter that directly controls the degree of constraint satisfaction. To investigate the sensitivity of FSNet to the number of unrolled iterations, we instead vary the stopping threshold and show the result in the table below. As expected, using a smaller threshold leads to tighter constraint satisfaction but requires more computation. Conversely, with a larger threshold, the solver may terminate early, and the neural network can produce solutions outside the feasible set that still have low objective values. This behavior can result in a negative optimality gap, as observed with a threshold of $1e{-4}$. \n\n*Table: Test results of FSNet on 2000 instances of convex QP problems with different stopping thresholds for the feasibility-seeking step.*\n|Threshold|Eq. Vio.|Ineq. Vio.|Opt. Gap (%)|Runtime (s)|\n|-|-|-|-|-|\n||Mean (Max)|Mean (Max)|Mean / Min (Max)|Batch|\n|1e-4|0.045 (0.103)|6.7e-3 (0.017)|7.9e-3 / -0.021 (0.058)|0.071|\n|1e-6|4.2e-3 (0.133)|1.8e-4 (8.4e-4)|0.022 / 7.7e-3 (0.056)|0.098|\n|1e-8|3.8e-4 (1.9e-3)|1.7e-5 (1.4e-4)|0.021 / 7.6e-3 (0.057)|0.126|\n|1e-10|3.4e-5 (1.9e-4)|7.2e-7 (9.1e-6)|0.018 / 6.9e-3 (0.054)|0.174|\n\n- **Number of unrolled steps in backward pass** (Appendix B.4): We also evaluated the performance of FSNet under different numbers of unrolled iterations in the feasibility-seeking steps included in the computational graph for backpropagation (please see Table B.4 in the Appendix). We find that excluding all iterations results in higher optimality gaps, indicating that the neural network fails to learn the optimal solutions. However, differentiating through 10–50 unrolled steps leads to consistent performance, with optimality gaps remaining below 1$\\%$.  This makes selecting the number of unrolled steps for the backward pass relatively easy in practice.\n\n> Can the authors provide quantitative analysis or further comments on FSNet’s performance when the smoothness or PL conditions are significantly violated—are there clear failure cases, or does the method degrade gracefully?\n\nIn our experiments, we have already evaluated FSNet in settings where the L-smoothness and the PL condition do not hold. For example, while the PL condition may be (locally) satisfied in some convex problems, it is generally violated in nonconvex problems, which can have saddle points with vanishing gradients and nonzero function values. Similarly, the L-smoothness condition is violated in both convex and nonconvex SOCP problems due to the presence of $\\ell_2$ norms in the constraints, which make the violation function $\\phi$ nondifferentiable and thus not L-smooth. In addition, we add the $\\ell_2$ norm regularization in the objective function in nonsmooth nonconvex cases (please see Table B.3), which also makes the objective functions not L-smooth. The ACOPF experiment provided earlier in this response is another example of a setting where L-smoothness and the PL condition are significantly violated. Despite these violations, FSNet performs robustly across all tested scenarios. \n\nWe would also like to emphasize that we do not use the L-smoothness and the PL condition as a starting point in designing FSNet. While the main contribution of our paper is the introduction of a strong empirical framework for deep learning with constraint enforcement in amortized optimization, the theoretical analysis is intended to provide *additional insight* into how and why the proposed framework works under certain conditions, and definitely does not cover all settings where FSNet can work well. As such, FSNet is not limited to the specific assumptions used in the theoretical analysis, and our experimental results demonstrate that it performs well even when these conditions are violated.\n>Are there foreseeable bottlenecks or challenges when scaling FSNet to thousands or tens of thousands of decision variables? Is the feasibility-seeking optimization step a limiting factor?\n\nPhilosophically, FSNet reduces the online solution complexity of the original constrained optimization problem to that of a (much easier) *un*constrained optimization problem – notably, finding a feasible point via the feasibility-seeking step.\nAs a result, the scalability of FSNet is directly related to the scalability of unconstrained optimization, with all toolkits and intuitions available for unconstrained optimization applying directly. Therefore, we believe that handling problems with thousands or even tens of thousands of decision variables is practically feasible, provided that appropriate scalable solvers are used for the feasibility-seeking step.\n\nFor this reason, we do not consider the feasibility-seeking step to be a limiting factor. In fact, as shown in our results in ACOPF (in the first comment), FSNet achieves greater speedup relative to traditional solvers as the number of variables and constraints increases. This is because unconstrained optimization is often faster, more scalable, and easier to parallelize than its constrained counterparts. Thus, FSNet can become even more advantageous in large-scale problem settings."}, {"review_id": "AlRet4Ecql", "reviewer": "Reviewer_q8xn", "summary": "In this paper, the Feasibility-Seeking-Integrated Neural Network (FSNet) is proposed to solve optimization problems with complex constraints. Core idea of this method is to introduce a differentiable feasibility search step based on neural network predictions, ensuring that the output solutions satisfy the constraints while enabling end-to-end training. This design not only provides theoretical guarantees of convergence (with exponential convergence under certain conditions), but also demonstrates excellent empirical performance across various types of problems (convex/non-convex, smooth/non-smooth).", "questions": "Please refer to the **Weaknesses** part.", "limitations": "Please refer to the **Weaknesses** part.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. Theoretical Rigor and Clarity: A detailed convergence analysis is provided, including exponential convergence for feasibility search and guarantees for end-to-end training, which enhances the credibility of the approach.\n\n2. Sufficient Empirical Evidence: Extensive experiments have been conducted on a variety of complex problems, comparing the performance with traditional solvers as well as other learning methods, and the results are convincing.\n\n3. Good Presentation: The paper is well-written, main body of the paper and the level of details in the appendix are appropriate.\n\n**Weaknesses**\n1. Assumption Dependency for Non-Convex Problems: Current convergence guarantees hold under certain assumptions (e.g., L-smoothness, PL condition). However, in practical applications of non-convex problems, these conditions might difficult to ensure. It is suggested to clearly define the scope of applicability of these conditions in the discussion (or limitation) and to look forward to future relaxation schemes.\n\n2. Algorithm Complexity and Practicality: Although the method performs excellently in several problems, the computational cost of the feasibility search step in high-dimensional, large-scale problems still needs attention. I suggest supplementing an analysis of its complexity and the discussion on its application potential in ultra-large-scale problems.\n\n3. Experimental Design and Comparison: The experiments primarily focused on smaller-scale problems (e.g., 100 decision variables). In the future, the scalability of the method could be validated in higher-dimensional, more complex real-world application scenarios.\n\nOverall, I consider this is an interesting paper. If the authors could address the concerns during the rebuttal process, I would be happy to increase my score.", "was_revised": true, "final_justification": "The authors have addressed my main concerns.", "rebuttal": "Thank you to the reviewer for their careful review and thoughtful comments.\n> Assumption Dependency for Non-Convex Problems: Current convergence guarantees hold under certain assumptions (e.g., L-smoothness, PL condition). However, in practical applications of non-convex problems, these conditions might be difficult to ensure. It is suggested to clearly define the scope of applicability of these conditions in the discussion (or limitation) and to look forward to future relaxation schemes.\n\nWe would like to clarify that the main contribution of this work is to **provide a strong empirical framework for deep learning with constraint enforcement in amortized optimization**. This framework is *backed by* theoretical analysis in order to provide further insight into performance and design choices, but importantly, *does not depend on* the assumptions used within the theoretical analysis. For instance, we do not use the L-smoothness and the PL condition as a starting point in designing FSNet. Moreover, our theoretical analysis definitely does not cover all settings where FSNet can work well. In other words, the theoretical analysis is an **additional contribution** compared to prior works providing only empirical methods, rather than being something upon which our framework is directly designed.\n\nIn our experiments, we have already evaluated FSNet in settings where the L-smoothness and the PL condition *do not* hold. For example, while the PL condition may be (locally) satisfied in some convex problems, it is generally violated in nonconvex problems, which can have saddle points with vanishing gradients and nonzero function values. Similarly, the L-smoothness condition is violated in both convex and nonconvex SOCP problems due to the presence of $\\ell_2$ norms in the constraints, which make the violation function $\\phi$ nondifferentiable and thus not L-smooth. In addition, we add the $\\ell_2$ norm regularization in the objective function in nonsmooth nonconvex cases (please see Table B.3), which also makes the objective functions not L-smooth. Despite these violations, FSNet performs robustly across all tested scenarios. \n\nFuture work includes further strengthening the theoretical analysis to capture more general settings, to better reflect the strong empirical performance across more general settings that we observe in practice. We will plan to add discussion on these points to the paper.\n> Algorithm Complexity and Practicality: Although the method performs excellently in several problems, the computational cost of the feasibility search step in high-dimensional, large-scale problems still needs attention. I suggest supplementing an analysis of its complexity and the discussion on its application potential in ultra-large-scale problems.\n\nWe agree that scalability is an important consideration for FSNet, especially in the context of high-dimensional and large-scale problems. To further evaluate this aspect, we have now conducted additional experiments on larger problems; detailed results are provided in our response to the third comment. Notably, in experiments on the AC optimal power flow (ACOPF) problem in power grids, we observe that FSNet's speedup gains actually *increase* with problem size – e.g., a **3**$\\times$ speedup on the IEEE 30-bus system (60 variables, 60 equality constraints, 248 inequality constraints) and an **11**$\\times$ speedup on the IEEE 118-bus system (236 variables, 236 equality constraints, 1196 inequality constraints). A similar pattern is observed when comparing FSNet's performance on 500-variable convex problems to the 100-variable cases. This trend highlights FSNet’s scalability and is attributable to the fact that FSNet solves an unconstrained optimization problem initialized by the neural network's output, which is generally more efficient and scalable than solving the original constrained optimization problem.\n\nRegarding computational complexity, the primary computational burden of FSNet lies in the feasibility-seeking step. However, since this step involves unconstrained optimization, standard complexity results and scalability techniques from classical unconstrained optimization can be directly applied. For ultra-large-scale problems, it will be important to incorporate techniques commonly used at this scale into FSNet, such as sparse matrix operations, scalable numerical linear algebra, and exploitation of problem structure. Based on our observation that FSNet achieves greater speedup gains in higher-dimensional and larger-scale problems, we expect it to dramatically outperform traditional solvers in ultra-large-scale settings.\n\nIn summary, we believe that FSNet can be readily used for large-scale applications given an appropriate solver design and implementation for the feasibility-seeking step. Thank you for pointing out the need for discussion on this point; we will plan to add this to the paper.\n> Experimental Design and Comparison: The experiments primarily focused on smaller-scale problems (e.g., 100 decision variables). In the future, the scalability of the method could be validated in higher-dimensional, more complex real-world application scenarios.\n\nWe have now run experiments on FSNet for the AC optimal power flow (ACOPF) problem in power systems, with results under a random seed presented in Table 1 below. FSNet satisfies equality and inequality constraints (numerical violations on a similar order to that of the IPOPT solver). Moreover, the optimality gap remains below 1$\\%$ in both test cases. In terms of runtime in the sequential setting, FSNet significantly outperforms IPOPT, achieving **3**$\\times$ speedups on the IEEE 30-bus system and **11**$\\times$ on the IEEE 118-bus system. The penalty method, on the other hand, despite its fast inference, suffers from high constraint violations, particularly in terms of maximum violations on the IEEE 118-bus case. Overall, this result is consistent with our previous findings on convex and nonconvex problems in the original manuscript.\n\n*Table 1: Test results of FSNet on 2000 instances of ACOPF problems.*\n| Method|Eq. Vio.|Ineq. Vio.|Opt. Gap (%)|Runtime (s)|\n|-|-|-|-|-|\n||Mean (Max)|Mean (Max)|Mean / Min (Max)|Sequential|\n|||**IEEE 30-bus**:| $n=60, n_\\text{eq}=60, n_\\text{ineq}=248$||\n| IPOPT Solver| 6.7e-7 (5.8e-6)| 5.4e-9 (1.1e-6)|--/--|51.53|\n| **FSNet** |1.1e-5 (5.1e-5)|1.3e-8 (2.9e-6)| 0.86 / 0.05 (3.63)|16.80|\n| Penalty|1.5e-3 (6.2e-3)|7.1e-6 (1.7e-3)|-5.91 / -9.90 (-1.69)|0.74|\n|||**IEEE 118-bus**:| $n=236, n_\\text{eq}=236, n_\\text{ineq}=1196$||\n|IPOPT Solver|1.2e-6 (2.8e-5)|3.1e-8 (1.1e-5)|--/--|392.83|\n|**FSNet**|2.1e-5 (4.9e-4)|9.9e-7 (4.1e-4)|0.78 / -2.66 (3.78)| 34.01|\n|Penalty|4.4e-3 (0.144)|2.9e-5 (0.259)|0.57 / -3.41 (1.61)|0.85|\n\nFor larger-scale problems, we have now evaluated FSNet on convex problems with 500 variables, 200 equality constraints, and 200 inequality constraints. This scale matches or exceeds those considered in recent studies (e.g., [1-4]). The results are presented in Table 2 below. In the batch setting, FSNet achieved speedups of **188**$\\times$ (QP) and **147**$\\times$ (QCQP), which are notably higher than the speedups obtained in smaller problems with 100 variables—namely, 31$\\times$ and 104$\\times$, respectively (see Table B.1). In the sequential setting, FSNet achieved speedups of **8**$\\times$ (QP) and **143**$\\times$ (QCQP), compared to 0.8$\\times$ and 6$\\times$ in the corresponding 100-variable cases. For SOCP, running the 500-variable experiment requires significantly more time. We have already run it for over a day, but were unable to complete it before the rebuttal deadline. We estimate that solving 2,000 instances in the sequential setting will take approximately 3.5 days, and we will update the results as soon as they become available. Based on our estimation, we believe that FSNet will exhibit similarly increasing speedup gains in the SOCP case, as observed in the QP and QCQP cases. This trend is consistent with our findings on ACOPF problems and supports the observation that FSNet becomes increasingly advantageous in larger-scale settings.\n\n*Table 2: Test results of FSNet on 2000 instances of smooth convex problems with 500 variables.*\n| Method|Eq. Vio.|Ineq. Vio.|Opt. Gap (%)|Runtime (s)|\n|-|-|-|-|-|\n||Mean (Max)|Mean (Max)|Mean / Min (Max)|Batch (Sequential)|\n||||**Convex QP**: $n=500, n_\\text{eq}=200, n_\\text{ineq}=200$||\n|Solver|1.9e-9 ± 2.9e-10  (1.2e-6 ± 9.8e-8)|1.4e-6 ± 9.8e-8 (9.3e-4 ± 1.3e-4)|--/--|112.962 ± 0.378 (1394.024 ± 94.868)|\n|**FSNet**|1.0e-4 ± 2.5e-6 (4.4e-3 ± 8.2e-4)|4.5e-6 ± 1.5e-7 (4.2e-4 ± 9.0e-5)|0.066 ± 2.7e-4 / 0.041 ± 6.6e-4 (0.119 ± 4.9e-3)|0.599 ± 1.8e-3  (163.558 ± 1.027)|\n||||**Convex QCQP**: $n=500, n_\\text{eq}=200, n_\\text{ineq}=200$||\n|Solver|1.3e-5 ± 4.3e-7 (3.1e-4 ± 7.7e-5)|5.0e-2 ± 9.9e-4 (2.1e-1 ± 7.3e-3)|--/--|5508.419 ± 44.220 (33362.147 ± 107.952)|\n|**FSNet**|2.7e-4 ± 1.4e-6 (3.7e-4 ± 3.3e-7)|9.9e-7 ± 4.5e-8 (1.0e-5 ± 4.9e-7)|0.143 ± 1.3e-3 / 0.087 ± 2.5e-3 (0.328 ± 0.034)|37.291 ± 0.237 (232.837 ± 1.229)|\n||||**Convex SOCP**: $n=500, n_\\text{eq}=200, n_\\text{ineq}=200$||\n|**FSNet**|2.7e-4 ± 1.4e-6 (3.7e-4 ± 2.1e-6)|0.0 ± 0.0 (0.0 ± 0.0)|0.730 ± 0.027 / 0.469 ± 0.246 (1.124 ± 0.048)|14.313 ± 0.073 (177.890 ± 1.098)|\n\n[1] Priya L. Donti, David Rolnick, and J Zico Kolter. DC3: A learning method for optimization with hard constraints. In International Conference on Learning Representations, 2021.\n\n[2] Enming Liang, Minghua Chen, and Steven H Low. Homeomorphic projection to ensure neural-network solution feasibility for constrained optimization. Journal of Machine Learning Research, 2024.\n\n[3] Park, Seonho, and Pascal Van Hentenryck. Self-supervised primal-dual learning for constrained optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 2023.\n\n[4] Youngjae Min, Anoopkumar Sonar, and Navid Azizan. Hard-constrained neural networks with universal approximation guarantees. arXiv preprint arXiv:2410.10807, 2024."}, {"review_id": "VKrY11l5c1", "reviewer": "Reviewer_wKYo", "summary": "The paper introduces FSNet, a novel framework that integrates feasibility-seeking procedures into neural network pipelines to solve constrained optimization problems efficiently and with theoretical guarantees. Unlike traditional neural solvers or NN-based surrogate models that often produce infeasible solutions, FSNet ensures constraint satisfaction by solving a differentiable, unconstrained optimization subproblem that minimizes constraint violations. This feasibility-seeking step is integrated into both the training and inference stages. The authors provide theoretical convergence guarantees under smoothness and PL conditions and demonstrate that FSNet achieves near-zero constraint violations with competitive or superior objective values, while being significantly faster than traditional solvers across convex, nonconvex, and nonsmooth problem settings.", "questions": "1. The introduction lacks clarity on the specific problem the paper addresses. It would help to present equations (1) and (2) earlier. The high-level idea in lines 29–36 is difficult to follow without knowing the underlying problem and assumptions.\n\n2. The feasibility and convergence guarantees are derived under the PL condition (line 43 and Assumption 1). However, how can the loss landscape induced by a neural network satisfy the PL condition? This assumption seems strong, particularly since PL does not generally hold even for some strongly convex functions.\n\n3. In line 111, it is stated that x represents parameters. It is unclear whether the optimization problems mentioned earlier (e.g., in energy systems, robotics) conform to the structure in equation (1). Clarifying this connection would be helpful.\n\n4. The assumption that x \\~ D is drawn from a finite distribution is strong. What happens if D is continuous or infinite? How would this affect the theoretical results and practical performance?\n\n5. Theorem 3 mentions convergence of a sequence to limit points. However, does the convergence of the underlying optimization procedure itself hold? Under what conditions?\n\n6. How does the scalability of FSNet depend on the number of variables, constraints, and the size of distribution D (x~D)? It would be valuable to analyze this both theoretically and through experiments on larger-scale problems.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper proposes FSNet, a neural network framework that integrates a feasibility-seeking step for solving constrained optimization problems. The key innovation lies in formulating an unconstrained, differentiable subproblem that enforces feasibility, enabling end-to-end training while guaranteeing convergence under certain assumptions. The method performs well empirically across a variety of convex, nonconvex, and nonsmooth problems, achieving near-zero constraint violation and competitive or superior objective values, often with large runtime improvements over traditional solvers.\n\nHowever, the presentation of the problem setup is unclear early in the paper. Equations (1) and (2), which are central to the formulation, are introduced only later, making it difficult to understand the context of the framework during the initial discussion. Furthermore, the theoretical analysis relies on strong assumptions such as the PL condition and boundedness of derivatives. These conditions are difficult to justify in the context of neural networks and general nonconvex optimization, especially given that even strong convexity does not necessarily imply PL. The connection between the general applications discussed in the introduction and the formulation in (1) also lacks clarity. Additionally, the assumption that the input distribution is finite (x ~ D) may be restrictive in practice, and it is unclear how the method would perform or be affected in infinite or continuous distributions. Finally, the scalability of FSNet with respect to problem size and dataset cardinality is not analyzed theoretically or validated through large-scale experiments.", "was_revised": true, "final_justification": "The rebuttal resolved my questions. However, I do not see the applicability of these questions. It seems tedious to use a neural network to approximate solving optimization questions. I raise my score to 4 as I acknowledge that this paper is well-written in a way that is theoretically rigorous and experimentally sound. I lower my confidence to 3 as I have some concern on the motivation.", "rebuttal": "Thank you to the reviewer for their comments and ideas for ways to further clarify our paper's content.\n>(W) The presentation of the problem setup is unclear early in the paper. Equations (1) and (2), which are central to the formulation, are introduced only later, making it difficult to understand the context of the framework during the initial discussion.  \n(Q) The introduction lacks clarity on the specific problem the paper addresses. It would help to present equations (1) and (2) earlier. The high-level idea in lines 29–36 is difficult to follow without knowing the underlying problem and assumptions.\n\nWe will plan to move some of these descriptions earlier in the paper to provide additional clarification and grounding to the discussion.\n\n> (W) The theoretical analysis relies on strong assumptions such as the PL condition and boundedness of derivatives. These conditions are difficult to justify in the context of neural networks and general nonconvex optimization, especially given that even strong convexity does not necessarily imply PL.   \n(Q) The feasibility and convergence guarantees are derived under the PL condition (line 43 and Assumption 1). However, how can the loss landscape induced by a neural network satisfy the PL condition? This assumption seems strong, particularly since PL does not generally hold even for some strongly convex functions.\n\nWe would like to clarify that the main contribution of this work is to provide **a strong empirical framework for deep learning with constraint enforcement in amortized optimization**. This framework is *backed by* theoretical analysis in order to provide further insight into performance and design choices, but importantly, *does not depend on* the assumptions used within the theoretical analysis. For example, we do not use the L-smoothness and the PL condition as a starting point in designing FSNet. Moreover, our theoretical analysis definitely does not cover all settings where FSNet can work well.  For instance, our experiments evaluate FSNet in settings where the L-smoothness and the PL condition do not hold – notably, the PL condition is generally violated in nonconvex problems, and the L-smoothness condition is violated in both convex and nonconvex SOCP problems and all nonsmooth nonconvex problems– and FSNet performs robustly across all tested scenarios.\n\nIn other words, the theoretical analysis is an **additional contribution** compared to prior works providing only empirical methods, rather than being something that our framework is directly dependent on.\n\nAs an additional clarification, our theoretical analysis does not assume the PL condition for the loss function or loss landscape of the neural network, but rather for the violation function defined as $ \\phi (s;x) :=||h(s;x)||_2^2 + || g^+(s;x) ||_2^2 $, which quantifies the constraint violation. Additionally, if a function is strongly convex, then it satisfies the PL inequality, but not vice versa. Thus, the PL condition is a weaker assumption than the strong convexity. For further reference, please refer to [1].\n\n[1] Karimi, Hamed, Julie Nutini, and Mark Schmidt. \"Linear convergence of gradient and proximal-gradient methods under the Polyak-Łojasiewicz condition.\" Joint European conference on machine learning and knowledge discovery in databases. Cham: Springer International Publishing, 2016.\n\n> (W) The connection between the general applications discussed in the introduction and the formulation in (1) also lacks clarity.   \n(Q) In line 111, it is stated that $x$ represents parameters. It is unclear whether the optimization problems mentioned earlier (e.g., in energy systems, robotics) conform to the structure in equation (1). Clarifying this connection would be helpful.\n\nThank you for highlighting the need to clarify this connection. Below are some illustrative examples, and we will incorporate this discussion into the revised manuscript.\n\nIn AC optimal power flow (ACOPF) for power systems, $x$ represents the load demands, and $y$ denotes the optimal power generation at the generators. In building temperature control, $x$ corresponds to the current temperature state of the building, and $y$ represents the control signal for actuators such as heaters and air conditioners. In robot motion planning, $x$ includes the initial and terminal states as well as the sequence of feasible corridors, while $y$ corresponds to the optimal traversal time.\n\nBeyond these, the formulation in equation (1) also captures a wide range of problems in meta-learning, reinforcement learning, optimization, and control. For additional examples, we refer the reader to the tutorial on amortized optimization [1].\n\n[1] Amos, Brandon. Tutorial on amortized optimization. Foundations and Trends® in Machine Learning (2023).\n\n> (W) The assumption that the input distribution is finite ($x \\sim \\mathcal{D}$) may be restrictive in practice, and it is unclear how the method would perform or be affected in infinite or continuous distributions.  \n(Q) The assumption that $x \\sim \\mathcal{D}$ is drawn from a finite distribution is strong. What happens if D is continuous or infinite? How would this affect the theoretical results and practical performance?\n\nWe *do not* assume the distribution $\\mathcal{D}$ is finite, nor do we make any assumptions on whether it is continuous or discrete.\nInstead, as standard in machine learning, we assume that we have access to a finite dataset drawn from $\\mathcal{D}$, and that the exact form of $\\mathcal{D}$ is unknown to the learner. This is a standard assumption in machine learning [1]. If the reviewer can point to any places in the manuscript from which this misunderstanding may have arisen, we will plan to make revisions to the manuscript accordingly to aim to mitigate any confusion.\n\n[1] Shalev-Shwartz, Shai, and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.\n\n> (W) The scalability of FSNet with respect to problem size and dataset cardinality is not analyzed theoretically or validated through large-scale experiments.  \n(Q) How does the scalability of FSNet depend on the number of variables, constraints, and the size of distribution $\\mathcal{D}$ ($x \\sim \\mathcal{D}$)? It would be valuable to analyze this both theoretically and through experiments on larger-scale problems.\n\nPhilosophically, FSNet reduces the online solution complexity of the original constrained optimization problem to that of a (much easier) *un*constrained optimization problem – notably, finding a feasible point via the feasibility-seeking step. As a result, the scalability of FSNet is directly related to the scalability of unconstrained optimization, with **all complexity and convergence analyses existing for unconstrained optimization applying directly**. Therefore, we believe that handling problems with thousands or even tens of thousands of decision variables is practically feasible, provided that appropriate scalable solvers are used for the feasibility-seeking step. In fact, FSNet achieves greater speedup relative to traditional solvers as the number of variables and constraints increases, as shown in our results in AC optimal power flow and larger-scale convex problems. For the detailed results, please refer to our response to the last comment of `Reviewer q8xn`.\n\nRegarding the impact of dataset size on the scalability of FSNet, we note that this primarily affects training time, not inference time. As the dataset grows, the time per epoch increases proportionally, leading to a longer total training duration. However, this is generally not a concern, as training is typically performed offline and only once, while the trained model can be used many times during inference. In particular, the overall motivation for the area of amortized optimization is that we can replace some of the expense of online inference with offline costs incurred during training, with the idea that online inference time (rather than offline training time) is the primary bottleneck.\n\n> (Q) Theorem 3 mentions the convergence of a sequence to limit points. However, does the convergence of the underlying optimization procedure itself hold? Under what conditions?\n\nWe would like to clarify that Theorem 3 does not mean the convergence of the entire sequence $\\{(y_\\tau, \\hat y_\\tau)\\}$. Rather, it shows that *if* the sequence admits a limit point $(y^\\star, \\hat{y}^\\star)$, then this limit point is a global minimizer of the original optimization problem (1) under the assumptions stated in the theorem. The only condition required for the existence of a limit point is that the sequence ${(y_\\tau, \\hat y_\\tau) }$ is bounded. In this case, the Bolzano-Weierstrass theorem guarantees the existence of a convergent subsequence, which defines the limit point. Our result is consistent with similar guarantees in classical penalty methods (see [1], p. 471).\n\nTo prove convergence of the entire sequence, we would require stronger assumptions, such as strong/strict convexity, which do not hold in our general nonconvex setting. For this reason, we do not extend the theorem to that case.\n\n[1] Bertsekas, Dimitri. Nonlinear Programming. Vol. 4. Athena Scientific, 2016."}]

Author rebuttal:

Meta-review: The paper proposes a NN based approach to solving constrained optimization problem. The approach is to have an NN network predict a candidate optimum, and then via sequential GD steps moved towards the constraint set and then an overall loss is applied balancing the optimality of the solution and the closeness of the original prediction and the feasibility seeking solution. 

The paper is very well written and presents the ideas in a clean and concise way. The reviewer discussion was highly productive with reviewers appreciating the more practice oriented results and clarifications to the questions raised. I recommend acceptance.
Decision: Accept (poster)