Ensuring DNN Solution Feasibility for Optimization Problems with Linear Constraints
Authors: Tianyu Zhao
Xiang Pan
Minghua Chen
Steven Low
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Deep learning
Deep neural network
Constrained optimization
Solution feasibility guarantee
Optimal power flow

Abstract: We propose preventive learning as the first framework to guarantee Deep Neural Network (DNN) solution feasibility for optimization problems with linear constraints without post-processing, upon satisfying a mild condition on constraint calibration. Without loss of generality, we focus on problems with only inequality constraints. We systematically calibrate the inequality constraints used in training, thereby anticipating DNN prediction errors and ensuring the obtained solutions remain feasible. We characterize the calibration rate and a critical DNN size, based on which we can directly construct a DNN with provable solution feasibility guarantee. We further propose an Adversarial-Sample Aware training algorithm to improve its optimality performance. We apply the framework to develop DeepOPF+ for solving essential DC optimal power flow problems in grid operation. Simulation results over IEEE test cases show that it outperforms existing strong DNN baselines in ensuring 100\% feasibility and attaining consistent optimality loss (<0.19%) and speedup (up to x228) in both light-load and heavy-load regimes, as compared to a state-of-the-art solver. We also apply our framework to a non-convex problem and show its performance advantage over existing schemes.

constraint Published as a conference paper at ICLR 2023
ENSURING DNN SOLUTION FEASIBILITY FOR OPTI- MIZATION PROBLEMS WITH LINEAR CONSTRAINTS
Tianyu Zhao!?, Xiang Pan", Minghua , Steven H. Low4 'enovv Machine Intelligence Center, "The Chinese University of Hong Kong,
*City University of Hong Kong, *California Institute of Technology
t eetaooe@eenvo.... com, Xpaně 1 linkiin..im cuhk. edu. edu.hk, minghua. chen coan@citin www.cittuueee. edu. hk ("Corresponding author), s lwwwwaalllee www.alleech.eeeeeeoo.oooeeeeeeee
ABSTRACT
We propose preventive learning the first framework to guarantee Deep Neural Network (DNN) solution feasibility for optimization problems with linear con- straints without post-processing, upon satisfying mild condition calibration. Without loss of generality, we focus on problems with only inequality constraints. We systematically calibrate the inequality constraints used in train- ing, thereby anticipating DNN prediction errors and ensuring the obtained solu- tions remain feasible. We characterize the calibration rate and a critical DNN size, based on which we can directly construct a DNN with provable solution feasibil- ity guarantee. We further propose an Adversarial- Sample Aware training algo- rithm to improve its optimality performance. We apply the framework to develop DeepOPF+ for solving essential DC optimal power flow problems in grid oper- ation. Simulation results over IEEE test cases show that it outperforms existing strong DNN baselines in ensuring 100% feasibility and attaining consistent opti- mality loss (<0.19%) and speedup (up to x228) in both light-load and heavy-load regimes, as compared to a state-of-the of-the-----e-tttttt solver. We also apply our framework to non-convex problem and show its performance advantage over existing schemes.
INTRODUCTION
Recently, there have been increasing interests in employing neural networks, including deep neural networks (DNN), to solve constrained optimization problems in various problem domains, espe- cially those needed to be solved repeatedly in real-time. The idea behind these machine learning approaches is to leverage the universal approximation capability of DNNs (Hornik et al.) 1989 Leshno et al. 1993) to learn the mapping between the input parameters to the solution of op- timization problem. Then one can directly pass the input parameters through the trained DNN to obtain a quality solution much faster than iterative solvers. For example, researchers have developed DNN schemes to solve essential optimal power flow problems in grid operation with sub- sub-percentage optimality loss and several orders of magnitude speedup as compared to conventional solvers (Pan) jet al., 2020a b) Donti et al. 202T; Chatzos et al., 2020 Lei et al. 2020). Similarly, DNN schemes also obtain desirable results for real-time power control and heam-forming design (Sun et al., 2018 Xia et al. 2019) problems in wireless communication in a fraction of time used by existing solvers.
Despite these promising results, however, major criticism of DNN and machine learning schemes is that they usually cannot guarantee the solution fcasibility with respect to all the inequality and cquality constraints of the optimization problem (Zhao al. 2020 Pan et al. (0200b). This is due to the inherent neural network prediction errors. Existing works address the feasibility concern mainly by incorporating the constraints violation (e.g., a Lagrangian relaxation
compute constraint vi- olation with Lagrangian multipliers) into the loss function to guide THE DNN training. These endeavors, while generating great insights the DNN design and working some extent in case studies, can not guarantee the solution feasibility without resorting to expensive post-processing procedures, e.g., feeding the DNN solution as warm start point into an iterative solver to obtain a feasible solution. See Sec. 2 for more discussions. To date, it remains a largely open issue of ensuring DNN solution (output of DNN) feasibility for constrained optimization problems.
In this paper, we address this challenge for general Optimization Problems with Linear (inequality) Constraints (OPLC) with varying problem inputs and fixed objective/constrains parameters. Since linear equality constraints can be exploited to reduce the number of decision variables without losing optimality (and removed), it suffices to focus on problems with inequality constraints. Our idea is to train DNN in a preventive manner to ensure the resulting solutions remain feasible even with prediction errors, thus avoiding the need of post-processing. We make the following contributions:

---

Published as a conference paper at ICLR 2023
After formulating OPLC in Sec. we propose preventive learning the first framework to ensure the DNN solution feasibility for OPLC without post-processing in Sec. 4 We systematically calibrate inequality constraints used in DNN training, thereby anticipating prediction emrors and ensuring the resulting DNN solutions (outputs of the DNN) remain feasible.
>> We characterize the calibration rate allowed in Sec. 4.1 i.e., the rate of adjusting (reducing) constraints limits that represents the room for (prediction) errors without violating constraints, and a sufficient DNN size for ensuring DNN solution feasibility in Sec. 4.2 We then directly construct
a DNN with provably guaranteed solution feasibility.
Observing the feasibllty-guuraateed DNN may not achieve strong optimality result, in Sec. we propose an adversarial training algorithm, called Adversarial-Sannn -ample Aware algorithm to further improve its optimality without sacrificing feasibility guarantee and derive its performance guarantee.
>> We apply the framework to design a DNN scheme, DeepOPF+, to solve DC optimal power flow (DC-OPF) problems in grid operation. Simulation results over IEEE 30/1 18/300-bbs test cases show that it outperforms existing strong DNN baselines in ensuring 100% feasibility and attaining consistent optimality loss (<0.15 19) and speedup (up to x228) in both light-load and heavy-load regimes, as compared to offe-of-the-an solver. We also apply our framework to non-convex problem and show its performance advantage over existing schemes.
2 RELATED WORK
There have been active studies in employing machine learning models, including DNNs, solve constrained optimizations directly (Kotary et al., 1b; Pan et al. 2019; 2020bb Zhou et al. 2022
Guha al., 2019; Zamzam & Baker 2020 Fioretto et al. [2200] Dobbe et al. 2019 Sanseverino
et al., 2016: Elmachtoub & Grigas] 2022 Huang et al. 2021 Huang & Chen 2021), obtaining
close-to-optimal solution much faster than conventional iterative solvers. However, these schemes usually cannot guarantee solution feasibility w.r.t. constraints due to inherent prediction errors.
Some existing works tackle the feasibility concern by incorporating the constraints violation in DNN
training (Pan et al. 2020a Donti et al. (02T). In (Nellikkath & Chatzivasileiadis) 2021 2022), )
con- physics-informed neural networks are applied to predict solutions while incorporating the KKT con-
ditions of optimizations during training. These approaches, while attaining insightful performance in case studies, do not provide solution feasibility guarantee and may resort to expensive projec- tion procedure (Pan et al. 2020b) or post-pr post-processing equivalent projection layers (Amos & Kolter 2017 AAgrawal al.) 2019) to recover feasibility. A gradient-bar based violation correction is proposed in (Donti et al. [02 ). Though a feasible solution can be recovered for linear constraints, it can be computationally inefficient and may not converge for general optimizations. A DNN scheme apply- ing gauge function that maps a point in an nn-mrrm unit ball to the (sub)-optimal solution is proposed in (Li et al. 2022). However, its feasibility enforcement achieved from computationally expen- sive potnt finder program. There is also line of work (Ferrari 2009 uul Abdeen et al., 2022] Qin et al. 2019: Limanond & Si 1998) focusing on verifying whether the output of a given DNN satisfies set of requirements/constrainss However, these approaches are only used for evaluation
and not capable of obtaining a DNN with caasibllitygguarnnte and strong optimality, To our best knowledge, this work is the first to guarantee DNN solution feasibility without post-processing.
Some techniques used in our study (for constrained problems) are related to those for verifying DNN accuracy against input perturbations for unconstrained classification (Sheikholeslami et al.) 2020). Our work also significantly differs from (Zhao et al. 2020) in we can provably guarantee DNN solution feasibility for OPLC and develop a new learning algorithm to improve solution optimality.
3 OPTIMIZATION PROBLEMS WITH LINEAR CONSTRAINTS (OPLC)
We focus on the standard OPLC formulated as (Faisca al. 2007):
min fxx 0) s.t. gz 0) 2 ay x + b, < ej JEE, var. Ig S mb I 3s, k == I. N.
(1)
(2)
a E R are the decision variables, E is the set of inequality constraints, and 0 E D are the OPLC (2)
inputs. Convex polytope D = (0 E R Apt < by. - : 2 hold} is specified by matrix A and vector by such that VO E D. OPLC in 2 admits a unique optimum l The OPLC objective if is general convex/non-conver function. For ease of presentation,
use gj (x, 0) to denote
'Our approach is also applicable to non-unique solution and unbounded a.. See Appendix A for discussion.
2

---

Published as a conference paper at ICLR 2023
Traling
Oinereee et. Candra 1a-1 Constroine
Inequality Condraint Caliration Rala
Suffiient DNN Siee for Erouring FassBilly
Adverserial- Sample Aware Algocitim
Dipelal
DNVOE: Destrable Optimality without Sacfficing Fazsibility
Oudy larber hobbery (DRLC)
DNNU-G direct ocnatruction Provatic
Syitton
Feasibility
Inot
WPt Calbbrtted
Conetrrints
Pertmaan
Figure 1: Overview of the preventive learning framework to solve OPLC. The calibration rate is first obtained. The sufficient DNN size in ensuring universal feasibility is then determined, and a DNN model can constructed directly with universal feasibility guarantee in this step. With the determined calibration rate and sufficient DNN size, DNN model with enhanced optimality without sacrificing feasibility is obtained using the Adversarial Sabrersarall-ammll Aware algorithm.
a- + by o We assume that each 9j (xx 0) S e, is active for at least one combination of 0 E D without loss of generality (otherwise 9j is unnecessary and can be removed). We note that linear equality constraints can be exploited (and removed) to reduce the number of decision variables without losing optimality as discussed in Appendix suffices to focus on OPLC with inequality constraints as formulated in (
and x satisfying 2
The OPLC in - has wide applications in various engineering domains, e.g., DC-OPF problems in power systems (aa et al., 2019) and model- prediitpre control problems in control systems (Be- imporad et al. 2000). While many numerical solvers, c.g., those based on interior-poii methods (Ye & Tse 1989). can be applied to obtain its solution, the time complexity can be significant and lim- its their practical applications especially considering the problem input uncertainty under various scenarios. The observation that opens the door for DNN scheme development lies in that solving OPLC is equivalent to learning the mapping from input to optimal solution x (0) (Pan et al., 2020a Bemporad & Filippi 2006). Thus, one can leverage the universal approximation capability DNNs (Hornik et al. 1989; Leshno et al. 1993) to learn the input- mapping and apply the trained DNN to obtain the optimal solution for any O E D with significantly lower time complexity. See a concrete example in (Pan al. 2020b). While DNN schemes achieve promising speedup and optimality performance, a fundamental challenge lies in ensuring solution feasibility, which is nontrivial due to inherent DNN prediction errors. In the following, we propose preventive learning as the first framework to tackle this issue and design DNN schemes for solving OPLC in (2)
4 PREVENTIVE LEARNING FOR SOLVING OPLC
We propose a preventive learning framework to develop DNN schemes for solving OPLC in We calibrate inequality constraints used in DNN training, thereby anticipating prediction errors and ensuring the resulting DNN solutions remain feasible. See Fig. 2 for illustrations.
we determine the maximum calibration rate for inequality constraints, so that solutions from a preventively-traing DNN using the calibrated constraints respect the original con- straints for all possible inputs. Here we refer the output of the DNN as the DNN solution.
First, in Sec. 4.1
determine sufficient DNN size that with preventive learning there exists a DNN whose worst-case violation on calibrated constraints is smaller than the maximum calibration rate, thus ensuring DNN solution feasibility, i.e., DNN's output always satisfies [ for any input. We construct a provable Ceasibility-guarantee DNN model, namely DNN-FG, as shown in Fig.
Second, in Sec. 4. 2
Third, observing DNN-FG may not achieve strong optimality performance, in Sec. 4.3 we propose
an adversarial Adversarial-Sann Sample Aware training algorithm to further improve DNN optimality
without sacrificing feasibility guarantee, obtaining an -enhaneety-ehancedd DNN as shown in Fig.
4.1 INEQUALITY CONSTRAINT CALIRRATION RATE
We calibrate each inequality limit gj (x, 0) < Ej.j by a calibration rate n 2 0
(z, 0) < é = [e (1 my), ef 2 0;
(3)
e, (1 m)), otherwise.
Multiple scalars by 0 E correlated via 0. Studying varying aj, by: e, is also promising future work. For with e₁- = 0. one can add an auxiliary constant ē; & 0 such that gz 0) + + S ē, for the design and formulation consistency. The choice of é, can be problem dependent. For example, in our simulation, éG is set the maximum slack bus generation for its lower hound limit in OPF discussed in Appendix [.
3

---

Published as a conference paper at ICLR 2023
However, inappropriate calibration rate could lead to poor performance DNN. If one adjusts the limits too much, some imput E 'D will become infeasible under the cali- brated constraints and hence lead to poor gen- eralization of the preventively-trained DNN. To this end, we solve the following bi-level optimization to obtain the maximum calibra- tion rate, such that the calibrated feasibility set of a can still support the input region, i.e., the
Infossible
region
Default ragion
Callorated foazible region
OPLC ground
Calibrated ground truth
DNN preri
:: ONN prediction error
OPLC in O with a reduced feasible set
has solution for any 0 E 'D2
Figure 2: Left: solution of DNN trained with de- fault OPLC ground-truth can be infeasible due to inevitable prediction errors. Right: solution of DNN trained with calibrated OPLC ground-truth ensures universal feasibility even with prediction errors.
min max D
(4)
BED 2,
s.t.
(eg - (0,0))))))/l EEE (5)
m 2 enforce the feasibility of x for input 0 E D. (5) represents the maximum element-wise least redundancy among all constraints, i.c., the maximum constraint calibration rate. Therefore, solving 4 gives the maximum allowed calibration rate for all inequality constraints and E D.
challenging to solve the above bi-level problem exactly. We apply the following procedure to obtain a lower bound of its optimal objective in polynomial time. See Appendix [C details.
Reformulate the bi-level problem (4 (4-(5) to an equivalent single-level one by replacing the inner problem with its KKT conditions (Boyd Vandenherghe| 2004).
Step 1.
Step 2. Transform the single-level optimization problem into MILP by replacing the bi- linear equality constraints (comes from the complementary slackness KKT conditions) with equivalent mrixed-integer linear inequality constraints.
Step 3.
Solve the MILP using the branch-and-bound algorithm (Lawler & Wood 1966). Let the obtained objective value be 2 0 from the primal constraint and constraint (5).
Remark: (i) the franch-and-boute algorithm returns A (lower bound of the maximum calibration rate u**) with a polynomial time complexity of O(MM + 4|8| + 5.N)2.55 (Vaidya 1989). where M and N are the dimensions of the input and decision variables, and |3] is the number of constraints. (ii) A is a lower bound to the maximum calibration rate as the algorithm may not solve the MILP exactly (by relaxing (some of) the integer variables). Such a lower bound still guarantees that the input region is supported. (iii) If A O, reducing the feasibility set may lead to no feasible solution for some inputs. (iv) If A O,
obtain DNN with provably solution feasibility guarantee as shown in Sec. 4.2 (v) After solving (5)) we set each ny in (3) to be A, such uniform constraints calibration forms the outer bound of the minimum supporting calibration region. See Appendix [ for a discussion; (vi) we observe that the branch-and-bourd algorithm can actually return the exact optimal S* less than 20 mins for numerical examples studied in Sec. 6
4.2 SUFFICIENT DNN SIZE FOR ENSURING FEASIBILITY
In this subsection, first model DNN with ReLU activations. Then, introduce method to determine the sufficient DNN size for guaranteeing solution feasibility.
4.2.1
DNN MODEL. REPRESENTATION.
We employ DNN with Nuia hidden layers (depth) and Nnen neurons in each hidden layer (width), using lulti-layer cord-forward neural network structure with ReLU activation function to approxi- mate the input-solution mapping for OPLC, which is defined as:
ho = 0, h; (W:hi-1 + by), ==11..... 1,..., Niid:
h = a (W + bo 2 +
-o
+.
0 is the DNN input and h; is the output of the i-th layer. W; and b; are the i-th layer's weight matrix and bias. (xx) = max(x; ( 0) the ReLU activation, taking element-wise max operation over the input vector. enforces output feasibility w.r.t. the lower bounds in (2) while final output & further satisfies upper bounds. Here present the last two clamp-equivalent actions as (6) for further DNN
4

---

Published as a conference paper at ICLR 2023
analysis. To better include the DNN equations in our designed optimization analysis DNN's worst case feasibility guarantee performance, we adopt the technique in (Tjeng et al., 2018 to reformulate
the ReLU activations expression in (6 For i = 1,..., Nua, let h, denotes W + b;- The output of neuron with ReLU activation is represented as: for k = ,,.... neu and ,..., Nui-
h S h S h
hi h.rinnkk (1 )
0 H M S N max.k 2/ 2, E (0. 13.
(8)
Here we use superscript & to denote the k-th element of a vector. = are (auxiliary) binary variables indicating the state of the corresponding neuron, i.e., 1 (resp. 0) indicates activated (resp. non- activated). We remark that given the value of DNN weights and bias, 2 can be determined (zf can k
be either 0/1 if h 0) for each input O. h; h /h, /h.iin. are the upper/lower bound on the neuron outputs. See Appendix [E.]] for a discussion. With 77 8 the input- relationship in DNN can be represented with set of mixed-intege determine the sufficient DNN size in guaranteeing universal feasibility in Sec. 4.2.2 representation, we use (W, b) to denote DNN weights and bias in the following.
linear inequalities. We discuss how to employ 77 8 to
For ease of
4.2.2 SUFFICIENT DNN SIZE IN GUARANTEEING UNIVERSAL FEASIBILITY.
As methodological contribution, we propose iterative approach to determine the sufficient DNN size for guaranteeing universal solution feasibility in the input region. The idea is to iteratively verify whether the worst-case prediction error of the given DNN is within the room of error (maximum calibration rate), and doubles the DNN's 's width (with fixed depth) if not. We outline the design of the proposed approach below, under the setting where all hidden layers share the same width. Let the depth and (initial) width of the DNN model be Nuit and Neau respectively. Here we define universal solution feasibility that for any input A DD the output of DNN always satisfies (1)-2).
For each iteration, the proposed approach first evaluates the least maximum relative violations among all constraints for all 0 E 'D for the current DNN model via solving the following bi-level program:
min max V', s.t. W.b BED
0 1 S i S Nua, 1 S k S Nneu:
v max (nax((9,(() (g,((, - e,)//e,1), JEE
(10)
where 10p express the outcome of the DNN as function of input O. uf is maximum constraints
violation degree among all constraints. Thus, solving (9)((10) gives the least maximum DNN con-
straint violation over the input region D. We apply gradient descent to solve the above bi-level optimization problem, see Appendix E for details. Let p be the obtained objective value of 9 TO
and (W/, bo) be the corresponding DNN parameters, with which we can directly construct a DNN model. Recall that the determined calibration rate is A, the proposed approach then verifies whether
the constructed DNN is sufficient for guaranteeing feasibility by the following proposition.
Proposition 1 Consider the DNN with Nua hidden layers each huving Nneu neurons and parameters (W7,b) b/). If S A, then v0 E DD the solution generated by this DNN is feasible w.rl -
The proof is shown in Appendix F Proposition T states that if p S A, the worst-case prediction error of current DNN model is within the maximum calibration rate and hence the current DNN size is sufficient for guaranteeing universal feasibility; otherwise, it doubles the width of DNN and moves to the next iteration. Recall that the input- soput-solution mapping for OPLC is continuous. Hence, there exists a DNN such that universal feasibility of DNN solution is guaranteed given the DNN size (width) is sufficiently large according to the universal approximation theorem (Hornik) 1991). See Appendix G for the discussion [ Details of the procedures are shown in Algorithm I After the initialization of DNN model (line 3-4), the proposed approach repeatedly compare the obtained maximum constraints violation (p) with the calibration rate (A), doubles the DNN width (line 5-7), and return the width as Ni until p S A. Thus, we can construct a provable DNN model by the proposed approach, namely DNN-FG as shown in Fig.
We remark that it is challenging to solve the bivle problem 9 10p exactly, i.e., the obtained p is an upper bound of the optimal objective of (9 -0) in each iteration. Nevertheless, as discussed in the following proposition, the upper bound is still useful for analyzing universal solution feasibility.
'One can also increase the DNN depth to achieve universal approximation for more degree of freedom in DNN parameters. In this work, we focus on increasing the DNN width for sufficient DNN learning ability.
5

---

Published as a conference paper at ICLR 2023
Algorithm 1: Determining Sufficient DNN Size
1: Input: A; Initial width Ne
2: Output: Determined DNN width: N
3: Set 0; Sct N neu = N Ninit new
Obtain p via solving 9 10
4: while p > A do
5: Set NT++11 = 2 x : Set = t + 1; Solve 9 (0-10) and update
neu
6: end while
7: Set neu = N
neu
8: Return: Nucu
Proposition 2 Assume > O, Algorithm|] 7 is guaranteed to terminate in a finite number of iter- ations. At each iteration t, consider the DNN with Nua hidden layers each having N neurons, we can obtain as an upper bound to the optimal objective of (0-(10) with a time complexity ((M | 22 2it Nig +4N)..9). p S A, then the DNN with depth Nui and width No. is suf- ficient in guaranteeing universal feasibility. Furthermore, one can construct a feasibility ffeasiblity-guurantee guaranteed DNN with the obtained DNN parameters (W such that for any 0 D, the solution generated by this DNN is feasible w.r.t. -
Proposition 2 indicates p can be obtained in polynomial time. If p S A, it means the current DNN size is sufficient to preserve universal solution feasibility in the input region; otherwise, it means the current DNN size may not be sufficient for the purpose and it needs double the DNN width.
We also remark that the obtained sufficient DNN size may not be the minimal sufficient one if the above bi-level optimization problem is not solved exactly. Please refer Appendix H for detailed discussions. In our case study in Sec. 6 we observe that the evaluated initial DNN size can always guarantee universal feasibility without constraints violation, and we hence conduct further simula- tions with such determined sufficient DNN sizes.
4.3 ADVERSARIAL-SAMIEI
AWARE ALGORITHM
While we can directly construct feasibility -aallitte-grrrteeeddd DNN (without training) shown in Propo- sition it may not achieve strong optimality performance. To this end, we propose an Adversarial- Sample Aware (ASA) algorithm further improve the optimality performance. The algorithm lever- ages the ideas of adversarial learning (Chakraborty et al.] 2018) and active learning (Ren et al.] 2021] techniques, which adaptively incorporates adversarial inputs, i.c., the inputs that cause in- fcasible DNN solutions, for pursuing strong optimality result while preserving universal fcasibility guarantee. We outline the algorithm in the following. Denote the initial training set as T contain- ing randomlyygeeneee geeerrt-geerrteddd input and the corresponding ground- -ruth obtained by solving the calibrated OPLC (with calibration rate A). The proposed ASA algorithm first pre-trains a DNN model with the sufficient size determined by the approach discussed in Sec. 4.2.2 using the initial training set To and the following loss function for each instance as the supervised learning approach:
W1
a x x11
W2
max(g, 0) ég.
(11)
JEE
We leverage the penalty- based training idea in (I). The first term is the mean square error between DNN prediction & and the ground-truth x² provided by the solver for each input. The second term
the inequality constraints violation w.r.t calibrated limits éz. Wi and w2 are positive weighting factors to balance prediction error and penalty. Hence, training DNN by minimizing (I1) can pursue a strong optimality performance as DNN prediction error is also minimized.
However, traditional peased training by only minimizing (11 can not guarantee universal fea- sibility (Venzke et al., 2020 Pan al. 2020b). To address this issue. the ASA algorithm repeatedly updates the DNN model with adversarial samples, anticipating the post-trained DNNs can eliminate violations around such inputs. Specificallyy, given current DNN parameters, it finds the worst-case input 0' E D by solving the inner maximization problem of (0)- Let 7 be the obtained objec- tive value. Recall that the calibration rate is A. If 7 S A, the algorithm terminates; otherwise, incorporates a subset of samples randomly sampled around 0 and solves the calibrated OPLC with A, and starts a new round of training. Details of the ASA algorithm are shown in Appendix] [ We highlight the difference between the DNN obtained in Sec. 4.2.2 and that from ASA algorithm as fol- lows. The former is directly constructed via solving 9 10) which guarantees universal feasibility whilst without considering optimality. In contrast, the latter expected to enhance optimality while preserving universal feasibility as both optimality and feasibility are considered during training. We further provide theoretical guarantee of it in ensuring universal feasibility in the following.

---

Published as a conference paper at ICLR 2023
Proposition 3 Consider DNN model with Nud hidden layers each having Nieu neurons. For each iteration i, assume such a DNN trained with the ASA algorithm can maintain feasibility at the constructed neighborhood Du = [0|0) (1 - a) I < A (1 + a), E D) around A with some small constant a > for Vj S i. There exists constant C. such that the algorithm is guaranteed to ensure universal feasibility as the number of iterations is larger than C.
The proof idea is shown in Appendix - Proposition 3 indicates that, with the iterations is large enough, the ASA algorithm can ensure universal feasibility by progressively improving the DNN performance around each region around worst-case input. It provides a theoretical understanding of the justifiability of the ASA algorithm. In practice, we can terminate the ASA algorithm whenever the maximum solution violation is smaller than the inequality calibration rate, which implies universal feasibility guarantee. We note that the feasibility enforcement in the empirical/heuriitic algorithm achieves strong theoretical grounding while its performance can be affected by the training method chosen. Nevertheless, as observed in the case study in Appendix M the ASA algorithm terminates in at most 52 iterations with 7% calibration rate, showing its efficiency in practical application.
5 PERFORMANCE ANALYSIS OF THE PREVENTIVE LEARNING FRAMEWORK
5.1 ALIVERSAL FEASIBILITY GUARANTEE
We provide the following proposition showing that the preventive learning framework generates two DNN models with universal feasibility guarantees.
Proposition 4 Let A. p. and y be the determined maximum calibration rate, the obtained objective value of (9)-10) to determine the sufficient DNN size, and the obtained maximun relative violation of the trained DNN from iiteesarrl--Sappll Aware algorithm followin following steps in preventive frame- work, respectively. Assume (i) A > 0, (üi) p S A, and (iii) 7i S A. The DNN-FG obtained from determining sufficient DNN size can provably guarantee universal feasibility and the DNN from ASA algorithm further improves optimality without sacrificing feasibility guarantee 40 E D.
Proposition 4 indicates the DNN model obtained by preventive learning framework expected to guarantee the universal solution feasibility, which is verified by simulations in Sec. 6
5.2 RUN-TIME COMPLEXITY
We present the complexity of the traditional method in solving the optimization problems with linear constraints. To the best of our knowledge, OPLC in its most general form is NP-hard cannot be solved in polynomial tic unless P=NP. To better deliver the results here, we consider the specific case of OPLC, namely the mp-QP problem, with linear constraints and quadratic objective function. We remark that the complexity of solving mp-QP provides a lower bower for the general OPLC problem.
Under this setting, the DNN based framework has a complexity of O (N2) whilst the best known
iterative algorithm Yee& & Tse 1989) requires O (N+ + | MM) This means that the computational complexity of the proposed tramework lower than that of traditional algorithms. The comparison results demonstrate the efficiency of the preventive learning framework. See Appendix K for details.
5.3 TRADE-OFF BETWEEN FEASIBILITY AND OPTIMALITY
We remark that guarantee universal feasibility, the preventive learning framework shrinks the feasible region used in preparing training data. Therefore, the DNN solution may incur a larger optimality loss due to the (sub)-optimal training data. It indicates trade-off between optimality and feasibility, i.e., larger calibration rate leads to better feasibility but worse optimality. To fur- ther enhance DNN optimality performance, one can choose a smaller calibration rate than A while enlarging DNN size for better approximation ability and hence achieve satisfactory optimality and guarantee universal feasibility simultaneously.
APPLICATION IN SOLVING DC-OPF AND NON-CONVEX OPTIMIZATION
6.1
DC-OPF PROBLEM AND DEEPOPF+
DC-OPF is a fundamental problem for modern grid operation. It aims to determine the least-cost generator output to meet the load in a power network subject to physical and operational constraints. With the penetration of renewables and flexible load, the system operators need to handle significant uncertainty in load input during daily operation. They need to solve DC-OPF problem under many scenarios more frequently and quickly in a short interval, e.g.. 1000 scenarios in 5 minutes, obtain a stochastically optimized solution for stable and economical operations. However, iterative solvers may fail to solve a large number of DC-OPF problems for large-scale power networks fast enough for the purpose. Although recent DNN-based schemes obtain close-to-optimal solution much faster than conventional methods, they do not guarantee solution feasibility. We design DeepOPF+ by em- ploying the preventive learning framework to tackle this issue. Consider the DC-OPF formulation:
7

---

Published as a conference paper at ICLR 2023
min Ea Pa. 1
s.l. P Pa S P .M. Pc- PD. Pimce Bine + Pume (12)
1EG
GG E R15 (resp. PG and line" E R are the minimumm (resp. maximum) generation limits of generaton and branch flow limits of the set of transmission lines denoted as K. G. B,
M. Bline. and o E R!5 denote the set of generators, buses, bus admittance matrix, line admittance matrix, and bus phase angles, respectively. The objective is the total generation cost and C; (-) is the cost function of each generator, which is usually strictly quadratic (Park et al. 1993] tpe 2018)
from generator's s heat rate curve. Constraints in (2) enforce nodal power balance equations and the limits on active power generation and branch flow, DC-OPF is hence a quadratic programming and admits a unique optimal solution w.r.t. load input Pp- Analogy to OPLC ( Eieg c (Pas) is the objective function f in [I. PD is the problem input and (PG. D) are the decision variables x.
We apply the proposed preventive-learning framework to design a DNN scheme, named Deep- OPF+, for solving DC-OPF problems. We refer interested readers to Appendix [ details. De- note A, p. and 7 as the obtained maximum calibration rate, the obtained objective value of (9)-TT) to determine sufficient DNN size, and the maximum relative violation of the trained DNN from Adversarial-Sample Aware algorithm in DeepOPF+ design, respectively. We highlight the feasi- bility guarantee and computational efficiency of DeepOPF+ in following proposition.
Corollary Consider the OC-OPF problem and DNN model defined in O Assume (i) A > 0, (ii) P S A, and (iii) y S A, then the DeepOPF+ generates a DNN guarantees universal feasibility for any PD E 'D. Suppose the DNN width is the same order of number of bus B, then DeepOPF+ has a smaller computational complexity of O (B²) compared with that of state-of- ahrtt--rrttt iterative methods O (B*), where B is the number of buses.
Corollary | says that DeepOPF+ can solve DC-OPF with universal feasibility guarantee at lower To
computational complexity compared to conventional iterative solvers as DNNs with width O(B) can achieve desirable feasibility/optimmaltyy Such assumption is validated in existing literature (Pan et al. 2019) and our simulation. To our best knowledge, DeepOPF+ is the first DNN scheme to solve DC-OPF with solution feasibility guarantee without post-processing. We remark that the DeepOPF+ + design can be easily generalized to other linearized OPF models (Cain et al. 2012
Yang et al. 2018 Bolognani & Dörfler 2015)
6.2 PERFORMANCE EVALCATION OVER IEEE TEST CASES
We evaluate its performance over IEEE 30-/118-/300 bus test cases tpc 2018 on the input load region of [100%, 130%| of the default load covering both the light-load (100%, 115%]) and heavy- load (1115%, 130%) regimes, respectively. We conduct simulations in CentOS 7.6 with a quad-core (i7-3770@3 3. 3.40G Hz) CPU and 16GB RAM. We compare DeepOPF+ with five baselines same training/test setting: (i) Pypower: the conventional iterative OPF solver; (ii) DNN-P: A DNN scheme adapted from (Pan al. 2019). It learns the load-solution mapping using penalty approach without constraints calibration and incorporates projection post processing if the DNN solution is infeasible; (iii) DNN-D: A penalty- based DNN scheme adapted from (Donti et al. 2021 It includes a correction step for infeasible solutions in training/testing: (iv) DNN-W: A hybrid method adapted from (Dong et al. 2020a). It trains a DNN to predict the primal and dual variables as the warm- start points to the conventional solver; (V) DNN-G: A gauge-function based DNN scheme adapted from (Li et al. 2022). It enforces solution feasibility by first solving a linear program to find a feasible interior point, and then constructing the mapping between DNN prediction in an loo unit ball and the optimum. For better evaluation, we implement two DeepOPF+ schemes with different DNN sizes and calibration rate (3%, 7%) that all within the maximum allowable one, namely DeepOPF+-3, and DeepOPF+-7. The detailed designs/results are presented in Appendix [
the
We use the following performance metrics: (i) the percentage of the feasible solution obtained by DNN, (ii) the average relative optimality difference between the objective values obtained by DNN and Pypower, (iii) the average speedup, i.c., the average running-timg ratios of Pypower to DNN-
$ PG. Pinii = Panax 0, Vi & G. and Pp. = 0, Vi & A. where A denotes the set of load buses. WW remark that the training of DNN conducted offline; thus, its complexity is minor as amortized over many DC-OPF instances, e.g., 1000 scenarios per 5 mins. Meanwhile, the extra cost to solve the new-introduced programs in our design is also minor observing that existing solvers like Gurobi can solve the problems effi- ciently, e.g., <20 minutes to solve th MILPs to determine calibration rate and DNN size. Thus, we consider the run-time complexity of the DNN scheme, which widely used in existing studies.
8

---

Published as a conference paper at ICLR 2023
Table 1: Performance comparison with SOTA DNN schemes in light-load and heavy-load regimes.
Case
Scheme
Average speedups
Peasibility ratc (8)
Optimality loss (8)
Worst-case Violation (%)
ligh-Innd
heavy-lond
lightload
Tocay--ond
nightload
heavy-lond
Tight
heavy- Inad
DNNFP
X85
86
100
88.12
0.02
0.03
5.43
DNN-D
X85
84
100
93.36
0.02
0.03
11.19
Case30
DNN-W
x0,90
X0,86
100
100
0
DNNNGG
x24
x-26
100
0.04
DeepOPF+1 3
X86
x92
100
100
0.03
0.04
DeepOFF+/
X86
x91
100
0.09
DNN-P
x137
x125
68.84
54.92
0.17
0.21
19.5
44.8
DNN-D
133
73:42
55.37
0:24
4.3
Case
W
x2.08
x2.20
100
100
0
DNN-G
x26
100
1.29
0.39
DeepOPF+
x201
x226
100
100
0.18
0.19
DeepOpF+
x202
x2228
100
100
0.37
0.41
DNN-P
x115
x98
91. 29
78.42
0.06
0.08
261.1
443.0
DNN-D
x115
x x102
91.99
K2.92
0.07
0.07
231.6
348.1
Case300
DNN-W
x1.14
x1.08
100
100
0
DNN-G
x2.44
X2.65
100
100
0.32
0.06
+
x12
x136
100
100
0.03
0.03
DeepOPF+> +-7
X
x138
100
100
0.10
0,06
Feasibilitt rate and Worst-case violation are the results before post-procssing. Feasibility rares (resp Worst-casc violation) after post-
processing are 100% (resp 0) for all DNN schemes. We hence report the results before post-pros processessing to better show the advantage of
our design. Speeduup and Optimality loss are the resalts after post processing of the final oblained feassibee solutions.
Thhe correction step DNN-D (with 10-3 rate) takes longer time compared with I -ppojecction in DNN-P, resulting in lower speedups.
We empirically observe that DNN- G requires more training epochs for satisfactory performance. We repoct its best results at 500 epoclis heavy-loud and the
Casel 18/300
at 400 epochs
other cases. The training epuchs other DNN schemes are 200.
based approach for the test instances, respectively. (iv) the worst-case violation rate, i.e., the largest constraints violation rate of DNN solutions in the entire load domain.
6.2.1 PERFORMANCE COMPARISONS BETWEEN DEEPOPF+ AND EXISTING DNN SCHEMES.
The results are shown in Table with the following observations. First, DeepOPF+ improves over DNN-P/DNN-D in that it achieves consistent speedups in both light-load and heavy-load regimes. DNN-P/DNN-D achieves a lower speedup in the heavy-load regime than in the light-load regime as a large percentage of its solutions are infeasible, and it needs to involve a post post-processing procedure to recover the feasible solutions. Note that though DNN-P/DNN-D may perform well on the test set in light-load regime with a higher feasibility rate, its worst-case performance over the entire input domain can be significant, e.g.. more than 443% constraints violation for Case300 in the heavy-load region. In contrast, DeepOPF+ guarantees solution feasibility in both light-load and heavy-load regimes, eliminating the need for post-processing and hence achieving consistent speedups. Sec- ond, though the warm-start/interior point based scheme DNN-W/DNN-G ensures the feasibility of obtained solutions, they suffer low speedups/large optimality loss. As compared, DeepOPF+ achieves noticeably better speedups as avoiding the iterations in conventional solvers. Third, the op- timality loss of DeepOPF+ is minor and comparable with these of the existing state-of-the-art DNN schemes, indicating the effectiveness of the proposed Adversarial-Samma Sample Aware training algorithm. Fourth, we observe that the optimality loss of DeepOPF+ increases with a larger calibration rate, which is consistent with the trade-off between optimality and calibration rate discussed in Sec. [.3] We remark that DC-OPF is an approximation to the original non-convex non-linear AC-OPF in power grid operation under several simplifications. DC-OPF is widely used for its convexity and scalability. Expanding the work to AC-OPF is a promising future work as discussed in Appendix B
Moreover, we apply our framework performance advantage over existing schemes. Detailed design/results are shown in Appendix N
non-convex problem in (Donti et al. 2021) and show its
7 CONCLUDING REMARKS
We propose preventive learning as the first framework to develop DNN schemes to solve OPL. with solution feasibility guarantee. Given sufficiently large DNN, we calibrate inequality constraints used in training, thereby anticipating DNN prediction errors and ensuring the obtained solutions remain feasible. We propose an Adversarial- optimality. We apply the framework to develop DeepOPF+ to solve and DC-OPF problems in grid operation. Simulations show that it outperforms existing strong DNN baselines in ensuring feasibil- ity and attaining consistent optimality loss and speedup in both light-load and heavy-load regimes. We also apply our framework to non-convex problem and show its performance advantage over existing schemes. We remark that the proposed scheme can work for large-scale systems because of the desirable scalability of DNN. Future directions include extending the framework general non- linear constrained optimization problems like ACOPF and evaluating its performance over systems with several thousand buses and realistic loads discussed in Appendix B
Aware training algorithm to improve DNN's
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
The work presented in this paper was supported in part by a General Research Fund from Research Grants Council, Hong Kong (Project No. 11203122), an InnoHK initiative, The Government of the HKSAR, and Laboratory for AI-Powered Financial Technologies. We thank Dr. Andreas Venzke for his contributions to some initial results and the discussions related the fully-developed results pre- sented in the paper. We would also like to thank the anonymous reviewers and program committee of ICLR 2023 for giving insightful comments on this article.
REFERENCES
Power Systems Test Case Archive, 2018 http:://abs.ece..uu
Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and Zico Kolter. Differentiable convex optimization layers. Advances in Neural Information Processing Systems, 32:9562-9574, 2019.
Brandon Amos and Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, pp. 136-145,
Alberto Bemporad and Carlo Filippi. An algorithm for approximate multiparametric convex pro- gramming. Computational optimization and applications, 35(1):87-108, 2006.
Alberto Bemporad, Manfred Morari, Vivek Dua, and Efstratios N Pistikopoulos. The explicit lution model predictive control via multiparametric quadratic programming. In Proceedings of the 2000 American Control Conference. ACC (IEEE Cat. No. 00CH36334), volume 2, pp- 872 876. IEEE, 2000.
Saverio Bolognani and Florian Dörfler. Fast power system analysis via implicit linearization of the power flow manifold. In 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 402-409, IEEE, 2015.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Mary B Cain, Richard O'enel,,, and Anya Castillo. History of optimal power flow and formulations. Federal Energy Regulatory Commission, 1:1-36, 2012.
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopad- hyay. Adversarial attacks and defences: A survey, arXiv preprint arXiv:1810.00060 2018.
Spyros Chatzivasileiadis. Optimization in modern power systems. Lecture Notes. Tech. Univ. of Denmark. Available online: https://arxiv. org/pdf/1 1t/pd///////// 1acp/f/8811.00944 1.00943. pdf. 2018.
Minas Chatzos, Ferdinando Fioretto, Terrence WK Mak, and Pascal Van Hentenryck. High- fidelity machine learning approximations of lcage-scale optimal power flow. arXiv preprint arXiv:2006.1633 2020.
John M Danskin. The theory of max-min and its application to weapons allocation problems, vol- ume 5. Springer Science Business Media, 2012.
Roel Dobbe, Oscar Sondermeijer, David Fridovich-Keil, Daniel Amoll, Duncan Callaway, and Claire Tomlin. Toward distributed energy services: Decentralizing optimal power flow with ma- chine learning. IEEE Transactions Smurt Grid, 1(2)):296-1300) 2019.
Wenqian Dong, Zhen Xie, Gokcen Kestor, and Dong Li. Smart-pgsim: using neural network to accelerate AC-OPF power grid simulation. In SC20: International Conference for High Perfor- mance Computing, Networking. Storage and Analysis, pp. 1-15. IEEE, 2020a.
Yinpeng Dong, Zhijie Deng, Tianyu Pang, Jun Zhu, and Hang Su. Adversarial distributional training for robust deep learning. In Advances in Neural Information Processing Systems, volume 33, pp. 8270-8283, 2020b.
10

---

Published as a conference paper at ICLR 2023
Priya L. Donti, David Rolnick, and J Zico Kolter. DC3: A learning method for optimization with hard constraints. International Conference on Learning Representations, 2021.
Adam N Elmachtoub and Paul Grigas, Smart "predict, then optimize". Management Science, 68(1); 9-26, 2022.
Nuno Faísca, Vivek Dua, and Efstratios N Pistikopoulos. Multiparametric linear and quadratic programming, 2007.
Silvia Ferrari. Multiobjective algebraic symthesis of neural control systems by implicit model fol- lowing. IEEE Transactions on Neural Networks, 20(3):406-419) 2009.
Ferdinando Fioretto, Terrence WK Mak, and Pascal Van Hentenryck. Predicting AC optimal power flows: Combining deep learning and lagrangian dual methods. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 34, pp. 630-637, 2020.
José Fortuny-Amat and Bruce McCarl. A representation and economic interpretation of a two-level programming problem. Journal of the operational Research Society, 32(9):783-792, 1981.
Neel Guha, Zhecheng Wang, Matt Wytock, and Arun Majumdar. Machine learning for AC optimal power flow. arXiv preprint arXiv:1910.088: 2019.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, (2)2551-257, 1991.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks uni- versal approximators. Neural networks, 2(5):359-366, 1989.
Wanjun Huang and Minghua Chen. DeepOPF-NGT: A fast unsupervised learning approach for solving AC-OPF problems without ground truth. In JCML 2021 Workshop on Tackling Climate Change with Machine Learning, 2021.
Wanjun Huang, Xiang Pan, Minghua Chen, and Steven H Low. Deepopf-V: Solving AC-OPF problems efficiently. IEEE Transactions on Power Systems, 37(1):800- 8030000--3,110 2021.
James Kotary, Ferdinando Fioretto, and Pascal Van Hentenryck. Learning hard optimization prob- lems: A data generation perspective. Advances in Neural Information Processing Systems, 34: 24981-24999 2021a.
James Kotary, Ferdinando Fioretto, Pascal Van Hentenryck, and Bryan Wilder. End-to-end con- strained optimization learning: A survey. In Proceedings of the Thirtieth International Joint Conference on Arlificial Intelligence, IJCAI, 2021b.
Eugene L Lawler and David E Wood. Branch-and-bound methods: A survey. Operations research, (K(4):699-719, 1966.
Xingyu Lei, Zhifang Yang. Juan Yu, Junbo Zhao, Qian Gao, and Hongxin Yu. Data-driven opti- mal power flow: physics-ifformed machine learning approach. IEEE Transactions on Power Systems, (6(())334-33)) 2020.
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward net- works with nonpolynomial activation function can approximate any function. Neural networks, 6(6):861-867, 1993.
Meiyi Li, Soheil Kolouri, and Javad Mohammadi. Learning solve optimization problems with hard linear constraints. arXiv preprint arXiv:2208.106 10611, 2022.
Suttipan Limanond and Jennie Si. Neural network- based control design: An LMI approach. 1EEE Transactions on Neural Networks, 1422 1998.
Rahul Nellikkath and Spyros Chatzivasileiadis. Pnysics-informed neural networks for minimising worst-case violations in DC optimal power flow. In 2021 IEEE International Conference Communications, Control. and Computing Technologies for Smart Grids (SmartGridComm) pp- 419-424. IEEF, 2021.
11

---

Published as a conference paper at ICLR 2023
Rahul Nellikkath and Spyros Chatzivasilciadis. Physics-informet neural networks for AC optimal power flow. Electric Power Systems Research, 212:10 2022.
Xiang Pan, Tianyu Zhao, and Minghua Chen. DeepOPF: Deep neural network for DC optimal
power flow. In 2019 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmarGridCommn) 2019.
Xiang Pan, Minghua Chen, Tianyu Zhao, and Steven H Low. DeepOPF: A Frasibllity-Ottimizeed Deep Neural Network Approach for AC Optimal Power Flow Problems. arXiv preprint arXiv:2007.01002 2020a.
Xiang Pan, Tianyu Zhao, Minghua Chen, and Shengyu Zhang. DeepOPF: A deep neural network approach for security-constrainee DC optimal power flow. IEEE Transactions on Power Systems, 36(3):1725-1735. 1735,, 2020b.
Xiang Pan, Wanjun Huang, Minghua Chen, and Steven H Low. Deepopf-AL: Augmented learning for solving AC-OPF problems with multiple sad-solluioon mappings. Proceedings of the Four- teenth ACM International Conference on Future Energy Systems (ACM e-Energy 2023), 2023,
J. H. Park, Y. Kim, I. K. Eom, and K. Y. Lee. Economic load dispatch for piecewise quadratic cost function using hopfield neural network. IEEE Transactions on Power Systems, 83)):1330-1033; Aug 1993. ISSN 0885-8950.
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12 (1):145-151, 1999.
Chongli Qin, Krishnamurthy (Dj) Dvijotham, Brendan O' O'Donoghue, Rudy Bunel, Robert Stanforth, Sven Gowal, Jonathan Uesato, Grzegorz Swirszez, and Pushmeet Kohli. Verification of non-linear specifications for neural networks. In International Conference on Learning Representations, 2019.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. A survey deep active learning. ACM computing surveys (CSUR), 54(9):1-40, 2021.
Riva Sanseverino, ML. Di Silvestre, Mineo, Favuzza, NQ Nguyen, and QTT Tran. multi- agent system reinforcement learning based optimal power flow for islanded microgrids. In 2016 IEEE 16th International Conference on Environment and Electrical Engineering (EEEIC), 2016.
Fatemeh Sheikholeslami, Ali Lotfi, and J Zico Kolter. Provably robust classification of adversarial examples with detection. In International Conference on Learning Representations, 2020.
Haoran Sun, Xiangyi Chen, Qingjiang Shi, Mingyi Hong, Xiao Fu, and Nicholas D Sidiropoulos. Learning to optimize: Training deep neural networks for interference management. IEEE Trans- actions on Signal Processing, 66(20):5438-555) -553385553320000 2018.
Vincent Tjeng, Kai Y Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. In International Conference on Learning Representations, 2018.
Zain ul Abdeen, He Yin, Vassilis Kekatos, and Ming Jin. Learning neural networks under input- output specifications. In 2022 American Control Conference (ACC), pp. 1515-1520. IEEE, 2022.
Pravin M Vaidya. Speeding-up linear programming using fast matrix multiplication. In IEEE FOCS, pp. 332-337, 1989.
Andreas Venzke, Guannan Qu, Steven Low, and Spyros Chatzivasileiadis. Learning optimal power flow: Worst-case guarantees for neural networks. In 2020 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmanGridConm), 2020.
Wenchao Xia, Gan Zheng, Yongxu Zhu, Jun Zhang, Jiangzhou Wang, and Athina P Petropulu. A deep learning framework for optimization of miso downlink beamforming. IEEE Transactions on Communications, (8(3):1866. 1880, 2019.
12

---

Published as a conference paper at ICLR 2023
Zhifang Yang, Kaigui Xie, Juan Yu, Haiwang Zhong, Ning Zhang, and Qing Xia. A general formu- lation of linear power flow models: Basic theory and error analysis. IEEE Transactions on Power Systems, 34(2):1315-1324 1324, 2018.
Yinyu Ye and Edison Tse. An extension of Karmarkar's projective algorithm for convex quadratic programming. Mathematical Programming, 44(1):157-179, May 1989.
Ahmed S Zamzam and Kyri Baker. Learning optimal solutions for extremely fast AC optimal power flow. In 2020 IEEE International Conference on Communications, Control, and Computing Tech- nologies for Smart Grids (SmariGridCommm) 2020.
Tianyu Zhao, Xiang Pan, Minghua Chen, Andreas Venzke, and Steven H Low. DeepOPF+: A deep neural network approach for DC optimal power flow for ensuring feasibility. In 2020 IEEE International Conference Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), 2020.
Min Zhou, Minghua Chen, and Steven H Low. DeepOPF-FT: One deep neural network for multiple AC-OPF problems with flexible topology. IEEE Transactions on Power Systems, 38(1): -677, 2022.
13


Reviews:
[{"review_id": "oVCZOKuyDf7", "reviewer": "Reviewer_Z4Mb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the first preventive learning based approach to guarantee DNN solution feasible for optimization problem with linear constrains. The key idea is to obtain an conservative satisfication of constriants and adjusting the size of DNN accordlying to guarantee that the redundant feasible region is large enough to offset the approximation error of DNN. The development of the algorithm is heavily based on bi-level optimization which is difficult to solve exactly and the author proposed several approximation approaches to address these issues.", "summary_of_the_review": "Overall I think this paper proposed a very interesting approach and most of the results look solid to me. However, since I am not the expert in this areas so I will leave the decision to other reviewers.", "strength_and_weaknesses": "Strength:\n(1) This paper proposed the first DNN-based approach to solve constrianed problem with linear constraints without post-processing, which could be interesitng to the community.\n(2) Most of the steps in the algorithm is supported by theoretical guarantees.\n(3) The effectiveness of the proposed method is supported by some empirical results.\n\nWeakness:\nI didn't see obvious weakness in this paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lubkuDlmU7", "reviewer": "Reviewer_f2t2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a workflow for using neural networks to solve optimisation problems while guaranteeing the feasibility of the solution.", "summary_of_the_review": "I like the direction this paper is taking and the proposed techniques seem promising. However I find the experimental evaluation a little insufficient to evaluate the effectiveness of the approach and I am concerned with the scalability of the approach.\n", "strength_and_weaknesses": "Strength: \n\n1. The paper tackles an important problem of guaranteeing correctness of DNN-based solution (i.e., does not violate constraints) for the optimization problem, which is a major limitation of end-to-end neural network solutions. \n\n2. The evaluation suggests that the proposed techniques outperform baseline methods either in feasibility rate or optimality loss. \n\nWeakness:\n1. The repeated solving of (9)-(10) seems to be very expensive as it involves repeatedly solving MILPs.  \n\n2. It is unclear from the paper how challenging the CD-OPT problem actually are. If would be helpful if the authors report the concrete runtime of Pypower on these problems. \n\n3. It is unclear how scalable the proposed approach is (e.g., how fast the size of the network grows as the number of constraints increases, how long each component of the procedure takes). It would be helpful to report these numbers. \n\nQuestions:\n1. Could the authors clarify why the branch-and-bound procedure for computing the calibration rate has polynomial time complexity? Isn't MILP solving an NP complete problem?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fEA80PxG-wH", "reviewer": "Reviewer_ek1R", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a framework for learning provably feasible approximations to optimization problems with linear constraints. This framework entails\n* Rewriting the original optimization problem, if it has (linear) equality constraints, in a solely inequality-constrained form, by using variable reduction techniques,\n* Constructing a buffer on the inequality constraints of the problem in a way that still ensures feasibility with respect to the input space,\n* Sizing a neural network to be large enough to be able to actually achieve feasibility over this input space (and actually solving for the parameters of such a feasible neural network, in order to initialize the network),\n* Training the neural network, using both input data and additional adversarial samples generated during neural network training to improve performance on the optimization objective.\n \nThe authors demonstrate the efficacy of their approach on DC optimal power flow, a linearly-constrained optimization problem used in power grids to schedule power generators under (approximate) physical constraints.\n", "summary_of_the_review": "This paper proposes a framework for learning provably feasible approximations to optimization problems with linear constraints. The submission is clearly-written, provides new ideas (including tying together many tools/pieces from across the literature), and is high-quality in both its theoretical justification and its experimental comparisons.", "strength_and_weaknesses": "Strengths: \n* The problem studied is one of importance, as providing fast and feasible approximators for optimization problems is widely applicable (including in power grid optimization, which the authors explicitly study).\n* The paper proposes a very interesting framework for addressing this problem, which nicely ties together many tools and pieces from across the literature.\n* The experimental comparison is thorough and well-done.\n* The writing and presentation is generally clear and well-done.\n \nWeaknesses: \n* For the ICLR audience, it is important to explain that DC-OPF is an approximation to the true power grid operation problem - otherwise, they may not be aware (and the applicability to DC-OPF only, rather than also AC-OPF, is a key limitation of the work).\n* Corollary 1 states: “Suppose the DNN width is the same order of number of bus $B$.” This assumption is not justified in the text.\n\nNote: I have previously reviewed this paper, and believe the authors have addressed all my major concerns from last time. I think this paper is deserving of publication at ICLR.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L1Gutnb5JQ", "reviewer": "Reviewer_crFR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an algorithm for producing feasible solutions for families of linearly constrained optimization problems with varying right-hand sides. The algorithm can provide feasibility guarantees under certain circumstances by 1) dilating each of the linear constraint (shrinking the feasible region), 2) training a neural network that maps a right-hand side to a \"good\" point, and then 3) bounding the approximation error of the NN w.r.t. the dilated region to provide a feasibility guarantee w.r.t. the original feasible region. Computational results on optimal power flow problems are provided.", "summary_of_the_review": "While the method proposed by the authors uses some interesting ideas, I have concerns about the rigor, framing, and generality of the contributions claimed by the authors.", "strength_and_weaknesses": "The approach of the paper is a reasonable one, and stitches together results from a number of diverse areas in discrete optimization, deep learning, and power systems. However, I have some concerns about a number of technical errors or confusing mathematical informalities, and also feel that the authors oversell the generality of their proposed method.\n\n* The authors assume that (1-2) admits a unique optimum for each value for theta. Is this a reasonable assumption? If so, why? It seems quite restrictive to me (for example, most \"interesting\" families of linear optimization problems will not satisfy this), and seems very dependent on the choices of script{D} and the objective. Please provide some examples, or some explanation of how you would verify this property a priori, or an explanation of how the algorithm proceeds without this guarantee. Additionally, on p8 the authors state that \"In general, DC-OPF is a quadratic programming problem and admits a unique optimal solution w.r.t. load input\". Can the authors provide a citation or proof to this effect? What conditions are placed on the objective and script{D} to make this result hold? I am skeptical such a result will hold if the quadratic objective is simply the zero function, for example.\n\n* The authors state that \"it suffices to focus on OPLC with inequality constraints\". The justification in Appendix B is that any equations can be projected out to deal only with the original variables. This is true, though the authors should take care to consider implied equations. Note that implied can appear conditionally based on the parameter values: consider a simple example with the constraints x <= 2 - theta and x >= theta, with script{D} = {theta in [0, 1]}.\n\n* I am very confused by Remark (i) on p4: the reference points to a paper proving a time complexity bound for linear programming, but the problem under consideration is a _mixed-integer linear programming problem_. In general these will be NP-hard to solve.\n\n* The abstract is promising too much: The \"guarantee...[of] solution feasibility for optimization problems with linear constraints\" is contingent on the existence and algorithmic discovery of quantities with certain properties that may not exist (and it seems difficult to prove a priori _if_ they will exist). Moreover, the paper focuses on learning over families of related instances with only variation of the constraint right-hand sides (essentially), which is not mentioned until Section 3. These restrictions are all perfectly fine and justifiable, but they should be made front-and-center.\n\n* I am somewhat skeptical that the proposed algorithm will work well for general OPLC problems. There are seemingly a number of places where the approach can break down; for example: a) if there does not exist a delta > 0 (see \"implied equations\" point above), b) if such a value for delta exists but cannot be computed in a reasonable amount if time, c) if optimizing a single calibration value in (4-5) for all constraints is too conservative, d) if the DNN size grows needed for a feasibility guarantee is impractically large, or e) if the best solutions lie near the feasible region boundary and are \"cut off\" by the dilation. The author's open the door for such questions by framing their approach as a general purpose one, and so I think the paper would be more defensible if either 1) the authors present a more diverse set of computational experiments, or 2) tighten the focus of the paper towards DC-OPF. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%