Optimization Proxies using Limited Labeled Data and Training Time -- A Semi-Supervised Bayesian Neural Network Approach
Authors: Parikshit Pareek
Abhijith Jayakumar
Kaarthik Sundar
Sidhant Misra
Deepjyoti Deka
Venue: ICML 2025 poster (2025)
Keywords: Bayesian Neural Network
Optimization Proxy
Semi-Supervised Learning
Primary area: probabilistic_methods->bayesian_models_and_methods
TL;DR: Leverage feasibility to train optimization proxies more efficiently by augmenting unlabeled datasets with feasibility labels. Enabled by a sandwich-style training approach, reducing the need for extensive labeled data.

Abstract: Constrained optimization problems arise in various engineering systems such as inventory management and power grids. Standard deep neural network (DNN) based machine learning proxies are ineffective in practical settings where labeled data is scarce and training times are limited. We propose a semi-supervised Bayesian Neural Networks (BNNs) based optimization proxy for this complex regime, wherein training commences in a sandwiched fashion, alternating between a supervised learning step for minimizing cost, and an unsupervised learning step for enforcing constraint feasibility. We show that the proposed semi-supervised BNN outperforms DNN architectures on important non-convex constrained optimization problems from energy network operations, achieving up to a tenfold reduction in expected maximum equality gap and halving the inequality gaps. Further, the BNN's ability to provide posterior samples is leveraged to construct practically meaningful probabilistic confidence bounds on performance using a limited validation data, unlike prior methods.

Optimization Proxies using Limited Labeled Data and Training Time
- -A Semi-Supervised Bayesian Neural Network Approach
Parikshit Pareek
Abhijith Jayakumar?
Kaarthik Sundar Sidhant Misra
Deepjyoti Deka
Abstract
considerable efforts to solve optimization problems using deep neural networks (DNNs) (Khadivi et al., 2025; Kotary et al., 2021; Fajemisin et al., 2024). The idea of learning input-to-output mappings has been explored via supervised and unsupervised methods, particularly in power system applications (Zamzam Baker, 2020; Donti et al., 2021; Park & Van Hentenryck, 2023; Fioretto et al., 2020; Ko- tary et al., 2021; Rolnick al., 2022a; Piloto et al., 2024). Additionally, constraint penalization approaches have been proposed to enforce fcasibility in predicted outputs within DNN loss functions (A14OPT, 2023).
Constrained optimization problems arise in vari- ous engineering systems such as inventory man- agement and power grids. Standard deep neural network (DNN) based machine learning proxies are ineffective in practical settings where labeled data is scarce and training times are limited. We propose a semi-supervised Bayesian Neural Net- works (BNNs) based optimization proxy for this complex regime, wherein training commences in sandwiched fashion, alternating between su- pervised learning step for minimizing cost, and an unsupervised learning step for enforcing con- straint fcasibility. We show that the proposed semi-supervised BNN outperforms DNN archi- tectures on important non-convex constrained op- timization problems from energy network opera- tions, achieving up to tenfold reduction in ex- pected maximum equality gap and halving the inequality gaps. Further, the BNN's ability to pro- vide posterior samples leveraged to construct practically meaningful probabilistic confidence bounds performance using a limited validation data, unlike prior methods.
Supervised DNN models rely on labeled datasets obtained by solving numerous 10 or more) instances of optimiza- tion problems. This data generation step poses significant limitation due to prohibitive computational times required for moderate to large problem instances, particularly if the system topology and other parameters change over time. For example, (Park & Van Hentenryck, 2023) report that generating labeled data for a medium-sized power grid prob- lem takes over three hours' Unsupervised methods aim to address the labeled data generation issue (Donti et al., 2021; Park & Van Hentenryck, 2023); however, they often have high training time requirements, particularly due to the use of projectioon/comrcciion steps to ensure feasibility within the framework (Donti et al., 2021; Gupta et al., 2022; Zamzam & Baker, 2020). Moreover, unsupervised methods still re- quire large number of labeled data perform validation and provide confidence bounds on error with respect true solution.
1. Introduction
Constrained optimization problems are fundamental in the optimal operation of various engineering systems, such as supply chains, transportation networks, and power grids. Learning forward mapping between the inputs and outputs of these problems can significantly reduce computational burdens, especially when rapid solutions are required, such as in electricity markets or real-time transportation planning.
Thus, it is important to note that practical performance of ML based optimization proxies needs to be qualified under constraints of both : (a) Total Labeled Data (training + validation), and (b) Training time. This is true in the context of bi-level optimization problems such in power grid planning where ML proxies may serve as subroutines to simulate decision-making processes for given first-level decisions (Ibrahim et al., 2020). Here, ML models must be adaptable in both training and validation to changing problem inputs and parameters. Minimizing or limiting both the time and data required for learning input/output mappings optimization problems is thus cru-
Recent advancements in machine learning (MI.) have led to
'Department of Electrical Engineering, Indian Institute of Tech- nology Roorkee, India Loos Alamos National Laboratory, NM, USA MIT Energy Initiative, MIT, Cambridge, USA. Correspon- dence to: Parikshit Pareck <pareck@ee in...it.........................
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'Sec Table (Park & Van Hentenryck, 2023).

---

Semi-Supervised Optimization Proxies using Bayesian Neural Network
cial.
et al., 2021; Park & Van Hentenryck, 2023; Kotary et al., 2021). The primary motivation is to replace time-consuming optimization algorithms with machine learning proxies, en- abling instantaneous solutions to large number of problem instances (Park & Van Hentenryck, 2023; Donti et al., 2021; Gupta et al., 2022; Zamzam & Baker, 2020).
Estimating the generalization error of ML models over the testing dataset is another aspect that is particularly relevant to engineered systems where the system must obey physical and safety limits. When limited labeled data is available for validation, one relies on concentration results such as Hoeffding's 's inequality (Sridharan, 2002: Hoeffdiing 1994) to develop error bounds using finite out-of-sample data. However, these bounds are often loose and impractical, and creates the need for frameworks that enable tighter expected error bounds with limited labeled validation data.
Outside the realm of optimization proxies, several semi- supervised learning methods have been proposed lever- age unlabeled data for improving ML model performance (Yang et al., 2022). These approaches include augmenting unlabeled data with inexpensive pscudo-labels and devel- oping unsupervised loss functions to be minimized along- side supervised loss functions (Sharma et al., 2024; Yang et al., 2022). Data augmentation has been used in image classification with Bayesian Neural Networks (BNNs) us- ing the notion of semantic similarity (Sharma et al., 2024). However, this concept is not readily extensible to MI prox- ies for constrained optimization problems, where semantic similarity is hard to quantify for input variations leading to changes in the output. To address this challenge, we propose a feasibility- data augmentation scheme that relates directly to the constraints of the optimization prob- lem. To the best of our knowledge, these ideas have not been explored in the context of BNN algorithms for solving large-scale optimization problems. A related but distinct line of work involves loss function-based prior design for output constraint satisfaction (Sam et al., 2024; Yang et al., 2020).
Motivated by the preceding discussion, this paper considers the problem of designing optimization proxies with improved confidence bounds in the setting of limited training time requirement and limited labeled data availability. Our major contribution is the develop- ment of a Bayesian Neural Network (BNN) coupled with a sumi-super training approach for this setting. that can be used to give tight confidence bounds on predictions. First, we propose the use of BNNs instead of DNNs for learning input-to-output mappings, as they provide intrinsic uncertainty quantification and allow the integration of prior beliefs (Papamarkou et al., 2024). Second, we introduce a Sandwich learning method for BNN, which integrates unlabeled data into training through feasibility- -based data augmentation. This approach enforces feasibility without requiring more labeled instances. Third, we utilize the pre- dictive variance information provided by BNNs to develop tight and practically useful expected error bounds using Bernstein concentration bounds (Audibert et al., 2007). We intentionally restrict ourselves to 1000 training instances and 10 minutes of training tim on a single CPU core to demon- strate the effectiveness of the proposed learning scheme under low-data, low-compute settings. For various power grid optimization problem instances (57-Bus, 118-Bus, 500- Bus and 2000-Bus), we show that (i) supervised BNNs outperform standard supervised DNN approaches under limited training time and data; (ii) the proposed Sandwich BNN enforces feasibility better than supervised BNNs with- out requiring additional training time or data; and (iii) the Bernstein bound-based expected error bounds are tight and practically useful for constraint satisfaction studies with- out extra computational effort. The proposed BNN-based approaches achieve at least an order of magnitude lower maximum equality gap compared to state-of-the-art DNN models, without compromising the optimality gap.
Contributions:
2. BackgrounddPrelinnnarees
2.1. Problem Setup and Assumptions
We consider nonlinear and non-convex, constrained op- timization problems involving both equality constraints g(x, y) = 0 and inequality constraints hx(x < 0, where y represents the decision variables and x represents the in- put variables, both as vectors. The objective is to minimize the cost function c(y). Mathematically. the optimization problem is as follows:
min c(y) s.t.
(la)
g(x, = 0
(1b)
h(x, y) < 0
(1c)
x is given (input vector)
(1d)
We assume that for all Vx E X. i.c., for any input vector in the set X (X' could be as simple as a hyper-rectangle) there exists at least one feasible solution to problem (1). Let D = {(x;, ) denote the labeled dataset, where y is the optimal solution obtained by solving optimization problem (1) for each x, Assuming that sampling the input vector x is inexpensive, we construct an unlabeled dataset De (x
1.1. Related Work
In recent years, Deep Neural Networks (DNNs) have been applied to solve various optimization problems with physics- based constraints, particularly in energy networks (Zamzam & Baker, 2020; Gupta et al., 2022; Donti et al., 2021; Singh
2

---

Semi-Supervised Optimization Proxies using Bayesian Neural Network
Our goal is to develop a BNN surrogate that provides an approximate optimal value of the decision variables y, for given test input vector E X. This work falls under the category of developing optimization proxies or surrogates, where the machine learning model serves as a direct for- ward mapping between the input and output variables of an optimization problem (see (Park& & Van Hentenryck, 2023)).
3. Semi-supervised Learning:
Sandwich BNN
We start by defining a suitable likelihood function for the unsupervised learning process. To that end, we augment the unlabeled data D" using the necessary feasibility conditions that the vector y must satisfy to be a solution of (I). We propose a function F(y, x) to measure the feasibility of a candidate solution y for a given input x. This function con- sists of two terms: one measuring the equality gap and the other measuring the one-sided inequality gap or violations. The relative emphasis on each term is determined by the parameters de and λr respectively, i.e.,
The paper proposes semi-supervised framework to solve this problem, wherein training alternates between a super- vised step-using labeled data 'D to minimize prediction error-and and an unsupervised step-using unlabeled data D" to enforce the feasibility of constraints in (1b) and (Ic). Both steps are implemented using a Bayesian Neural Network (BNN).
Fly, x) - > [19(xx))) +Xx |eLUT)(x,
(2)
Equality Gap
Inequelity Gap
2.2. Bayesian Neural Network
For any given feasible solution y. for the optimization prob-
lem in (1)?, we have F(y x) = 0 for the given input x E X. Furthermore, because of our assumption in Sec. 2.1 that the problem in (1) has at least one feasible solution, the minimum value F(: x) = 0 for any x E XX. Therefore, we can augment the unlabeled dataset D to create a labeled fea- sibility dataset, i.c., DJ = (x. x) = 1- Since input sampling is inexpensive, constructing this feasibility dataset DJ incurs additional computational cost. Similar to the supervised step in Sec. 2.2, we now define Gaussian likelihood for the unsupervised training step, with o as the noise variance for unsupervised learning and Xj E DJ, as
We consider a Bayesian Neural Network (BNN) denoted as In (x), where w represents all the weights and biases of the network. These weights are assigned an isotropic normal prior p(w) with covariance I²I, meaning that each weight is independently normally distributed with zero mean and variance o².
In the supervised training of the BNN, the goal is to com- pute the posterior distribution over the weights given the labeled data D = y). This posterior is expressed as p(w x) x p(y x, w) p(w). Here, p(y x, w) is the likelihood of the labeled data given the weights, and p(w) is the prior over the weights. The posterior distribution p(w| y) encapsulates the uncertainty about the weights after observing the data. Due to computational challenges in calculating the normalization constant of the posterior, approximate methods such as stochastic variational infer- ence (SVI) with the mean-field assumption are employed the posterior distribution estimation (see (Jospin et al., 2022)).
p(F x. w)
II [ (x))
x))
To obtain an optimization proxy, we parameterize the candidate solution (x) using a Deep Neural Network (DNN)-style architecture and employ a sandwich-style semi- supervised training for the BNN, as illustrated in Figure 1. The fundamental idea of this training method is to update the network weights and biases through multiple rounds of training in which each round alternatives between using the labeled dataset DD for prediction or cost optimality, and the augmented feasibility data set DJ for constraint feasi- bility. We let Sup and UnSup denote the inference steps in the BNN training using D and DJ, respectively. Both Sup and UnSup are performed for a fixed maximum time, with the total training time constrained to Tmmx- Finally, the prediction of the mean estimate Ey. and the predictive variance estimate Vy, is accomplished using an unbiased Monte Carlo estimator by sampling 500 weights from the final weight posterior pMp
For making predictions, the posterior predictive distribution is approximated as p(y" x², D) = Ep(w) D) (o (x') }) where x' is a test input vector, and the expectation is taken over the approximate posterior distribution of the weights. Moreover, we assume a Gaussian likelihood for output:
ply x) II N (y:1 fu()),
with of being a parameter in the SVI that controls the spread (noise variance) around the target values, and (x; y:) E D. Adapting this approach to update the BNN using the unsu- pervised data D" requires the definition of a suitable likeli- hood function, detailed in the next section, along with the remi-supervised framework to obtain the BNN surrogate.
3.1. Selection via Posterior (SvP)
In Bayesian Neural Network (BNN) literature, the stan- dard approach is to use the mean posterior prediction
Not necessarily optimal for (1).
3

---

Semi-Supervised Optimization Proxies using Bayesian Neural Network
pi = p(i0| y, x) x p(y| x, 1) p
Pw p
p(w
x)
p(F X, u)
p
piv
Piv
Piv
PW
PW
Y Byt, Vy*
Sup
UnSup
Sup
UnSup
Sup
Predict
o'O,o"I)
Ts
T
T
T max
Figure 1. Flowchart of the proposed sandwich-style BNN learning. The Sup block represents the supervised learning stage with labeled dataset D, and the UnSup block represents the unsupervised learning with the augmented feasihility dataset D'. Learning time upper limits are represented as Ts. Tu. and Tix for Sup, UnSup, and the complete semi-supervised BNN learning, respectively. At the prediction stage, denotes the posterior prediction matrix (PPM) for one test input sample, where each column represents the predicted output obtained via one weight sample from the posterior.
(fu for a test input x². This is similar to using the mean prediction of ensemble Deep Neural Networks (DNNs), However, unlike DNNs, BNNs can provide multi- ple predictions without additional training cost, as we can sample multiple weight instances from the posterior distribu- tion pW and construct the posterior prediction matrix (PPM) Y (see Figure 1 for details and Appendix B.3 for structure of the PPM.). We propose to use the PPM to improve the feasibility of the predicted output of the optimization proxy. Each column of the PPM represents one predicted output vector corresponding to a weight sample. We select the weight sample W that minimizes the maximum equality gap, defined as:
the error for any new input. Specifically, PCBs assert that the expected error will be within of the empirical errors computed from M out-of-sample inputs, with a high prob- ability (usually 0.95). Mathematically, we aim to provide guarantee on the error e = y y², where y' is the BNN prediction and y is the true value, as
1 M lesl
P Bllel|
Σ
<
> 1
(4)
where B[[e[] represents the expected absolute error, 1 - 5 is the confidence level, and s is the allowable prediction error.
W = arg min max (x(,, y,,)].
(3)
Ideally, we would like to evaluate our model on a large num- ber of samples since, as M + oo, the error bound - 0. However, increasing M leads to a higher requirement for labeled data, which defeats the purpose of training using low labeled data 4. To address this issue, confidence inequal- ities are commonly used to provide PCBs, with Hoeffding's ss inequality ((Hoeffding, 1994)) being one of the most widely used bounds. As stated in Appendix E, Hoeffding's ' inequal- ity assumes that the error is bounded (i.e., |es| & R for all i) and provides PCBs whose tightness is governed by M, with the relationship e = Rv log(2/6)) However, the Hoeffding's 2M
where Y. is the j-th column of the PPM, and (-(-)-) rep- resents the i-th equality constraint function. 3 The output prediction corresponding to the weight sample W* will have the minimum equality gap, and we term this process Selec- tion via Posterior (SvP). Note that the numerical operation in (3) can be performed in parallel and has minimal com- putational cost compared analytical projection methods in (Zamzam& & Baker, 2020) that focus on projecting the prediction onto (1c) to satisfy the inequality constraints in the problem (1). Note that it is an application motivated design choice to emphasize enforcement of the equality gap by using the SvP in (3). This can easily be adapted to account for inequality constraints without any significant computational overhead.
bound can often be too loose to be practically relevant.
To improve upon this, we propose using Bernstein's 's inequal- ity (see (Audibert et al., 2007)) as the concentration bound, which utilizes the total variance in error (TVE) information along with M, under the same bounded error assumption. The main challenge in using the Bernstein bound is obtain- ing the TVE without extensive out-of of-pal testing. One possible solution is to use the empirical Bernstein bound as in (Audibert et al., 2007; Mnih et al., 2008), which em- ploys the empirical variance of the error V.. obtained from the same M testing samples, and accounts for the error in variance estimation by modifying the theoretical Bernstein inequality. Mathematically, the PCB using the Empirical
4. Probabilistic Confidence Bounds
This section focuses on providing bounds on the expected absolute error of our method, i.c., testing error. We explore probabilistic confidence bounds (PCBs) for optimization proxies. The core concept of PCBs to first evaluate mod- els labeled testing dataset with M samples, compute the empirical mean error, and then probabilisticalyy bound
IIn a general setting of constrained optimization problems, there may be multiple equality and inequality constraints.
'Note that the total laheled data requirement is the sum of training and testing samples, i.c., N + M
4

---

Semi-SSpervised Optimization Proxies using Bayesian Neural Network
Bernstein inequality is
MPV information is readily available with BNNs but cannot be obtained from DNN-based optimization proxies.
2 log (3/6) E
BRlog (3/6)
In the next section, we perform a meta-study using different BNN models on different test cases to show the performance of our proposed learning architecture as well as demonstrate that hypothesis (5) indeed holds for the proposed optimiza- tion proxy learning problems.
+
M
M
as given in (Audibert et al., 2007; Mnih et al., 2008). Since the term under the square root depends on the empirical TVE Ve rather than R. the empirical Bernstein bound becomes tighter more quickly with increasing M if V. < R.
5. Numerical Results and Discussion
To further tighen the bound, we propose using the Theo- retical Bernstein bound (Sridharan, 2002) (Theorem E.3 in Appendix E) with the Mean Predictive Variance (MPV) as a proxy for the TVE Ve. The MPV is the mean of the predictive variance of testing samples, i.e. MPV M Ex=1 Vw [Y]] where variance Vw [Y:]] for the Athh test sample using entries of the posterior prediction matrix across columns, generated using posterior weights, for 2th output variable. In principle, the MPV captures the expected variance in the predictions due to the posterior distribution of BNN weights. We hypothesize that with a constant mul- tiplier a > 1,
We test the proposed method on the Alternating Current Optimal Power Flow (ACOPF) problem, essential for the economic operation of electrical power grids (Molzahn et al., 2019). Efficient ACOPF proxies can mitigate cli- mate change by enabling higher renewable energy integra- tion, improving system efficiency by minimizing losses and emissions, and enhancing grid resiliency against extreme weather conditions (Rolnick et al., 2022a). ACOPF is a constrained optimization problem with nonlinear equality constraints and double-sided inequality bounds. It aims to find the most cost-effective generator set points while satisfying demand and adhering to physical and engineer- ing constraints. The inputs are active and reactive power demands; outputs include generator settings, voltage magni- tudes, and phase angles at each bus. We adopt the standard ACOPF formulation (Babueinjaasarrooollaee et al., 2019; Park & Van Hentenryck, 2023; Coffrin et al., 2018) and benchmark our method using the open-source OPFDataset from Torch Geometric, which provides numerous solved ACOPF instances (see (Lovett et al., 2024)).
a: MPV > V. = EM [Vw + Vw [e]] > Viel. (5)
where EM and Ew denote expectations with respect M testing samples and posterior weight samples, respectively, and V [e] and [w[e] denote the variance of the error with respect to M testing samples and posterior weight samples, respectively. The equality in (5) follows from the law of total variance (Blitzstein & Hwang, 2019). Viel represents the variance of the absolute value of the error, which is lower than the variance of the error.
In our results, 'Gap%' denotes the average relative opti- mality gap compared to the objective values in the labeled testing instances. 'Max Eq." and 'Mean Eq.' represent the maximum and mean equality gaps over all equality con- straints, while 'Max Ineq.' and 'Mean Ineq.' indicate the same for inequality gaps in per unit, all averaged over test- ing instances. We compare our proposed method with the following state-of-thee the-art baseline supervised learning mod- els available in the literature using the same labeled dataset and training time constraints, utilizing AI4OPT's ML40PF package (AI4OPT, 2023) (network and hyper- ppprm-er details are in Appendix B):
Notice that the first term of the TVE, EM VV [e]]. is inde- pendent of the labeled testing dataset because the true output is constant with respect to posterior weight samples; thus, Vwle) [e) = Ww ly- - y²] = Vw [y']. Furthermore, from the definition of MPV, we have MPV = E④ [V w [e]]. As an ex- ample, if Vw [Ear[e] < MPV, our hypothesis (5) holds with a = 2. Consequently, we can use 2 x MPV as an upper bound for V. in the Theoretical Bernstein bound (see Theorem E.3 in Appendix E), which gives the error bound
4 x MPV (1/8) 2o (1/8)
M
3M
Naïve MAE and Naïve MSE (Supervised): Use 11- norm and 12-norm loss functions, respectively, to mea- sure differences between predicted and actual optimal solutions (Park & Van Hentenryck, 2023), incorporat- ing a bound repair layer with sigmoid function. The bound repair layer ensures that inequality constraints are always satisfied.
which is better than the Empirical Bernstein bounds. The hypothesis in (5) and the corresponding constant a can be computed by using application specific information or performing meta-study like in Section 5.
The strength of this approach is that we do not require labeled testing samples to calculate MPV, thus incurring no additional computational burden from generating labels. Also, note that this constitutes an advantage of BNNs since
MAE + Penalty, MSE + Penalty, and LD (Super- vised): Add penalty terms for constraint violations
5

---

Semi-Supervised Optimization Proxies using Bayesian Neural Network
Table I. Comparative performance results for the ACOPF Problem for 'case57' with 512 labeled training samples, 2048 unlabeled samples.
and Timx = 600 sec.
Method
Gap%
Max Eq.
Mean Eq.
Max Ineq.
Mean Ineq.
Sandwich BNN SvP (Ours)
0.928
0.027
0,006
0.000
0.000
Sandwich BNN (Ours)
0.964
0.045
0.005
0.000
0.000
Supervised BNN SvP (Ours)
3.195
0.083
0.011
0.000
0,000
Supervised BNN (Ours)
3.255
0.130
0.011
0.000
0.000
Sandwich DNN (Ours)
2.878
0.358
0,014
0.006
0.000
Naïve MAE
4.029
0,518
0.057
0.000
0.000
Naive MSE
3.297
0,541
0.075
0.000
0.000
MAE + Penalty
3,918
0.370
0,037
0.000
0.000
MSE+ Penalty
3.748
0.298
0.039
0.000
0,000
LD MAB
3.709
0.221
0.033
0.000
0.000
to the naïve MAE or MSE loss functions (Park & Van Hentenryck, 2023). The Lagrangian Duality (LD) method applies the 4--ormm as outlined in (Fioretto et al., 2020; Park& & Van Hentenryck, 2023), and also uses a bound repair layer with a sigmoid function.
posed Sandwich BNN trained with labeled and unla- beled data as discussed in Section 3. Unsupervised training utilizes four times the number of labeled data samples. The network uses ReLU activation and no bound repair layer.
We exclude self-supervised constrained optimization meth- ods like Primal-Dual Learning (PDL) (Park & Van Hen- tenryck, 2023) and DC3 (Donti et al., 2021) due to their significantly higher training times and computational de- mands, which violate the premise of this paper. For ample, PDL requires over 125 minutes of training time for the ACOPF problem on 118-node power network us- ing a Tesla RTX6000 GPU. Methods that require solving alternating current power flow to recover solutions (e.g.. (Za- mzam Baker, 2020)) are also excluded, as they result in high prediction times compared to DNN or BNN forward passes'. Furthermore, Graph Neural Network-based large models, such as (Piloto et al., 2024), require extensive train- ing datasets-for for instance, (Piloto et al., 2024) utilizes 270k training samples.
Importantly, we intentionally keep the BNN architecture un- optimized, constructing using four different sub- networks (one for each of the four ACOPF outputs: real power genera- tion, reactive power generation, voltage magnitude, and volt- age angle). Each sub-network has two hidden layers with the number of hidden neurons equal to 2 X input size. In the Sup stage of the Sandwich BNN, both weights and biases are updated, whereas in the UnSup stage, only the weights are modified via SVI. Best model out of five random trials is selected.
Tables and 2 present the comparative performance of different methods for solving the ACOPF problem on the 'case57' and 'case118" test cases, containing 57 and 118 nodes, respectively. For 'case57', the Sandwich BNN SvP method achieves the best Max Eq. performance (0.027), outperforming all other methods, including the standard Sandwich BNN, without compromising other metrics. All the methods proposed in this paper outperform the best DNN results, typically achieved with the LD+MAE model (last row in the tables). Similar trends are observed for the 'casel18 11'' as well. 6 It is important to contextualize the significance of these numerical improvements. In ACOPF problems, cost values are in USD, with the mean cost for 'casel 18 being $97,000 or 9.7 in the per-unit system. There- fore, a 1% Gap corresponds to an expected difference of $970 across the testing instances. A 'Max Eq.' value of 0.08 implies a maximum expected power imbalance of 8.0 Megawatts among all 118 nodes of 'casel 18". Thus, re- ducing the Max Eq. from 1.284 with the LD+MAE model to 0.089 with our Sandwich BNN SyP model represents significant improvement.
To demonstrate the effectiveness of the proposed BNN learn- ing methods, we conduct simulation studies on both our models and the ML40PF models using an MI Max CPU with 32 GB RAM, without any GPU. This setup highlights performance improvements due to the learning mechanism rather than computational power. We propose two classes of different models as:
Supervised BNN and Supervised BNN SyP: Stan- dard BNN learning with labeled data, utilizing mean prediction and Selection via Posterior (SvP), respec- tively. The network uses ReLU activation and no bound repair layer.
Sandwich BNN and Sandwich BNN SvP: The pro-
"see prediction time studies in (Donti et al., 2021) which sug- gest 10 times higher prediction time with power flow based projec- tions (0.080 sec. compared to 0.001 sec. for DNN and 0.003 see. per testing instance for proposed BNNs).
"ee Appendix C for similar tabular results and discussion of trends on larger test instances.
6

---

Semi-Supervised Optimization Proxies using Bayesian Neural Network
Table Comparative performance results for the ACOPF Problem for 'casel18' with 512 labeled training samples, 2048 unlabeled samples, and mix 600 sec.
Method
Gap%
Max Eq.
Mean Eq.
Max Ineq.
Mean Ineq.
Sandwich BNN SvP (Ours)
1.484
0.089
0.018
0.008
0.000
Sandwich BNN (Ours)
1,485
0.100
0.016
0.008
0.000
Supervised BNN SvP (Ours)
1.568
0.147
0.022
0.013
0,000
Supervised BNN (Ours)
1.567
0.205
0.020
0.013
0.000
Sandwich DNN (Ours)
3.298
0.585
0.034
0.158
0.000
Naive MAE
1.638
2.166
0.187
0.000
0,000
Naive MSE
1.622
3.780
0.242
0.000
0.000
MAE + Penalty
1.577
1.463
0.102
0.000
0.000
MSE Penalty
1.563
2.637
0.125
0.000
0.000
LD MAE
1.565
1.284
0.083
0.000
0.000
Sandwich BNN SvP Supervised BNN SvP
MAE + Penalty
MSE + Penalty
LD + MAE
60
18
Max Ineq.
0.9
Gap%
40
Max Eq.
12
0.6
20
0.3
0
0.0
102
10a
10a
10a
10a
10a
System Size
System Size
System Size
Figure 2. Growth trajectories of performance metrics for ACOPF across system sizes for different methods. Detailed results for "case500"" and 'case2000'' given with Table 8 and Table 9 respectively, in Appendix C.
Sandwich DNN in these tables is a DNN with the same network architecture as the BNN, trained under the same time constraints. This network is trained by alternating between minimizing supervised MSE loss and and unsupervised feasibility loss as in (2). In both these rounds, an 4 weight regularization is added to prevent over- over-fitting. We see that this DNN model shows performance comparable to other DNN approaches. This a strong indication that the superior performance of the sandwich BNN model is largely due to the superior performance of the BNN approach in the low data regime.
We perform robustness study on 'casel18', by running five different trials of 10-min and 15-min training time with 512, 1024, and 5048 supervised training samples keeping 2048 unsupervised samples constant across models and tri- als (See Appendix C for detailed graphs and tables). The results clearly shows that increasing training time and num- ber of supervised samples decreases decreases %Gap, Max Eq, and Max Ineq., consistently, and keep Mean Eq. one order of magnitude lower and Mean Ineq. order 10 10-5, The Sandwich BNN SvP model, provides 1.50% Gap. 0.094 Max Eq. and 0.014 Max Ineq. with 512 supervised training samples and 10-Min training time,, on average over five trials. The same model provides 1.46% Gap, 0.080 Max Eq. and 0.011 as Max Ineq. with 2048 supervised train- ing samples and 1 15-Min Training time. This implies that five minute increase in training time with four times more training samples worth 2.66% reduction in optimality error (%Gap) 17.5% reduction in equality constraint feasibility error (Max Eq.) and more than 27% reduction in inequality constraint feasibility error (Max Ineq.).
Figure 2 illustrates the growth of various metrics with in- creasing system size while keeping training resources con- stant. The proposed BNN methods exhibit significantly lower scaling in the expected maximum equality gap. Al- though Sandwich BNN SvP shows slightly higher scaling inequality gaps, the relative improvement in Max Eq. far out- weighs these minor drawbacks. Notably, for "case500' (see Table 8 in the Appendix C), the expected maximum power imbalance is below 0.5% of the mean real power demand (1.7 x 1 MW) and power grids already are equipped with spinning reserves (see (Ela et al., 201 1)) that have reserve capacity to handle these imbalances. This a significant im- provement over the DNN models which have much higher 'Mean Eq.' values. Morcover, the 'Max Ineq.' growth could be easily suppressed by incorporating bound repair layers, as used in DNN models in ML40PF (AI4OPT, 2023).
This detailed study also highlight that smaller training times are not good enough to extract complete information from larger training datasets various models with 2048 super- vised samples perform better with 15-min training come compared to 10-min training time. For instance, Max Eq. is 6. 188 lesser for 2048 samples when we increase training time from 10-Min to 15-Min, and %Gap values reduces

---

Semi-Supervised Optimization Proxies using Bayesian Neural Network
Sandwich BNN
Supervised BNN
1e=3
1e-4
1e-3
MPV (log)
MPV x (log)
4.5
MPV 2 (log)
1.6
2-
3.0
3.0
0.0
1.6
Total Variance in Error (log)
Total Variance in Error (log)
1c-4
Total Variance in Error (log)
Figure 3. Empirical study comparing total variance in error Ve with 2 x MPV across different cases of ACOPF and the proposed learning mechanisms.
Voltage Magnitude
Voltage Angle
Errors Rad. (log)
Errors PU (log)
10 2
10
10.3
10-4
10-4
10-5
0
10
20
30
40
50
10
20
30
40
50
Bus Index
Bus Index
Empirical Expected Error
Theoretical Bernsteins MPV)
Empirical Bernsteins
Hoeffdings
Voltage Magnitude
Voltage Angle
Errors PU (log)
10-1
Errors Rad. (log)
10-1
10-3
10-3
10
10-5
20
60
100
120
120
Bus Index
Bus Index
Figure Comparison voltage magnitude and voltage angle error bounds (in logarithmic scale) across indices 'case57" (top row) and "casel 18' (bottom row). The plot illustrates that PCBs using thcoretical Bernstein bounds with 2 x MPV from hypothesis (5) tightest among all PCBs. We consider 5 0.95 and 1000 out-of- sample testing data points i.c. 1000.
from 1.52 to 1.46, on average. Lastly, we also observe that variability in training quality reduces with increasing number supervised samples and training time.
Compared to Hoeffding bound (0.064 pu) or the empirical Bernstein bound (0.018 pu), the Bernstein bound (0.010 pu) is far tighter and more practical, highlighting the benefits of BNNs for optimization proxies. The error bounds for the "case500' provided in Fig. the Appendix C. Fi- nally, note that both Hoeffding' 's and empirical Bernstein bounds can also be obtained by testing DNN models across M samples
Next, we present results Proabilistic Confidence bounds, described in Section 4. Figure 3 shows that 2 MPV consis- tently serves empirical upper bound for total variance in error, validating its robustness across models and sys- tem configurations" In Figure 4, the theoretical Bernstein bounds using V. = 2 MPV provide tight, practical bounds, whereas Hoeffding's 's bounds are overly conservative and example, 'case 18", the Bernstein bound ensures a probabilistic guarantee on voltage constraint satisfaction, such that predicted voltage between 0.91-1.09 pu guarantees violations within the ACOPF limits of 1.90-1.10 pu, i.e., the maximum value of the Bernstein bound on the error is 0.010 pu across all nodes.
6. Conclusions
not useful for grid operations.
In conclusion, this paper introduces a semi-supervised Bayesian Neural Network (BNN) approach address the challenges of high labeled data requirements and limited training time in learning input-to-output maps for con-
*A form of MPV can be obtained via Ensemble DNNs (Ganaic et al.. 2022), however, will lead to very high computational requirement compared to the BNN.
Total variance in enror is assumed to stabilizing with 1000 testing samples (see Figure in Appendix C).
8

---

Semi-SSpervised Optimization Proxies using Bayesian Neural Network
strained optimization problems. The proposed Sandwich BNN method incorporates unlabeled data through input data augmentation, ensuring constraint feasibility without rely- ing on a large number of labeled instances. We provide tight confidence bounds by utilizing Bernstein's inequality, en- hancing the method's practical applicability. Results show that BNNs outperform DNNs in low-data, low-compute settings, and the Sandwich BNN more effectively enforces feasibility without additional computational costs compared to supervised BNNs.
Coffrin, C., Bent, R., Sundar, K., Ng. Y., and Lubin, M. Powermodels.i: An open-source framework for explor- ing power flow formulations. In 2018 Power Systems Computation Conference (PSCC), pp. 1-8, June 2018.
Donti, P.,, Rolnick, D., and Kolter, J. Z. Dc3: A learning method for optimization with hard constraints. In Inter- national Conference on Learning Representations, 2021.
Ela, E., Milligan, M., and Kirby, B. Operating reserves and variable generation. Technical report, National Renew- able Energy (NREL,,,), Golden, CO (United States), 2011.
Reproducibility
We use the open-source ACOPF datasets, provided with Torch geometric (Lovett et al., 2024), to train and test our models as well as standard DNN models. Furthermore, the beginning of Section 5 provides details of the DNN mod- els used to compare the performance of the proposed meth- ods. These models are available in AI4OPT's open-source ML40PF package (AI4OPT, 2023). Hyper-parameter details for these models and the proposed methods are provided in Appendix B. The code used in this paper can be found at https:/ htt::/ggitttuu www.coo//aaaae Additional experimental results be found in the Sup- plementary Information.
Fajemisin, A. O., Maragno, D., and den Hertog, D. Op- timization with constraint learning: A framework and survey. Earopean Journal of Operational Research, 314 (1):1-14, 2024.
Fioretto, F., Mak, T. W., and Van Hentenryck, P. Predicting optimal power flows: Combining deep learning and lagrangian dual methods. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 630- 637, 2020.
Ganaie, M. A., Hu, M., Malik, A. K., Tanveer, M., and Suganthan, P. N. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115: 105151, 2022.
Impact Statement
Gupta, S., Misra, S., Deka, D., and Kekatos, V. Dnn-based policies for stochastic ACOPF. Electric Power Systems Research, 2 213:108563, 2022.
This paper presents work whose goal is to advance the field of Machine Learning. In particular, enhancing the solution of optimization problems will result in more efficient re- source utilization, helping industries lower costs and reduce environmental impact. Additionally, refining the ACOPF solution pipeline will play crucial role in addressing cli- mate change by optimizing renewable energy usage and ensuring the reliable operation of the power grid (Rolnick et al., 2022b).
Hoeffding, W. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffd- ing, pp. 406, 1994.
Ibrahim, M. S., Dong, W., and Yang, Q. Machine learning driven smart electric power systems: Current trends and new perspectives. Applied Energy, 272:115237, 2020.
References
Jospin, L. V., Laga, H., Boussaid, F., Buntine, W., and Bennamoun, M. Hands-on bayesian neural networks-a tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2):29 2022. doi: 10. 109/
A14OPT. MI4opf: machine learning library for optimal power flow problems. https://gt sttps://github... htgggituub.com com/AI40PT/ ML4DPF, 2023.
MC1.2022.3155322.
Audibert, J.-Y., Munos, R., and Szepesvári, C. Tuning bandit algorithms in stochastic environments. In Inter- national conference on algorithmic learning theory, pp. 150-165. Springer, 2007.
Khadivi, M., Charter, T., Yaghoubi, M., Jalayer, M., Ahang, M., Shojaeinasab, A., and Najjaran, H. Deep reinforce- ment learning for machine scheduling: Methodology, the state-of-the-art, and future directions. Computers & In- dustrial Engineering, pp. 110856, 2025.
Babaeingjadsarooklleee, S. et al. The power grid library for benchmarking AC optimal power flow algorithms. arXiv preprint arXiv:1908 02788, 2019.
Kotary, J., Fioretto, F., van Hentenryck, P., and Wilder, B. End-to-end constrained optimization learning: A survey, In 30th International Joint Conference on Artificial Intel- ligence, IJCAI 2021, pp. 4475 4482. International Joint Conferences on Artificial Intelligence, 2021.
Blitzstein, K. and Hwang, J. Introduction to probability. Chapman and Hall/CRC, 2019.
9

---

Semi-Supervised Optimization Proxies using Bayesian Neural Network
Lovett, S., Zgubic, M., Liguori, S., Madijhhurrmm S., Tom- linson, H., Elster, S., Apps, C., Witherspoon, S., and Piloto, L. Opfdata: Large-scale datasets for optimal power flow with topological perturbations. arXiv preprint arXiv:2406.07234, 2024.
Sridharan, K. A gentle introduction to concentration in- equalities. Dept. Comput. Sci., Cornell Univ., Tech. Rep. pp. 8, 2002.
Yang. W., Lorch, L.,, Graule, M., Lakkaraju, H., and Doshi-
Velez, Incorporating interpretable output constraints in bayesian neural networks. Advances in Neural Informa- tion Processing Systems, 33:12721 2020.
Mnih, V., Szepesvári, C., and Audibert, J.-Y. Empirical bern- stein stopping. In Proceedings of the 25th international conference on Machine learning, pp. 672-679, 2008.
Yang, X., Song, Z., King. I., and Xu, 7. A survey on deep semi-supervised learning. IEEE Transactions on Knowl- edge and Data Engineering, 35(9))8934-8954 2022.
Molzahn, D. K., Hiskens, A., et al. A survey of relax- ations and approximations of the power flow equations. Foundations and Trends@ in Electric Energy Systems, 4 (1-2):1-221, 2019.
Zamzam, A. S. and Baker, K. Learning optimal solutions for extremely fast ac optimal power flow. In 2020 IEEE international conference on communications, control, and computing technologies for smart grids (SmartGrid-
2019.
Papamarkou, T., Skoularidou, M., Palla, K., Aitchison, L., Arbel, J., Dunson, D., Filippone, M., et al. Position: Bayesian deep learning is needed in the age of large-scale AL. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 39556-399999 39586. PMLR, 21-27 Jul 2024.
Comm). pp. 1-6. IEEE, 2020.
Park, S. and Van Hentenryck, P. Self-supervised primal- dual learning for constrained optimization. Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 4052 4060, 2023.
Piloto, L., Liguori, S., Madjiheurem, S., Zgubic, M.,, Lovett, S., Tomlinson, H., Elster, S., Apps, C., and Wither- spoon, S. CANOS: A fast and scalable neural AC- OPF solver robust to n-l perturbations. arXiv preprint arXiv:2403. 17660, 2024.
Rolnick, D., Donti, P. L., Kaack, L. H., Kochanski, K., et al. Tackling climate change with machine learning. ACM Comput. Surv, 55(2), February 2022a. ISSN 0360-0300. doi: 10. 00.1145/3485128.
Rolnick, D., Donti, L., Kaack, L. H., Kochanski, K., et al. Tackling climate change with machine learning. ACM Computing Surveys (CSUR), 55(2):1-99)) 2022b.
Sam, D., Pukdee, R.. Jeong, D. P., Byun, Y., and Kolter, J. Z. Bayesian neural networks with domain knowledge priors. arXiv preprint arXiv:2402.13410 2024.
Sharma, M., Rainforth, T., Teh, Y. W., and Fortuin, V. Incor- porating unlabelled data into bayesian neural networks. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Singh, M. K., Kekatos, V., and Giannakis, G. B. Learning to solve the AC-OPF using sensitivity-infoomee informed deep neural networks. IEEE Transactions on Power Systems, 37(4): 2833-2846, 2021.
10

---

Semi-Supervised Optimization Proxies using Bayesian Neural Network
Supplementary Information
Optimization Proxies using Limited Labeled Data and Training Time A Semi-Supervised Bayesian Neural Network Approach
A. ACOPF Problem: Modeling and Dataset
The alternating current optimal power flow (ACOPF) problem is essential for power grid operations and planning across various time scales. It determines generator set set-points for real and reactive power that minimize generation costs while meeting power demand and satisfying physical and operational constraints on output variables. We follow the ACOPF model given in PowerModelss (Coffrin et al., 2018), and take the dataset from Torch geometric ((Lovett et al., 2024)).
Table 3. Sets for ACOPF
N: buses
G: generators, generators at bus (G;)
E: branches (forward and reverse orientation, ER)
S: shunts, shunts bus (S,)
L: loads, loads bus (L₃)
R: reference buses
Table 4. Data for ACOPF Problem
Symbol Sy, S. EG
Description
Generator complex power bounds.
oi ci &i G
Generator cost components.
of o V E
Voltage bounds.
Load complex power consumption.
Y Y
Bus shunt admittance.
Yg, Y., (i,) E
Branch x-section parameters.
Ty V(i,))
Branch complex transformation ratio.
V(.5) E ((LD)) 04. 29.022(1.D)) E
Branch apparent power limit.
Branch current limit.
Branch voltage angle difference bounds.
Variables
Symbol
Description
S Vk E G Vi E N (IVALZ0) Sij V(i.j) E E ER
Generator complex power dispatch.
Bus complex voltage (Magnitude/Anglo
Branch complex power flow.
Table Objective Function Constraints
Description
Equation
Objective
min (Ere (c5(S))) + st cf))
Reference bus voltage angle
2V. Vr E
Generator power bounds
S S S VkEG
Bus voltage bounds
5 vi ViEN
Power balance Vi E N
Exec, S -Eket, St E V/ EaseEBER S4,
Branch power flow (( E E
Sij YG/h/+Y2VVV /Ti] is +YV.V,
+
Branch apparent power limits
|Sas S o4 V(i.)) E ERER
Branch current limits
[Sy S ITA V(i,) EU
Branch angle difference bounds
0 s (VVV3) s 0 v(i,)) E E

---

Semi-Supervised Optimization Proxies using Bayesian Neural Network
B. Experimental Setting Details
B.1. Architectures
Supervised BNN and Sandwich BNN: For ACOPF problem we use real and reactive power demands as input while predicting all decision and state variables using separate networks
Input real and reactive power demands: Two times the number of nodes having non-zero load
Output real and reactive power generation setpoints, voltage magnitude and voltage angle at each node: Two times the number of generators + Two times the number of nodes
B.2. Plyper-paramettn
B.2.1. ML40PF:
The optimization performed using the Adam optimizer with a learning rate of 1 10-4 (taken from (Park & Van Hentenryck, 2023)). All networks have two hidden layers, each with size of Number of outputs. For hidden layers, ReLU activation function selected while, the bound repair mechanism handled using Sigmoid function (Park Van Hentenryck, 2023).
config:
comalty-confffl Multiplier of 10-2, with no excluded keys.
and a kick-in value of O. The LDF update frequency is set to 1, and no keys are excluded 1df config: Step size x 10-2, and from the updates.
All other hyperparameters are default ML40PF values (AI4OPT, 2023).
B.2.2. PROPOSED BNNS
For all simulations, maximum training time 10 min., Toaa 600 seconds, per round time T, is 200 seconds with 40-60 split between Sup and UnSup models (T, = 80 amd T, = 120 seconds). For VI, we use MeanFieldELa loss function from Numpyro and Adam optimizer with a initial learning rate rate of x 10-3, and decay rate of 1 x 10-4, both of which are selected via grid search.
We use decay schedule as initial learning rate/( + decay rate step) within a Sup or UnSup model, while simple step decay among different rounds of Sandwich BNN. Learning done with single batch for all models and each of the BNN weight and bias is parameterized using mean and variance parameters of a Gaussian distribution, with mean field assumption i.e. independent from each other. The prior for each weight and bias has zero mean and 10-2 variance while likelihood noise variance is initialized with 10 mean and 10-6 variance for Sup and fixed at 10- for UnSup.
B.3. Structure of posterior prediction matrix (PPM)
Y1
VIH :
O Number of variables
YVariable, Sample
Number of posterior samples
yoo
YOH
12

---

Semi-Supervised Optimization Proxies using Bayesian Neural Network
C. Robustness Study on 118-System
Avg. (Sandwich)
Avg. (Supervised)
Error Bars (Sandwich)
Error Bars (Supervised)
% Gap
Max Eq.
Max Ineq.
1.6055
0.1829
0.02153
% Gap
1.5479
Max Eq.
0.1464
Max Ineq.
0.01477
1.4904
0.1099
0,00800
1.4328
0.0734
0.00124
512
1024
2048
512
1024
2048
512
1024
2048
# #Samples (N)
#Samples (N)
#Samples (N)
Figure 5. Summary of Results with more variations in supervised data, for higher training (15 min) for sandwich and supervised models
with SvP. Note that this computation was done Max Mini M4 with 24GB RAM.
Table 6. Comparison of Average Values (Over Five Trials) for BNN Models: 10/15 min, Sandwich SyP/Supervised SvP. Note that this computation was done Max Mini M4 with 24GB RAM.
Sandwich BNN SvP, 15-Min
Supervised BNN SvP, 15-Min
N
%Cost
Max Eq
Mean Eq
Max Ineq
Mcan Ineq
%Cost
Max Eq
Mcan Eq
Max Ineq
Mcan Incq
512
1,5305
0.1041
0.01998
0.01285
0,000046
1,5904
0.1755
0.02403
0,01446
0,000060
1024
1,5097
0.08948
0.01872
0.01442
0.000049
1,5615
0.
0.02075
0.01892
0.000075
2048
1.4691
0.08005
0.01709
0.01 170
0.000042
1,5534
0.08776
0.01839
0.01906
0.000073
Sandwich BNN SvP, 10-Min
Supervised BNN SvP, 10-Min
N
%Cost
Max Eq
Mean Eq
Max Ineq
Mean Ineq
%Cost
Max Eq
Mean Eq
Max Ineq
Mean Ineq
512
1.5071
0.09438
0.01912
0.01453
0.000054
1,5807
0.1644
0.02323
0.01524
0.000061
1024
1.5194
0.08693
0,01865
0.01415
0.000047
1.5562
0.1108
0.02025
0,01679
0,000065
2048
1,5297
0.08532
0.01855
0.01288
0.000042
1,5387
0.08686
0.01843
0,01912
0.000072
Table 7. Comparison of Average Values (Over Five Trials) for BNN models: 10/15-min, Sandwichh/Sppervieed Note that this computation was done on a Max Mini M4 with 24GB RAM.
Sandwich BNN, 15 Min
Supervised BNN, 15 Min
N
Cost
Max Fq
Mean Fq
Max Ineq
Mean Ineq
%Coost
Max Eq
Mean Eq
Max Ineq
Mean Ineq
512
1.5298
0.1349
0.01810
0.01255
0.000045
1.5905
0.2281
0.02318
0.01425
0.000060 0.000074
1024
1.5074
0.1106
0.01661
0,01407
0.000048
1.5594
0.1609
0.01957
0.01868
2048
1.4687
0.09156
0.01474
0.01135
0.000041
1.5540
0.1114
0.01690
0.01890
0.000072
Sandwich BNN,
10-Min
Supervised BNN,
10-Min
N
&Cost
Max Eq
Mean Eq
Max Ineq
Mean Ineq
%Cost
Max Eq
Mean Eq
Max Ineq
Mean Ineq
512
1.5054
0.1194
0,01710
0,01426
0.000053
1.5780
0.2204
0,02215
0,01502
0.000060
1024
1,5188
0.1064
0.01654
0.01376
0.000046
1.5573
0.1499
0,01888
0,01652
0.000064
2048
1.5284
0.09725
0.01623
0,01251
0.000041
1.5377
0.1081
0.01664
0.01887
0.000071
Figure 5 shows results of 5-Trials of proposed method of learning for three key metrics-99 % Gap. Max Eq., and Max across different sample sizes (N = 512, 1024, 2048) for "case118 on a Mac Mini M4 with 24GB RAM. The red dashed lines with circular markers represent the mean values obtained from the sandwich training approach, while the blue solid lines with square markers correspond to the mean values from the supervised training approach. Error bars depict the minimum and maximum observed values for each method, providing insight into the variability of results. From the figure, we observe that as the number of samples increases, both methods exhibit trend of increasing Max Eq. and Max Ineq. values, while % Gap remains relatively stable.
The supervised approach generally achieves lower variability (shorter error bars), indicating more consistent perfor- mance. The sandwich method, while slightly less stable, maintains comparable mean values across the three metrics. These results suggest that supervised learning provides more predictable outcomes, whereas the sandwich method may introduce greater flexibility with minor variations in performance. Detailed results and and graphs are given in Table 7, Table 6, Figure to Figure 15.
13

---

Semi-Supervised Optimization Proxies using Bayesian Neural Network
Results in Figure 8 15 of five trials with identical hyperparameters demonstrate consistent improvement across all parameters except for max inequality. Max inequality remains challenging suppress further, it is already order of magnitude better than max equality. Also the spread of error decreases with more samples indicating robust learning performance with more data. Note that these computations was done on Mac Mini M4 with 24GB RAM.
C.1. Larger System Results
Table 8. Comparative performance results for the ACOPF Problem for casc500 with 512 labeled training samples, 2048 unlabeled samples
and Tma = 600 sec.
Method
Gap%
Max Eq.
Mean Eq.
Max Ineq.
Mean Ineq.
Sandwich BNN SvP (Ours)
2.009
0.770
0.066
0.190
0.000
Sandwich BNN (Ours)
2.002
0.781
0.056
0.191
0,000
Supervised BNN SvP (Ours)
1.191
2.204
0.088
0.141
0.000
Supervised BNN (Ours)
1.191
2.401
0.072
0.140
0.000
Sandwich DNN (Ours)
1.782
13,800
0,268
0.2707
0.000
Naive MAE
1.208
20.818
0.905
0.000
0.000
Naïve MSE
1.201
24.089
1.031
0.000
0.000
MAE Penalty
1.205
11.833
0.580
0.000
0,000
MSE + Penalty
1.215
10.314
0.475
0.000
0.000
LD + MAE
1.279
11.166
0.532
0.000
0.000
Table Comparative performance results for the ACOPF Problem for case2000 with 512 labeled training samples, 2048 unlabeled samples and Tmaz = 600 sec.
Method
Gap
Max Eq.
Mcan Eq.
Max Ineq.
Mcan Ineq.
Sandwich BNN SyP (Ours)
0.514
5.114
0.324
0.196
0.000
Sandwich BNN (Ours)
0.503
5.409
0,262
0.187
0.000
Supervised BNN SvP (Ours)
0.461
4.107
0.238
0,917
0.000
Supervised BNN (Ours)
0,451
4.225
0.193
0.922
0,000
Sandwich DNN (Ours)
54,505
27.747
1.659
1.228
0.339
Naive MAE
56.365
43.529
4.392
0,000
0.000
Naive MSE
56.366
43.085
4.261
0.000
0,000
MAE+ + Penalty
56.349
19.591
1.055
0.000
0.000
MSE Penalty
56.377
12.592
0.682
0.000
0.000
LD + MAE
57.257
18,870
0.647
0.000
0.000
Here, we present the complete results analogous to Tables 1 and 2 for the larger test cases: 'case500' and 'case2000" with 500 and 2000 nodes, respectively, in Tables 8 and 9. In general, it is clear from the tables that the approaches presented in this paper outperform the sfate-of-the-ar art DNN-based approaches when limited training time and compute resources are provided. Between the supervised and the sandwich BNN models, appears that there no clear winner. Table indicates that the supervised BNN outperforms the sandwich BNN model on the Max. Ineq, gap metric for the 'case500' whereas the trend is reversed for the case2000' in Table 9. This minor variation is attributed the fact that the maximum training time of Tmax = 600 sec. is insufficient for both cases, and with more training time, these trends should look similar to the ones in Tables and 2.
Another critical observation for both cases is that the 'Max. Eq." value is substantially high even for the proposed best model (in Tables 8 and 9, respectively). However, is clear that the proposed BNN proxies are better than standard DNN models with an order-of-magnitude difference. On the surface, readers may dismiss the efficacy of the proposed BNN models due to these large values. Still, these values have to be examined in conjunction with the error bounds on the voltage magnitude and voltage angles in Fig. 7. Examined together, the BNN model predictions have very low errors for the voltage magnitude and angles, and it is very much possible to develop computationally inexpensive projection algorithm that projects this prediction onto the feasibility set of the ACOPF problem on lines of (Zamzam & Baker, 2020). This procedure would further reduce the Max. Eq. values and makes the projected solution usable. This is a minor detail regularly tackled in power grid context and is not dealt with extensively in this paper.
Finally, Figure shows the stabilization of the mean errors and the MPV with varying testing samples and posterior
14

---

Semi-Supervise Optimization Proxies using Bayesian Neural Network
Voltage Magnitude
Real Power Generation
Voltage Magnitude
Real Power Generation
Voltage Angle
Reactive Power Generation
Voltage Angle
Reactive Power Generation
1c-2
1c-6
16-6
1e-6
Meen Error
Total Var. Error
4.5
2.4
Todal Var. Eiror
2.4
3.0
MPV
1.6
1.6
0.8
0.8
1.5
0.0
0.0
#Testing Samples
Testing Samples
0 Posterior Samples
Pooterior Samples
1e-4
1e-4
1e-4
1.00
Total Var ETOr
Total Var. Error
2.4
0.75
MPP
1.6
0.50
2.4
0.8
0.8
2.1
0.0
0.0
#Tssting Samples
Testinn Samples
#Posterior Sampies
#Posterior Samples
Figure 6. Convergence of mean error, MPV, total variance in error with respect to number of testing samples, and number of posterior samples.
Voltage Magnitude
Voltage Angle
10
Errors Rad. (log)
10-1
Errors PU (log)
v feet
10-3
10-3
10
10
100
200
300
400
500
100
200
300
400
500
Bus Index
Bus Index
Empirical Expected Error
Theoretical Bernsteins (2 x MPV)
Empirical Bernsteins
Hoeffdings
Figure 7. Comparison of voltage magnitude and voltage angle bounds (in logarithmic scale) across bus indices for case500. The plot illustrates that PCBB using theoretical Bernstein bounds with 2 x MPV from hypothesis (5) are tightest among all PCBs. We consider 8 0.95 and 1000 out-of-sample testing data points i.e. M 1000.
samples, respectively. These plots validate the number of testing and posterior samples chosen for generating the results presented in Section
D. Performance on a non-OPF dataset
The semi-supervised training approach described in this work can be used to train optimization proxies for any constrained optimization problem over continuous variables. In this section we look at the performance our method on following non-convex optimization problem described in Donti et.al. (Donti et al., 2021).
1 /y "u + sin(y), s.t. Ay = Gy S h.
min VER"
(6)
The sinusoid function in the objective makes this non-convex problem. We used the code provided by Donti et.al. to generate two sets of randomized instances of this problem. The comparative performance of the sandwiched training method against purely supervised BNN training is given in Table 10
15

---

Semi-Supervised Optimization Proxies using Bayesian Neural Network
Model
(nV, nÊq, nInEq)
Max-Eq violation
Max-InEq violation
Gap%
Time
BNN- -uN-Suuere
(70,20,50)
0.484
0.170
2.6
800s
BNN-Sandwich
(70,20,50)
0.298
0.109
7.1
800s
BNN-Super
(20, 20,20
0.266
0. 101
1.4
400s
BNN-Sandwich (20,10,20)
0.114
0.000
0.07
400s
Table 10. nV, nBq and nlnEq respectively the number of variables, equality constraints and inequality constraints in the optimization problem. For these experiments we use 2º² samples in the supervised stage and 2 samples in the unsupervised stage. Results above are evaluated 100 testing instances and averaged
three different random instantiations of the training procedure. These show the advantages of using the sandwiched approach for training BNN model the lov-data/lime-contrainee regime for this problem. The hardware used here the Google Collab instance with Intel(R) (eon(R) CPU @ 2.20GHz.
E. Concentration Bounds
This section presents three bounds used in Section 4. Here, X represents generic random variable and is not related to the optimization proxy variables. Since these are well-known inequalities, we omit the proofs, which can be found the respective references.
Theorem E.1 (Hoeffding's s). (Hoeffding, 1994) Let X .MM ii.d random variables and suppose that [x]] & R with expectation [X)), and let XM 37 Ein Xi. Then, with probability at least 1 8,
1og(2/8)
|M B(X,) R
2M
Theorem E.2 (Empirical Bernstein). (Audibert al., 2007; Mnih al., 2008) Let X,,..., NM be i..dd and suppose that [x, & R, expectation B(X;) and let XM Σ i== X, With probability least 1 5,
M
2 log(3/0)
2R log(3/5)
AM - B(X.) &
M
M
where, V (1/M) - 2 is empirical variance.
Theorem E.3 (Bernstein). (Sridharan, 2002) Let X1...., XM l be i..dd. and suppose that [X,l < mean E(X;) and Var(X,). With probability at least 1 6.
2V log(1/0)
2R log(1/5)
XM E(X,)
M
3M
Table 11. Error bound in PCBs, provided by different concentration inequalities.
Hoeffding's
Empirical Bernstein
Bernstein
Ry 1og(2/8)
24. log (3/5)
3R log(/8))
2 (og(1/8)
28 log(1/8)
16

---

Semi-Supervised Optimization Proxies using Bayesian Neural Network
Gap% (SvP)
Max (5.()))
Mean Eq./5vP)
Max (SvP)
0.100
Mean
0.0195
0.020
1.52
0.095
Min
M
0.0190
0.015
1.50
0.090
1.48
512
1024
2048
512
1024
2048
512
1024
2048
512
1024
2048
# #Samples
#Samples
# #Samples
#Samples
Figure 8. Performance metrics for Sandwich BNN SvP model with 10-minute training.
Gap% (SvP)
Max Eq.(SvP)
Mean Eq.(SVP)
Max Ineq./SvP)
0.16
Mean Min
1.58
0.022
0.14
0.018
1.56
Max
0.12
0.020
1.54
0.10
512
1024
2048
512
1024
2048
512
1024
2048
512
1024
2048
#Samples
#Samples
#Samples
#Samples
Figure 9, Performance metrics for Supervised BNN SvP model with 10-minute training.
Gap%
Max
Mean
Max Ineq.
1.54
0,0180
Mean Min M ax
0.13
0.0175
0.12
0.0170
1.50
0.11
0.0165
1.48
0.10
512
1024
2048
512
1024
2048
512
1024
2048
512
1024
2048
# #Samples
#Samples
#Samples
#Samples
Figure 10. Performance metries for Sandwich BNN model with 10-minute training.
Gap%
Eq.
Eq.
Ineq.
Mean
0.022
1.58
0.20
0.018
1.56
Max
0.020
0.15
0.016
1.54
0.014
1024
2048
1024
2048
1024
2048
1024 #Samples
#Samples
# #Samples
# #Samples
Figure 1I. Performance metries for Supervised BNN model 10-minute training.
17

---

Semi-Supervised Optimization Proxies using Bayesian Neural Network
Max Eq.(SvP)
Mean Eq.(SVP)
Max Ineq.
1.60
Mean
0.11
1.55
Min
0.015
0.10
Max
0.010
1.50
0.09
0.018
0.005
1.45
0.08
0.017
512
1024
2048
1024
2048
1024
2048
1024
2048
#Samples
#Samples
#Samples
#Samples
Figure 12. Performance metries for 'Case 18' with fixed unsupervised samples (2048) over T max 900sec (15 minutes) using the
Sandwich BNN SvP model.
Gap% (SvP)
(q.(SvP)
Eq.(SvP)
Ineq. (SvP)
1.60
Mean
0.024
0.020
1.58
0.150
Min
Max
0.022
0.018
0.125
0.016
1.56
0.020
0.100
0.014
1024
1024
#Samples
# Samples
# #Samples
# #Samples
Pigure 13. Performance metrics Supervised BNN SvP model model with 15-minute training.
Gap%
Max Eq.
Mean Eq.
Max Ineq.
1.60
0.16
Mean Min Max
1.55
0.018
0.015
0.010
1.50
0.005
1.45
0.014
512
1024
2048
512
1024
2048
512
1024
2048
1024
2048
#Samples
#Samples
#Samples
#Samples
Figure 14, Performance metrics for Sandwich BNN model with 15-minute training.
Gap%
Max Eq.
Mean Eq.
Max Ineq.
1.60
Mean Min
0.022
1.58
0.018
Max
0.020
0.016
1.56
0.15
0.018
0.014
1.54
512
1024
2048
512
1024
2048
512
1024
2048
512
1024
2048
#Samples
#Samples
#Samples
#Samples
Figure 15, Performance metrics for Supervised BNN model with 15-minute training.
18


Reviews:
[{"review_id": "NEFVeonZCH", "reviewer": "Reviewer_Uqx1", "summary": "The paper proposes a new Bayesian Neural Network (BNN) for solving non-linear constrained optimization problems that can be computationally expensive. The Bayesian Neural Network serves as a proxy for solving the original problem and is computationally more efficient. The downside of similar existing approaches is that they can perform poorly if labeled data and training times are limited. To address this deficiency, the paper's BNN leverages unsupervised learning that enforces constraint feasibility. Specifically, the model alternates between phases of supervised and unsupervised learning. Finally, it samples weights for the neural network from the posterior distribution to generate a posterior prediction matrix and selects the weight that best satisfies the equality constraints. The paper then demonstrates the effectiveness of their approach compared to deep neural network alternatives against standard benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses**\n\n**1. Scalability** \nWe agree that exploring scalability is important. To address this concern, we have added additional experimental results showing that while our method excels in low-data regimes, it also remains competitive as the amount of available data increases. These experiments are performed on the case118  system due to the limited rebuttal window. But similar experiments can be performed and added to the paper for the camera ready version. \n\n**Avg. error values on `case118` with different numbers of supervised samples for Sandwich BNN SvP** \n| N |Train time | %Gap | Max Eq  | Max Ineq  | \n|-|-|-|-|-|\n|512 |15 min|1.5305|0.1041|0.01285|\n|1024|15 min|1.5097|0.0894|0.01442|\n|2048|15 min|1.4691|0.0800|0.01170|\n\nFor instance, the Sandwich BNN SvP model achieves a 1.50\\% Gap, 0.094 Max Eq., and 0.014 Max Ineq. with 512 supervised samples and a *10-min* training time (avg. of 5 trials). While, with 2048 supervised samples & *15-min* of training, it improves to a 1.46\\% Gap, 0.080 Max Eq., and 0.011 Max Ineq. Detailed results can be found at: [Link](https://shorturl.at/nFTE9)\n\n**2. Non-Uniform Performance** We acknowledge the non-uniform performance in certain experiments. We find that this is due to variations in convergence rates that arise under the tight training time constraint (10 mins) that we have imposed on the system.  For the 118 bus system trained on 1024 and 2048 samples, where increasing the training time from 10 to 15 minutes, the performance improves considerably. See the table above and [Link](https://shorturl.at/nFTE9)\n\n**3. Notation & Definitions**\nWe will revise the notation. Here $ \\mathcal{D}_{f} $ is constructed from the the unlabeled dataset $\\mathcal{D}^u$. As a valid solution must fully satisfy all constraints, it has a true feasibility gap $\\mathcal{F}$ of zero (eq. 2 in paper). Thus, we transform $\\mathcal{D}^u$ into a labeled feasibility dataset $\\mathcal{D}^f$ where  each input $\\mathbf{x}_j$ has a corresponding feasibility label of zero, i.e.,  $ \\mathcal{D}^f =  ( (\\mathbf{x}_j, \\mathcal{F}(\\cdot, \\mathbf{x}_j) = 0))$. As generating input samples is computationally cheap (just random variables within bounds), creating this feasibility dataset $\\mathcal{D}^f$ is cheap.  Note that during training, weights $w$ & corresponding output $f_w(x_j)$ are tuned to bring the computed feasibility gap $F(f_w(x_j),x_j)$ close to its true value of $0$. \n\nAlso, $ p^m_W $ is the posterior distribution of the BNN weights after $ m$ training cycles, which include both supervised & unsupervised learning steps. This serves as the final posterior distribution used for making predictions.\n\n**Questions**\n\n1&2. Check response under Weakness 3. \n\n3. Our experiments suggest that early stopping is essential while alternating between supervised & unsupervised stages. In the limited data setting, waiting for each training phase to converge before switching to the next phase can be detrimental. For instance, spending a lot of time in the unsupervised layer can improve feasibility metrics but can adversely affect the %Gap (cost).\n\n4. Yes. $f_W(\\cdot)$ represents the function that NN computes when the weights are set to $W$. And $f_W(x^t)$ is the output of this function for input $x^t$.\n\n5. Our method has an imposed time constraint and also early stopping to prevent overfitting. Under these conditions, we do not find that increasing unsupervised samples to have a significant effect on the training out comes. Learning with unsupervised data is expensive, hence significant increasing unsupervised data under short training time constraints can only enhance the performance so much. This aligns with other self-supervised DNN methods that require significant compute (Park & Hentenryck (AAAI 2023)).  We performed a preliminary scaling study on case118 by increasing the unsupervised samples to $2^{12}$ and $2^{13}$ while keeping the number of supervised samples at $512$. We only see marginal improvements with $T_{max} = 600s$.\n\n ### **Sandwich BNN SvP**\n| M (UnSup)| %Gap | Max Eq  | Max Ineq  |\n|-|-|-|-|\n| $2^{12}$ | 1.5471 | 0.077811  | 0.009252  |\n| $2^{13}$ | 1.54431 | 0.07689 | 0.008866  |\n\nWe believe that these results can be improved by changing the training constraints to better exploit more unsupervised data. We leave that study for future work.\n\n6. See reply to weakness 1.\n\n7.  We use Gaussian prior having 0 mean & 0.01 variance. We tried 1.0, 0.1 and 0.01 as variance of prior and found 0.01 to be most suitable, although quality of solution was not drastically different (all results show same order of magnitude ). We choose very low variance parameter for likelihood ($10^{-6}$) because we know that the ground truth solution of feasibility layer is a delta distribution zero (as discussed for $\\mathcal{D}^f$). So, no tuning was performed on likelihood parameters. Hyperparameter details are provided with supplementary material in config.json format and in code.  \n\n8. See response to point 7.", "claims_and_evidence": "The work is mainly empirical and the claims are supported by the numerics seem to make sense.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria make sense for the problem and application. The paper utilizes unsupervised learning to address limited data and a Bayesian approach to search for solutions that are more feasible. While I am less familiar with the evaluation criteria, it seems to be a standard benchmark for this type of work.", "theoretical_claims": "The paper makes few theoretical claims, mainly those related to Bernstein and Hoeffding bounds which seem correct.", "experimental_designs_or_analyses": "I primarily evaluated the experimental design of the paper which seems to use standard benchmarks from the open-source OPFDataset from Torch Geometric. The results seem to make sense and seem promising. The only issues I see are the robustness of the results as they only focus on testing their approach on a single sample size and training time setting. It would be interesting to see how robust their results are as you vary sample size and training time, especially since the paper claims their approaches are computationally efficient.", "relation_to_broader_scientific_literature": "The key contributions of the paper build on the literature of end-to-end constrained optimization learning which is well outlined in the survey paper [1]. Specifically, the paper focuses on machine learning predicting solutions to constrained optimization problems. Within the space, they focus on the key challenges such as limited data and expensive compute during training.  \n\n___\n[1] Kotary, J., Fioretto, F., van Hentenryck, P., and Wilder, B. End-to-end constrained optimization learning: A survey. In 30th International Joint Conference on Artificial Intelligence, IJCAI 2021, pp. 4475–4482. International Joint Conferences on Artificial Intelligence, 2021.", "essential_references_not_discussed": "I am less familiar with this area and thus do not have any additional reference suggestions.", "other_strengths_and_weaknesses": "Strengths\n1) The proposed approach produces compelling numerical results in low-data settings, which for the most part make intuitive sense or are discussed by the authors. \n2) The choice to combine Bayesian neural networks and unsupervised learning makes intuitive sense as both are suitable for low-data settings. Being able to sample multiple predictions is also an interesting feature as it allows users to cheaply tune the level of feasibility of the solutions which may be useful in practice. \n3) The paper generally feels well written and well organized. \n\nWeaknesses\n1) The paper only focuses in low-data and low-computation settings so for readers less familiar with the literature it's hard to contextualize how the method performs compared to more computationally expensive approaches or approaches that do better with more data. They also do not scale the amount of data so it's hard to understand if their approach also performs well when the amount of data increases or how the improvement scales as data increases. \n2) The improved performance of the approach is not uniform overall the experiments, specifically the ones found in the appendix. The paper makes some claims on why (mainly related to training time), so it would be useful to numerically verify the claims. \n3) There are some minor clarity issues related to notation. For example, it was hard to understand how exactly $\\mathcal{D}\\_f$ was constructed based on the notation and the notation $p_{W}^{m}$ doesn't seem to be explicitly defined in the main body.", "other_comments_or_suggestions": "1. Typo on pg 2:  \"10 minutes of training tim on a single CPU core\"\n2. In Eq (3) you give the constraint $g$ an index $I$. It may be more clear if you included it in the formulation of Eq (1). \n3. Missing a word: \"Sandwich DNN in these tables a DNN with the same network architecture as the BNN, trained under the same time constraints.\"", "questions_for_authors": "1) I don’t understand how $\\mathcal{D}\\_{f}$ is constructed since the notation $\\mathcal{D}\\_f = \\left( (\\bf{x}\\_j, \\mathcal{F}(\\cdot, x) \\right)^{M}_\\{j=1}$ is confusing. Are you trying to say you just sample some arbitrary feasible solution? Why is that computational inexpensive? Can more detail be provided?\n2) What is $p_{W}^{m}$? It should maybe be formally defined.\n3) When you alternate between unsupervised and supervised learning, is there benefit in learning until parameters converge? \n4) To clarify, is the output of the model $f_{W^*}(\\bf{x}^t)$?\n5) Does performance improve with more unsupervised data? \n6) How does performance scale as you increase data?\n7) Are there any hyper parameters that need to be tuned for the BNN? How does the tuning affect performance?\n8) How do you choose the variance parameters for the prior and likelihood? How do these parameters affect the performance of your model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1YG2b85y2r", "reviewer": "Reviewer_iXWs", "summary": "The paper introduces a Bayesian Neural Network (BNN) framework as a surrogate model for constrained optimization, leveraging its intrinsic uncertainty quantification for robust predictions. It employs a semi-supervised training strategy that alternates between supervised learning on limited labeled data and unsupervised learning that enforces constraint feasibility via data augmentation. Experiments  show that this approach significantly reduces constraint violations and prediction errors, outperforming traditional deep neural network methods in low-data, low-compute settings. Additionally, by applying Bernstein’s inequality, the model derives tight probabilistic confidence bounds, offering practical error estimates without extensive validation data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Claims And Evidence: The 2×MPV Heuristic and Its Broader Validation** \n\nWe appreciate this insightful comment. We believe that the 2MPV heuristic is sufficient to capture total variance in the error, as substantiated by the studies shown in Fig 4. This heuristic will require similar empirical validation before it can be used for other datasets. While the law of total variances argument in Eq.(5) makes the first step in understanding the theory behind this, a full theoretical understanding of why one term dominates at selected training data sizes will be the object of a future work.\n\n**Experimental Designs Or Analyses: Lack of Confidence Intervals or Statistical Significance Tests**\n\nWe acknowledge the importance of assessing the robustness of our experimental results. We have run a new batch of experiments for the 118 bus problem with increased dataset size. The variance in the results from different learning experiments can be seen here. The time budget used here is 900s or 15 mins. Results (including the base case) are slightly different from what is reported in the paper due to the use of a different machine. Despite these changes, we see that the results are quire robust.\n\n#### **Comparison of Max, Avg, and Min values for different $N$ for 15 Min Training on `case118`**\n\n| N    | Type | %Gap     | Max Eq   | Max Ineq  |\n|------|------|----------|---------|----------|\n| 512  | Max  | 1.59 | 0.115 | 0.018  |\n| 512  | Avg  | 1.53  | 0.104 | 0.012 |\n| 512  | Min  | 1.50 | 0.089 | 0.002 |\n|-|\n| 1024 | Max  | 1.53 | 0.093 | 0.017 |\n| 1024 | Avg  | 1.50  | 0.089 | 0.014  |\n| 1024 | Min  | 1.47 | 0.084  | 0.009 |\n|-|\n| 2048 | Max  | 1.47 | 0.082 | 0.010 |\n| 2048 | Avg  | 1.45 | 0.079 | 0.009 |\n| 2048 | Min  | 1.44 | 0.078 | 0.009 |\n\n\nUpdated results with error bars can be found here: [Link](https://drive.google.com/file/d/1sm9eEiYutFk89ZqKwJNI4pf8HQueUJmy/view?usp=share_link)\n\n**Essential References Not Discussed:**\n\nWe appreciate the reviewers' suggested works. While related, we note that our focus is not directly on \"Bayesian optimization\" but on building a \"Bayesian surrogate\" for the optimization problem. In particular, we do not focus on selecting additional training data via acquisition functions/active learning. However, techniques mentioned in (paper 1) such as PFNs can help potentially reduce training time of Bayesian inference used within our framework (as shown in *Transformers Can Do Bayesian Inference*, ICLR 2022). However, PFNs have so far been applied only to relatively small BNNs due to the high computational cost of the attention mechanism. We are actively exploring ways to scale PFNs, such as using linear attention or alternative architectures like CNNs.  \n\nUnlike boundary exploration for identifying unknown constraints (paper 2), the constraints are explicitly defined in our work. It will be of interest to integrate our approach with boundary exploration—potentially for feasibility verification or active learning.  We will include a discussion of these works to better position our contributions.\n\n**Weaknesses:**\n\nWe agree that actively exploring feasibility and optimality is an important direction for future research. Our problem is not black-box optimization but rather surrogate modeling. It is challenging to model optimality in BNN with unsupervised data, as the target distribution of optimality is not known a priori without supervised data. Only the feasibility target distribution can be determined, which corresponds to a delta distribution at zero. Therefore, the simultaneous exploration of both optimality-and feasibility, including the right mix of training data with sandwich approach, will be our future work. \n\n**Questions**\n\n 1. As shown in the original draft, we did sandwich learning on DNN and showed that under low supervised data, BNN outperforms DNN considerably.  Also, the comparative results with other DNN methods indicate that sandwiching remains effective.\n\n2.  As shown in (min-max) Table above,  we see that our results are quite robust to changes in the training set, time budget and report the max and min values of all the performance metrics and additional results in an updated pdf file here [Link](https://drive.google.com/file/d/1sm9eEiYutFk89ZqKwJNI4pf8HQueUJmy/view?usp=share_link)\n\n3. The current work is not focused on active learning or intelligent sampling of training data. Incorporating regret guarantees is an interesting direction for future work. One possible direction we see is to use a sandwich model with very low supervised data, which can provide a computationally cheap baseline, then use that as a surrogate in active search of training samples and developing regret guarantees.", "claims_and_evidence": "The experimental results in the submission strongly support its core claims within the context of AC optimal power flow problems, demonstrating improvements in constraint satisfaction and reduced prediction errors under low-data and low-compute conditions. However, the generality of the approach beyond ACOPF is less clearly supported, as most evidence is specific to this application domain. In addition, the claim that a simple multiplier (2×MPV) reliably bounds the total variance in error is based on a hypothesis validated only in these experiments and may require further theoretical backing and broader empirical evaluation. Overall, while the claims are convincing for the studied case, their applicability to other constrained optimization problems remains somewhat problematic.", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criteria are well-aligned with the application. The paper’s use of semi-supervised Bayesian neural networks addresses the challenge of limited labeled data and short training times, which are key constraints in real-world power grid optimization. Evaluation metrics such as the optimality gap, equality/inequality constraint violations, and probabilistic confidence bounds directly assess performance in areas critical to ACOPF problems. Moreover, using established benchmark datasets ensures that the evaluation is robust and comparable to baselines.", "theoretical_claims": "I reviewed the derivations related to the probabilistic confidence bounds, which rely on established concentration inequalities like Hoeffding’s and Bernstein’s inequalities. The proofs themselves are adaptations of well-known results rather than new theoretical contributions, and they primarily serve to justify the use of the 2×MPV heuristic as a bound on the total variance in error. I did not encounter any significant issues in the correctness of these derivations, but it’s important to note that the paper does not present any substantial new theory. Overall, the theoretical claims are based on solid, previously established results, with the novel contribution being in their application within the BNN-based optimization proxy framework.", "experimental_designs_or_analyses": "I examined the experimental designs used for evaluating the ACOPF problems across different system sizes, where the authors compare metrics such as optimality gap, equality/inequality violations, and error bounds among various models. The overall design is sound for the application at hand, with a reasonable selection of benchmark datasets and a clear focus on both supervised and semi-supervised BNN approaches. However, one issue is that the analyses do not include any confidence intervals or error bars for the reported metrics, which makes it harder to assess the statistical significance and variability of the improvements. This omission limits our ability to fully evaluate the robustness of the experimental findings.", "relation_to_broader_scientific_literature": "The paper’s key contributions are well-rooted in existing literature on constrained optimization and semi-supervised learning. Notably, the authors propose a sandwich training framework that alternates between a supervised phase on limited labeled data and an unsupervised phase that enforces constraint feasibility via data augmentation. Although demonstrated with Bayesian Neural Networks (BNNs) to leverage uncertainty quantification and tight error bounds via Bernstein’s inequality, the sandwich training framework itself is not limited to BNNs; it has broader potential for application across various neural architectures. This approach builds on prior methods—such as penalty methods and projection techniques for constrained optimization, as well as semi-supervised learning strategies like pseudo-labeling—and extends these ideas to effectively address real-world problems with scarce labeled data and strict operational constraints.", "essential_references_not_discussed": "Yes, there are related works that could further contextualize the paper’s key contributions, particularly in the areas of surrogate modeling with limited data and constraint handling. For example:\n- ***Prior Fitted Neural Network as Surrogate:***\nMüller, Samuel, Matthias Feurer, Noah Hollmann, and Frank Hutter’s “Pfns4bo: In-context learning for Bayesian optimization” (ICML 2023) addresses the challenge of limited data by leveraging a prior fitting approach. This work provides an alternative perspective on building surrogate models that effectively operate in data-scarce regimes, which is closely related to the paper’s focus on constrained optimization proxies.\n \n- ***Boundary Exploration for Bayesian Optimization with Unknown Physical Constraints:***\nTian, Yunsheng, Ane Zuniga, Xinwei Zhang, Johannes P. Dürholt, Payel Das, Jie Chen, Wojciech Matusik, and Mina Konakovic Lukovic’s work “Boundary Exploration for Bayesian Optimization With Unknown Physical Constraints” (ICML 2024) employs ensemble methods to explore constraint boundaries. This study offers insights into handling unknown physical constraints through an ensemble approach, complementing the paper’s sandwich training framework that enforces feasibility via semi-supervised learning.\n\nIncluding and discussing these works would enhance the understanding of how the paper’s contributions fit within the broader landscape of surrogate-based optimization and constrained learning methods.", "other_strengths_and_weaknesses": "Strengths:\n- The paper introduces a novel sandwich training framework that alternates between supervised and unsupervised phases to enforce constraint feasibility. Importantly, this framework is not limited to Bayesian Neural Networks (BNNs), suggesting broader applicability to various neural architectures in surrogate modeling and optimization tasks.\n\nWeaknesses:\n - The focus on surrogate modeling involves a largely passive approach to identifying feasibility, actively optimizing only within a pre-identified feasible region rather than integrating feasibility and optimality exploration simultaneously. This approach is reminiscent of recent work such as CONFIG by Xu et al. (Xu, Wenjie, Yuning Jiang, Bratislav Svetozarevic, and Colin Jones. “Constrained efficient global optimization of expensive black-box functions.” In International Conference on Machine Learning, pp. 38485-38498. PMLR, 2023), which more actively integrates constraint exploration. Additionally, the paper does not provide any regret guarantees, a limitation also noted in similar treatments like CONFIG.", "other_comments_or_suggestions": "Merged with the section below.", "questions_for_authors": "1.  While the sandwich training framework is demonstrated with BNNs, have you considered or experimented with applying it to other neural architectures, and what challenges do you anticipate in such extensions?\n\n2. The experiments currently lack confidence intervals or statistical significance tests. Could you provide additional details on the variability of your results and any measures taken to assess statistical robustness?\n\n3.  Given that similar approaches like CONFIG (Xu et al., 2023) offer integrated exploration and regret guarantees, do you see a pathway for incorporating regret guarantees into your framework, and how might that impact performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eibyQrFXFD", "reviewer": "Reviewer_2Rny", "summary": "This paper proposes and evaluates a semi-supervised approach to training Bayesian Neural Network (BNN) optimization proxies for predicting solutions to constrained optimization problems. Specifically, the paper proposes augmenting training on labeled data (from solving optimization problem instances) with training on unlabeled data about constraint satisfaction in a \"sandwich\" fashion. The paper is concerned with settings in which both labeled data (for training and validation) and training time for the model are constrained. The authors argue that the use of BNNs allows both an increase in sample efficiency compared to (non-Bayesian) DNNs as well as a principled predictive uncertainty quantifications, for which they develop a new approach based on Bernstein concentration bounds. The paper evaluates the proposed approach on optimal electric power flow optimization benchmarks, demonstrating improved performance in the small-sample regime compared to a number of baseline methods.\n\n### Update after rebuttal\nOverall, given the authors' responses I feel slightly more positive about the paper, but not enough to raise my score to a full accept. For that I would still like to see a more detailed ablation study on the training setup, as well as a comparison with more compute-intensive baselines (even though they may not be feasible from an applied perspective).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Methods and evaluation criteria**\n\nWe agree with the reviewer that BNN training can be sensitive to various random choices in the training. We have run a new batch of experiments for the 118 bus problem with increased dataset size. The variance in the results from different learning experiments can be seen here. The time budget used here is 900s. Results are slightly different from what is reported in the paper due to us using a different machine. Despite these changes, we see that the results are quire robust.\n\n#### **Comparison of Max, Avg, and Min values for different $N$ for 15 Min Training on `case118`**\n\n| N    | Type | %Gap  | Max Eq  | Max Ineq |\n|------|------|-------|--------|---------|\n|      | Max  | 1.597 | 0.115  | 0.019   |\n|  512  | Avg  | 1.531 | 0.104  | 0.013   |\n|      | Min  | 1.506 | 0.090  | 0.002   |\n|-----------|\n|      | Max  | 1.530 | 0.094  | 0.017   |\n|  1024 | Avg  | 1.510 | 0.089  | 0.014   |\n|      | Min  | 1.473 | 0.084  | 0.009   |\n|---------|\n|      | Max  | 1.473 | 0.083  | 0.010   |\n|  2048 | Avg  | 1.456 | 0.079  | 0.009   |\n|      | Min  | 1.441 | 0.078  | 0.009   |\n\nMore experiment results can be found here: [Link](https://drive.google.com/file/d/1sm9eEiYutFk89ZqKwJNI4pf8HQueUJmy/view?usp=share_link)\n\n**Comparison with ensemble DNN and GNN**\n\nWe agree that ensemble methods can potentially help in quantifying uncertainty. Our comparison, for the OPF benchmark, is currently with state of the art scalable Neural network based methods given our limited training time and data availability. At present we are not aware of any existing work on ensemble methods for the case of OPF. If individual ML methods are trained sequentially without our framework, scalability might be an issue. However developing ensemble methods for OPF and comparing with our BNN model will an useful avenue of future work.  \n\n**Questions**\n\n**1.** We did not do this for our experiments as we see that the inequality violations are already low, especially in comparison to the equality violations. Moreover, our mean inequality Gap results are similar to that of DNNs with repair obtained by ML4OPF tools.  Due to limited time for rebuttable we did not setup those bound repair experiments with BNN. Also, note that criteria of SvP selection is only minimization of maximum equality gap. We also note that one can include inequality constraints in SvP criteria to improve upon further. We can include it in our final draft.\n\n**2.** This effectiveness of sandwich method for case57 likely due to the limited time budget of 600s that we have used for all the experiments in the paper. The sandwich BNN for case57 has ample time to converge, but larger cases are stopped early. Even under this constraint, we see that the Sandwich BNN still outperforms other methods even for larger cases. Further, case57 has less active constraints and thus intuitively we can say that unsupervised data is able to enforce feasibility much better in sandwiched learning. \n\nWe conduct a robustness study on `case118' by running five trials with training times of 10 and 15 minutes, using 512, 1024, and 2048 supervised training samples, while keeping 2048 unsupervised samples constant across all models and trials. The results clearly demonstrate that increasing training time and the number of supervised samples consistently reduces \\%Gap, Max Eq., and Max Ineq. Additionally, Mean Eq. remains an order of magnitude lower, and Mean Ineq. stays at the order of $10^{-5}$. For instance, the Sandwich BNN SvP model achieves a 1.50\\% Gap, 0.094 Max Eq., and 0.014 Max Ineq. with 512 supervised samples and a 10-minute training time, averaged over five trials. In contrast, with 2048 supervised samples and 15 minutes of training, the same model improves to a 1.46\\% Gap, 0.080 Max Eq., and 0.011 Max Ineq. This implies that increasing training time by five minutes while quadrupling the supervised samples results in a 2.66\\% reduction in optimality error (\\%Gap), a 17.5\\% reduction in equality constraint feasibility error (Max Eq.), and more than a 27\\% reduction in inequality constraint feasibility error (Max Ineq.).  \n\n**3.** We find that early stopping the unsupervised round is necessary to prevent it from overfitting. The intuition we follow is that we do not want to fit either supervised  or unsupervised models too well, to avoid biasing our surrogate towards either feasibility or optimality. Further, we observe that too many unlabeled examples will require too much compute to extract any meaningful information, thus we kept them constant at 2048. We fix each round of unsupervised training to 150 sec. and supervised training time as 100 sec. for 512 samples. While we do not fine tune this choice in our experiments, cros-validation on different relative training sizes might be a way to understand the dependence of hyperparameters that includes number of unlabeled examples and division of time between supervised and unsupervised training.", "claims_and_evidence": "The claims appear well-supported overall. However, the empirical evaluation could be more comprehensive and include some other (e.g. non-power-flow) example applications as well as more detailed ablation studies of some of the hyperparameters (e.g. number of unlabeled examples, learning rates, etc.)", "methods_and_evaluation_criteria": "* The optimal power flow problems are fine benchmark problems, but given that the proposed method is a generic one I would have expected to see some problems from other applications to better understand how the approach performs in other domains.\n* The evaluation criteria generally make sense to me - as someone with limited familiarity with the specific literature in this area. However, I am surprised to only see the performance compared on single instances without providing variances across the randomness in the results (e.g. w.r.t. the testing instances / data generation / splitting and w.r.t. to the weight initialization and other randomness in the methods themselves). This leaves open the possibility that the provided results are somewhat cherry-picked -- I am not suggesting that they are, rather that it's not clear to me from the paper, and the paper could to better at proactively assuaging this concern.\n* The evaluation excluded a number of baseline methods that would be interesting to have for comparison, such as self-supervised constrained optimization methods and Graph Neural Network-based large models. The authors rightfully argue that these methods may not be feasible in the setting targeted by the paper due to excessive computational requirements -- however, it would be very helpful from a scientific perspective to see the performance penalty incurred by using fewer samples with the proposed approach to understand the broader tradeoffs between the different methods.\n* Related to the above, the paper also mentions Ensemble DNNs as an alternative approach, but does not discuss them in detail due to \"very high computational requirements\". It's not necessarily clear to me that this will always be an issue. For instance, in settings where label generation is extremely costly but DNN training time is not a major concern, ensemble DNNs could be a good alternative. Given that a claimed primary contribution is the uncertainty quantification, having the evaluation against ensemble DNNs as at least one baseline would be highly valuable.", "theoretical_claims": "N/A -- there are no novel theoretical claims (the authors apply a \"theoretical\" version of Bernstein's inequality using the Mean Predictive Variance, but are basing this on a hypothesis that they only check empirically in a limited number of examples).", "experimental_designs_or_analyses": "Not in detail, the experimental setup seems reasonable (modulo the comments on evaluation criteria above). \n\nOne thing I'd like to see more of are ablations of some of the training settings. BNN learning can be rather finicky, and so it would be important to understand how sensitive the setup is to the number of training samples and unlabeled samples and hyperparameters such as learning rates etc.", "relation_to_broader_scientific_literature": "* The main novel angle of the paper is the focus on and analysis of the small-sample regime in which labeled data and training time is limited. Previous works have often focused on settings where that data and time was less of a constraint, so the paper provides an interesting additional perspective.\n* Both the use of BNNs as optimization proxies and the semi-supervised data augmentation using feasibility information appears novel (though I have very limited familiarity with the literature).", "essential_references_not_discussed": "I am not familiar enough with the specific literature to assess this properly.", "other_strengths_and_weaknesses": "* The paper is generally well written. I appreciated that it does a decent job introducing the problem setting and some of the background material, allowing me as someone less intimately familiar with the literature to follow the presentation without too much trouble in most places.\n* The focus on limited data and training time and comparing performance under these constraints is a nice way of looking at the practicality of the approach (though I would like to understand the gap to an ideal baseline, see comments above)", "other_comments_or_suggestions": "N/A", "questions_for_authors": "* You mention that \"the ‘Max Ineq.’ growth could be easily suppressed by incorporating bound repair layers, as used in DNN models in ML4OPF. Why not do this? This would be a very useful ablation to understand the effect of a bound repair layer on the performance. Especially in light of the fact that you observe that \"Sandwich BNN SvP shows slightly higher scaling inequality gaps\".\n* It appears that Sandwich learning is a lot more helpful for case57 - why? Is this b/c constraints are more challenging to satisfy in this problem? It would be helpful if you could provide this kind of intuition more broadly in the discussion of your results.\n* How to select the number of unlabeled examples and schedule the learning steps such that the model doesn't over-fit to the constraints? Is there a principled way to do this? Can you provide any ablations on this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a semi-supervised BNN approach for building optimization proxies in constrained settings with limited labeled data and short training times. The discussion focused on whether the method generalizes beyond the presented power flow application, the limited amount of labeled data in experiments, and the potential benefits of additional baselines. Nonetheless, reviewers noted that the proposed framework shows convincing improvements in non-convex constrained optimization tasks and provides meaningful uncertainty estimates under tight training budgets.

The authors should modify the paper as highlighted in the discussion for the final version: especially in providing further details on the training setup of sensitivity and include additional examples or baselines wherever possible to strengthen the evidence for their approach versatility.
Decision: Accept (poster)