Optimization Proxies using Limited Labeled Data and Training Time - A Semi-Supervised Bayesian Neural Network Approach

## Abstract

Abstract Constrained optimization problems arise in various engineering system operations such as inventory management and electric power grids. However, the requirement to repeatedly solve such optimization problems with uncertain parameters poses a significant computational challenge. This work introduces a learning scheme using Bayesian Neural Networks (BNNs) to solve constrained optimization problems under limited labeled data and restricted model training times. We propose a semi-supervised BNN for this practical but complex regime, wherein training commences in a sandwiched fashion, alternating between a supervised learning step (using labeled data) for minimizing cost, and an unsupervised learning step (using unlabeled data) for enforcing constraint feasibility. Both supervised and unsupervised steps use a Bayesian approach, where Stochastic Variational Inference is employed for approximate Bayesian inference. We show that the proposed semi-supervised learning method outperforms conventional BNN and deep neural network (DNN) architectures on important non-convex constrained optimization problems from energy network operations, achieving up to a tenfold reduction in expected maximum equality gap and halving the optimality and inequality (feasibility) gaps, without requiring any correction or projection step. By leveraging the BNN’s ability to provide posterior samples at minimal computational cost, we demonstrate that a Selection via Posterior (SvP) scheme can further reduce equality gaps by more than 10%. We also provide tight and practically meaningful probabilistic confidence bounds that can be constructed using a low number of labeled testing data and readily adapted to other applications.

## 1 Introduction

Constrained optimization problems are fundamental in the optimal operation of various engineering systems, such as supply chains, transportation networks, and power grids. Learning a forward mapping between the inputs and outputs of these problems can significantly reduce computational burdens, especially when rapid solutions are required, such as in electricity market clearing or real-time transportation planning.

Recent advancements in machine learning (ML) have led to considerable efforts to solve optimization problems using deep neural networks (DNNs) Khadivi et al. (); Kotary et al. (); Fajemisin et al. (). The idea of learning input-to-output mappings has been explored via supervised and unsupervised methods, particularly in power system applications Zamzam & Baker (); Donti et al. (); Park & Van Hentenryck (); Fioretto et al. (); Kotary et al. (); Rolnick et al. (). Additionally, constraint penalization approaches have been proposed to enforce feasibility in predicted outputs within DNN loss functions AI4OPT ().

Supervised DNN models rely on labeled datasets obtained by solving numerous instances of optimization problems. This data generation step poses a significant limitation due to the prohibitive computational time required for moderate to large problem instances. For example, Park & Van Hentenryck () report that generating labeled data to train supervised DNN models for a medium-sized power grid problem takes over three hours(^1^11See Table 4 in Park & Van Hentenryck () for the computation time comparison.). Unsupervised methods aim to address the labeled data generation issue Donti et al. (); Park & Van Hentenryck (); however, they often have high training time requirements. Methods requiring projection steps to recover feasible solutions involve time-consuming constraint correction within the framework Donti et al. (); Gupta et al. (); Zamzam & Baker (), resulting in slower training and prediction. Moreover, unsupervised methods still require testing data and consequently the associated data generation time in order to perform validation and provide confidence bounds on error with respect to true solution. Thus, it is important to note that that total time requirement to deploy these ML based optimization proxies is: $\texttt{Training Data Generation}+\texttt{Testing Data Generation}+\texttt{Training}$.

In practice, the input space of the problem in many applications is dynamic and changes constantly, necessitating frequent retraining and evaluation for ML models to remain effective amid distributional shifts in the input data. In the context of longer horizon planning problems involving bi-level optimization problems, these ML proxies serve as subroutines to simulate stochastic decision-making processes for given first-level decisions Ibrahim et al. (). Consequently, these models must be adaptable in both training and testing to changing problem inputs and parameters. Minimizing or limiting the time required for learning input/output mappings in optimization problems is thus crucial.

Estimating the generalization error over the testing dataset is another critical aspect, and is particularly relevant to planning and operations of engineered systems where the system must obey the physical and safety limits. As a result, when limited labeled data is available, one must rely on concentration bounds such as Hoeffding’s inequality Sridharan (); Hoeffding () to develop expected error bounds using finite out-of-sample testing data. However, these bounds are often loose and impractical, creating the need for tighter expected error bounds with limited labeled testing data.

##### Contributions:

Motivated by the preceding discussion, this paper considers the problem of designing optimization proxies with guaranteed confidence bounds in the setting of limited *total time* requirement and limited labeled sample availability. For practical applicability, the *total time* must account for training and testing data generation, model training time and prediction time.
Our major contribution is the development of a semi-supervised Bayesian Neural Network (BNN) approach for this setting, that can be used to give tight confidence bounds on predictions. First, we propose using BNNs instead of DNNs for learning input-to-output mappings, as they provide intrinsic uncertainty quantification and allow the integration of prior beliefs Papamarkou et al. (). Second, we introduce a Sandwich learning method for BNN, which integrates unlabeled data into training through feasibility-based data augmentation. This approach enforces feasibility without requiring solved labeled instances. Third, we utilize the predictive variance information provided by BNNs to develop tight and practically useful expected error bounds using Bernstein concentration bounds Audibert et al. (). We intentionally restrict ourselves to 10 minutes of training time on a single CPU core to demonstrate the effectiveness of the proposed learning scheme under low-data, low-compute settings. For various power grid optimization problem instances, we show that (i) supervised BNNs outperform standard supervised DNN approaches under limited training time; (ii) the proposed Sandwich BNN enforces feasibility better than supervised BNNs without requiring additional computation time for training or data generation; and (iii) the Bernstein bound-based expected error bounds are tight and practically useful for constraint satisfaction studies without extra computational effort.

### 1.1 Related Work

In recent years, Deep Neural Networks (DNNs) have been applied to solve various optimization problems with physics-based constraints, particularly in energy networks Zamzam & Baker (); Gupta et al. (); Donti et al. (); Singh et al. (); Park & Van Hentenryck (); Kotary et al. (). The primary motivation is to replace time-consuming optimization algorithms with machine learning proxies, enabling instantaneous solutions to a large number of problem instances Park & Van Hentenryck (); Donti et al. (); Gupta et al. (); Zamzam & Baker ().

Outside the realm of optimization proxies, several semi-supervised learning methods have been proposed to leverage unlabeled data for improving ML model performance Yang et al. (). These approaches include augmenting unlabeled data with inexpensive pseudo-labels and developing unsupervised loss functions to be minimized alongside supervised loss functions Sharma et al. (); Yang et al. (). Data augmentation has been used in image classification with Bayesian Neural Networks (BNNs) using the notion of semantic similarity Sharma et al. ().
However, this concept is not readily extensible to ML proxies for constrained optimization problems, where semantic similarity is hard to quantify for input variations leading to changes in the output. To address this challenge, we propose a feasibility-based data augmentation scheme that relates directly to the constraints of the optimization problem. To the best of our knowledge, these ideas have not been explored in the context of BNN algorithms for solving large-scale optimization problems. A related but distinct line of work involves loss function-based prior design for output constraint satisfaction Sam et al. (); Yang et al. ().

## 2 Background/Preliminaries

### 2.1 Problem Setup and Assumptions

We consider nonlinear and non-convex, constrained optimization problems involving both equality constraints $g(\mathbf{x},\mathbf{y})=0$ and inequality constraints $h(\mathbf{x},\mathbf{y})\leqslant 0$, where $\mathbf{y}$ represents the decision variables and $\mathbf{x}$ represents the input variables, both as vectors. The objective is to minimize the cost function $c(\mathbf{y})$. Mathematically, the optimization problem is as follows:

$$ $\displaystyle\min_{\mathbf{y}}\quad$ $\displaystyle c(\mathbf{y})$ (1a) s.t. $\displaystyle g(\mathbf{x},\mathbf{y})=0$ (1b) $\displaystyle h(\mathbf{x},\mathbf{y})\leqslant 0$ (1c) $\displaystyle\mathbf{x}\text{ is given (input vector)}$ (1d) $$

We assume that for all $\forall\mathbf{x}\in\mathcal{X}$, i.e., for any input vector in the set $\mathcal{X}$ ($\mathcal{X}$ could be as simple as a hyper-rectangle), there exists at least one feasible solution to problem equation 1a. Let $\mathcal{D}=\{(\mathbf{x}_{i},\mathbf{y}^{\star}_{i})\}_{i=1}^{N}$ denote the labeled dataset, where $\mathbf{y}^{\star}_{i}$ is the optimal solution obtained by solving optimization problem equation 1a for each $\mathbf{x}_{i}$. Assuming that sampling the input vector $\mathbf{x}$ is inexpensive, we construct an unlabeled dataset $\mathcal{D}^{u}=\{\mathbf{x}_{j}\}_{j=1}^{M}$.

Our goal is to develop a BNN surrogate that provides an approximate optimal value of the decision variables $\widehat{\mathbf{y}}_{t}$ for a given test input vector $\mathbf{x}_{t}\in\mathcal{X}$. This work falls under the category of developing optimization proxies or surrogates, where the machine learning model serves as a direct forward mapping between the input and output variables of an optimization problem (see Park & Van Hentenryck ()).

The paper proposes a semi-supervised framework to solve this problem, wherein training alternates between a supervised step—using labeled data $\mathcal{D}$ to minimize prediction error—and an unsupervised step—using unlabeled data $\mathcal{D}^{u}$ to enforce the feasibility of constraints in equation 1b and equation 1c. Both steps are implemented using a Bayesian Neural Network (BNN).

### 2.2 Bayesian Neural Network

We consider a Bayesian Neural Network (BNN) denoted as $f_{w}(\mathbf{x})$, where $w$ represents all the weights and biases of the network. These weights are assigned an isotropic normal prior $p(w)$ with covariance $\sigma^{2}I$, meaning that each weight is independently normally distributed with zero mean and variance $\sigma^{2}$.

In the supervised training of the BNN, the goal is to compute the posterior distribution over the weights given the labeled data $\mathcal{D}\equiv(\mathbf{x},\mathbf{y})$. This posterior is expressed as

$$ $p(w\mid\mathbf{x},\mathbf{y})\propto p(\mathbf{y}\mid\mathbf{x},w)\,p(w)$ $$

Here, $p(\mathbf{y}\mid\mathbf{x},w)$ is the likelihood of the labeled data given the weights, and $p(w)$ is the prior over the weights. The posterior distribution $p(w\mid\mathbf{x},\mathbf{y})$ encapsulates the uncertainty about the weights after observing the data. Due to computational challenges in calculating the normalization constant of the posterior, approximate methods such as stochastic variational inference (SVI) with the mean-field assumption are employed the posterior distribution estimation (see Jospin et al. ()).

For making predictions, the posterior predictive distribution is approximated as
$p(\mathbf{y}^{t}\mid\mathbf{x}^{t},\mathcal{D})=\mathbb{E}_{p(w\mid\mathcal{D}
)}\left[p\big{(}f_{w}(\mathbf{x}^{t})\big{)}\right]$,
where $\mathbf{x}^{t}$ is a test input vector, and the expectation is taken over the approximate posterior distribution of the weights. Moreover, we assume a Gaussian likelihood for output:

$$ $p(\mathbf{y}\mid\mathbf{x},w)=\prod_{i}\mathcal{N}\left(\mathbf{y}_{i}\mid f_{ w}(\mathbf{x}_{i}),\sigma_{s}^{2}\right),$ $$

with $\sigma_{s}^{2}$ being a parameter in the SVI that controls the spread (noise variance) around the target values, and $(\mathbf{x}_{i},\mathbf{y}_{i})\in\mathcal{D}$. Adapting this approach to update the BNN using the unsupervised data $\mathcal{D}^{u}$ requires the definition of a suitable likelihood function, detailed in the next section, along with the semi-supervised framework to obtain the BNN surrogate.

## 3 Semi-supervised Learning: Sandwich BNN

We start by defining a suitable likelihood function for the unsupervised learning process. To that end, we augment the unlabeled data $\mathcal{D}^{u}$ using the necessary feasibility conditions that the vector $\mathbf{y}$ must satisfy to be a solution of equation 1a. We propose a function $\mathcal{F}(\mathbf{y},\mathbf{x})$ to measure the feasibility of a candidate solution $\mathbf{y}$ for a given input $\mathbf{x}$. This function consists of two terms: one measuring the equality gap and the other measuring the one-sided inequality gap or violations. The relative emphasis on each term is determined by the parameters $\lambda_{e}$ and $\lambda_{i}$, respectively, i.e.,

$$ $\displaystyle\mathcal{F}(\mathbf{y},\mathbf{x})=\lambda_{e}\underbrace{\big{\| }g(\mathbf{x},\mathbf{y})\big{\|}^{2}}_{\texttt{Equality Gap}}+\lambda_{i} \underbrace{\big{\|}\text{ReLU}[h(\mathbf{x},\mathbf{y})]\big{\|}^{2}}_{ \texttt{Inequality Gap}}.$ (2) $$

For any given feasible solution $\mathbf{y}_{c}$ for the optimization problem in equation 1a(^2^22Not necessarily optimal for equation 1a.), we have $\mathcal{F}(\mathbf{y}_{c},\mathbf{x})=0$ for the given input $\mathbf{x}\in\mathcal{X}$. Furthermore, because of our assumption in Sec. 2.1 that the problem in equation 1a has at least one feasible solution, the minimum value $\mathcal{F}(\cdot,\mathbf{x})=0$ for any $\mathbf{x}\in\mathcal{X}$. Therefore, we can augment the unlabeled dataset $\mathcal{D}^{u}$ to create a labeled feasibility dataset, i.e., $\mathcal{D}^{f}=\{(\mathbf{x}_{j},\mathcal{F}(\cdot,\mathbf{x})=0)\}_{j=1}^{M}$. Since input sampling is inexpensive, constructing this feasibility dataset $\mathcal{D}^{f}$ incurs no additional computational cost. Similar to the supervised step in Sec. 2.2, we now define a Gaussian likelihood for the unsupervised training step, with $\sigma_{u}^{2}$ as the noise variance for unsupervised learning and $\mathbf{x}_{j}\in\mathcal{D}^{f}$, as

$$ $p(\mathcal{F}\mid\mathbf{x},w)=\prod_{j}\mathcal{N}\left(0\mid\mathcal{F}\big{ (}f_{w}(\mathbf{x}_{j}),\mathbf{x}_{j}\big{)},\sigma_{u}^{2}\right),$ $$

To obtain an optimization proxy, we parameterize the candidate solution $f_{w}(\mathbf{x})$ using a Deep Neural Network (DNN)-style architecture and employ a sandwich-style semi-supervised training for the BNN, as illustrated in Figure 1. The fundamental idea of this training method is to update the network weights and biases through multiple rounds of training in which each round alternatives between using the labeled dataset $\mathcal{D}$ for prediction or cost optimality, and the augmented feasibility data set $\mathcal{D}^{f}$ for constraint feasibility. We let Sup and UnSup denote the inference steps in the BNN training using $\mathcal{D}$ and $\mathcal{D}^{f}$, respectively. Both Sup and UnSup are performed for a fixed maximum time, with the total training time constrained to $T_{\text{max}}$. Finally, the prediction of the mean estimate $\mathbb{E}_{\mathbf{y}^{t}}$ and the predictive variance estimate $\mathbb{V}_{\mathbf{y}^{t}}$ is accomplished using an unbiased Monte Carlo estimator by sampling 500 weights from the final weight posterior $p_{W}^{m}$.

Figure: Figure 1: Flowchart of the proposed sandwich-style BNN learning. The Sup block represents the supervised learning stage with labeled dataset $\mathcal{D}$, and the UnSup block represents the unsupervised learning with the augmented feasibility dataset $\mathcal{D}^{f}$. Learning time upper limits are represented as $T_{s}$, $T_{u}$, and $T_{\text{max}}$ for Sup, UnSup, and the complete semi-supervised BNN learning, respectively. At the prediction stage, $\mathbf{Y}$ denotes the posterior prediction matrix (PPM) for one test input sample, where each column represents the predicted output obtained via one weight sample from the posterior.

### 3.1 Selection via Posterior (SvP)

In Bayesian Neural Network (BNN) literature, the standard approach is to use the mean posterior prediction $\mathbb{E}_{p(w|\mathcal{D})}[f_{w}(\mathbf{x}^{t})]$ for a test input $\mathbf{x}^{t}$. This is similar to using the mean prediction of ensemble Deep Neural Networks (DNNs). However, unlike DNNs, BNNs can provide multiple predictions without additional training cost, as we can sample multiple weight instances from the posterior distribution $p_{W}^{m}$ and construct the posterior prediction matrix (PPM) $\mathbf{Y}$ (see Figure 1 for details and Appendix B.3 for structure of the PPM.). We propose to use the PPM to improve the feasibility of the predicted output of the optimization proxy. Each column of the PPM represents one predicted output vector corresponding to a weight sample. We select the weight sample $W^{\star}$ that minimizes the maximum equality gap, defined as:

$$ $\displaystyle W^{\star}=\arg\min_{j}\left[\max_{i}\,\left|g_{i}(\mathbf{x}^{t} ,\mathbf{Y}_{\cdot j})\right|\right],$ (3) $$

where $\mathbf{Y}_{\cdot j}$ is the $j$-th column of the PPM, and $g_{i}(\cdot,\cdot)$ represents the $i$-th equality constraint function.(^3^33In a general setting of constrained optimization problems, there may be multiple equality and inequality constraints.) The output prediction corresponding to the weight sample $W^{\star}$ will have the minimum equality gap, and we term this process Selection via Posterior (SvP). Note that the numerical operation in equation 3 can be performed in parallel and has minimal computational cost compared to analytical projection methods in Zamzam & Baker () that focus on projecting the prediction onto equation 1c to satisfy the inequality constraints in the problem equation 1a. Note that it is an application motivated design choice to emphasize enforcement of the equality gap by using the SvP in equation 3. This can easily be adapted to account for inequality constraints without any significant computational overhead.

## 4 Probabilistic Confidence Bounds

This section focuses on providing bounds on the expected absolute error of our method, i.e., testing error. We explore probabilistic confidence bounds (PCBs) for optimization proxies. The core concept of PCBs is to first evaluate models on a labeled testing dataset with $M$ samples, compute the empirical mean error, and then probabilistically bound the error for any new input. Specifically, PCBs assert that the expected error will be within $\varepsilon$ of the empirical errors computed from $M$ out-of-sample inputs, with a high probability (usually 0.95). Mathematically, we aim to provide a guarantee on the error $e=y-y^{t}$, where $y^{t}$ is the BNN prediction and $y$ is the true value, as

$$ $\displaystyle\mathbb{P}\bigg{\{}\bigg{|}\mathbb{E}[|e|]-\frac{1}{M}\sum^{M}_{i =1}|e_{i}|\bigg{|}\leqslant\varepsilon\bigg{\}}\geqslant 1-\delta$ (4) $$

where $\mathbb{E}[|e|]$ represents the expected absolute error, $1-\delta$ is the confidence level, and $\varepsilon$ is the allowable prediction error.

Ideally, we would like to evaluate our model on a large number of samples since, as $M\rightarrow\infty$, the error bound $\varepsilon\rightarrow 0$. However, increasing $M$ leads to a higher requirement for labeled data, which defeats the purpose of training using low labeled data (^4^44Note that the total labeled data requirement is the sum of training and testing samples, i.e., $N+M$). To address this issue, confidence inequalities are commonly used to provide PCBs, with Hoeffding’s inequality (Hoeffding ()) being one of the most widely used bounds. As stated in Appendix D, Hoeffding’s inequality assumes that the error is bounded (i.e., $|e_{i}|\leqslant R$ for all $i$) and provides PCBs whose tightness is governed by $M$, with the relationship
$\varepsilon=R\sqrt{\frac{\log\left({2}/{\delta}\right)}{2M}}.$ However, the Hoeffding bound can often be too loose to be practically relevant.

To improve upon this, we propose using Bernstein’s inequality (see Audibert et al. ()) as the concentration bound, which utilizes the total variance in error (TVE) information along with $M$, under the same bounded error assumption. The main challenge in using the Bernstein bound is obtaining the TVE without extensive out-of-sample testing. One possible solution is to use the empirical Bernstein bound as in Audibert et al. (); Mnih et al. (), which employs the empirical variance of the error $\widehat{\mathbb{V}}_{e}$, obtained from the same $M$ testing samples, and accounts for the error in variance estimation by modifying the theoretical Bernstein inequality. Mathematically, the PCB using the Empirical Bernstein inequality is

$$ $\varepsilon=\sqrt{\frac{2\widehat{\mathbb{V}}_{e}\log\left({3}/{\delta}\right) }{M}}+\frac{3R\log\left({3}/{\delta}\right)}{M},$ $$

as given in Audibert et al. (); Mnih et al. (). Since the term under the square root depends on the empirical TVE $\widehat{\mathbb{V}}_{e}$ rather than $R$, the empirical Bernstein bound becomes tighter more quickly with increasing $M$ if $\widehat{\mathbb{V}}_{e}\ll R$.

To further tighen the bound, we propose using the Theoretical Bernstein bound Sridharan () (Theorem 3 in Appendix D) with the *Mean Predictive Variance* (MPV) as a proxy for the TVE $\mathbb{V}_{e}$. The MPV is the mean of the predictive variance of testing samples, i.e., $\text{MPV}=\frac{1}{M}\sum_{k=1}^{M}\mathbb{V}_{W}\left[\mathbf{Y}_{i\cdot}^{k
}\right],$
where variance $\mathbb{V}_{W}\left[\mathbf{Y}_{i\cdot}^{k}\right]$ for the $k^{th}$ test sample using entries of the posterior prediction matrix across columns, generated using posterior weights, for $i^{th}$ output variable. In principle, the MPV captures the expected variance in the predictions due to the posterior distribution of BNN weights. We hypothesize that with a constant multiplier $\alpha>1$,

$$ $\displaystyle\alpha\,\text{MPV}\geqslant\mathbb{V}_{e}=\mathbb{E}_{M}\left[ \mathbb{V}_{W}[e]\right]+\mathbb{E}_{W}\left[\mathbb{V}_{M}[e]\right]\geqslant \mathbb{V}_{|e|},$ (5) $$

where $\mathbb{E}_{M}$ and $\mathbb{E}_{W}$ denote expectations with respect to $M$ testing samples and posterior weight samples, respectively, and $\mathbb{V}_{M}[e]$ and $\mathbb{V}_{W}[e]$ denote the variance of the error with respect to $M$ testing samples and posterior weight samples, respectively. The equality in equation 5 follows from the law of total variance Blitzstein & Hwang (). $\mathbb{V}_{|e|}$ represents the variance of the absolute value of the error, which is lower than the variance of the error.

Notice that the first term of the TVE, $\mathbb{E}_{M}\left[\mathbb{V}_{W}[e]\right]$, is independent of the labeled testing dataset because the true output $y$ is constant with respect to posterior weight samples; thus, $\mathbb{V}_{W}[e]=\mathbb{V}_{W}[y-y^{t}]=\mathbb{V}_{W}[y^{t}]$. Furthermore, from the definition of MPV, we have $\text{MPV}=\mathbb{E}_{M}\left[\mathbb{V}_{W}[e]\right]$. As an example, if $\mathbb{E}_{W}\left[\mathbb{V}_{M}[e]\right]\leqslant\text{MPV}$, our hypothesis in equation 5 holds with $\alpha=2$. Consequently, we can use $2\times\text{MPV}$ as an upper bound for $\mathbb{V}_{e}$ in the Theoretical Bernstein bound (see Theorem 3 in Appendix D), which gives the error bound

$$ $\varepsilon=\sqrt{\frac{4\times\text{MPV}\log\left({1}/{\delta}\right)}{M}}+ \frac{2R\log\left({1}/{\delta}\right)}{3M},$ $$

which is better than the Empirical Bernstein bounds. The hypothesis in equation 5 and the corresponding constant $\alpha$ can be computed by using application specific information or performing a meta-study like in Section 5.

The strength of this approach is that we do not require labeled testing samples to calculate MPV, thus incurring no additional computational burden from generating labels. Also, note that this constitutes an advantage of BNNs since MPV information is readily available with BNNs but cannot be obtained from DNN-based optimization proxies.

In the next section, we perform a meta-study using different BNN models on different test cases to show the performance of our proposed learning architecture as well as demonstrate that hypothesis equation 5 indeed holds for the proposed optimization proxy learning problems.

## 5 Numerical Results and Discussion

We test the proposed method on the Alternating Current Optimal Power Flow (ACOPF) problem, essential for the economic operation of electrical power grids Molzahn et al. (). Efficient ACOPF proxies can mitigate climate change by enabling higher renewable energy integration, improving system efficiency by minimizing losses and emissions, and enhancing grid resiliency against extreme weather conditions Rolnick et al. (). ACOPF is a constrained optimization problem with nonlinear equality constraints and double-sided inequality bounds. It aims to find the most cost-effective generator set points while satisfying demand and adhering to physical and engineering constraints. The inputs are active and reactive power demands; outputs include generator settings, voltage magnitudes, and phase angles at each bus. We adopt the standard ACOPF formulation Babaeinejadsarookolaee et al. (); Park & Van Hentenryck (); Coffrin et al. () and benchmark our method using the open-source OPFDataset from Torch_Geometric, which provides numerous solved ACOPF instances (see Lovett et al. ()).

In our results, ‘Gap%’ denotes the average relative optimality gap compared to the objective values in the labeled testing instances. ‘Max Eq.’ and ‘Mean Eq.’ represent the maximum and mean equality gaps over all equality constraints, while ‘Max Ineq.’ and ‘Mean Ineq.’ indicate the same for inequality gaps in per unit, all averaged over testing instances. We compare our proposed method with the following state-of-the-art baseline supervised learning models available in the literature using the same labeled dataset and training time constraints, utilizing AI4OPT’s ML4OPF package AI4OPT () (network and hyper-parameter details are in Appendix B):

- •
Naïve MAE and Naïve MSE (Supervised): Use $l_{1}$-norm and $l_{2}$-norm loss functions, respectively, to measure differences between predicted and actual optimal solutions Park & Van Hentenryck (), incorporating a bound repair layer with a sigmoid function. The bound repair layer ensures that inequality constraints are always satisfied.
- •
MAE + Penalty, MSE + Penalty, and LD (Supervised): Add penalty terms for constraint violations to the naïve MAE or MSE loss functions Park & Van Hentenryck (). The Lagrangian Duality (LD) method applies the $l_{1}$-norm as outlined in Fioretto et al. (); Park & Van Hentenryck (), and also uses a bound repair layer with a sigmoid function.

We exclude self-supervised constrained optimization methods like Primal-Dual Learning (PDL) Park & Van Hentenryck () and DC3 Donti et al. () due to their significantly higher training times and computational demands, which violate the premise of this paper. For example, PDL requires over 125 minutes of training time for the ACOPF problem on a 118-node power network using a Tesla RTX6000 GPU. Methods that require solving alternating current power flow to recover solutions (e.g., Zamzam & Baker ()) are also excluded, as they result in high prediction times compared to DNN or BNN forward passes(^5^55see prediction time studies in Donti et al. () which suggest 10 times higher prediction time with power flow based projections (0̃.080 sec. compared to 0.001 sec. for DNN and 0.003 sec. per testing instance for proposed BNNs).).

**Table 1: Comparative performance results for the ACOPF Problem for ‘case57’ with 512 labeled training samples, 2048 unlabeled samples, and $T_{\text{max}}=600$ sec.**
| Method | Gap% | Max Eq. | Mean Eq. | Max Ineq. | Mean Ineq. |
| --- | --- | --- | --- | --- | --- |
| Sandwich BNN SvP (Ours) | 0.928 | 0.027 | 0.006 | 0.000 | 0.000 |
| Sandwich BNN (Ours) | 0.964 | 0.045 | 0.005 | 0.000 | 0.000 |
| Supervised BNN SvP (Ours) | 3.195 | 0.083 | 0.011 | 0.000 | 0.000 |
| Supervised BNN (Ours) | 3.255 | 0.130 | 0.011 | 0.000 | 0.000 |
| Naïve MAE | 4.029 | 0.518 | 0.057 | 0.000 | 0.000 |
| Naïve MSE | 3.297 | 0.541 | 0.075 | 0.000 | 0.000 |
| MAE + Penalty | 3.918 | 0.370 | 0.037 | 0.000 | 0.000 |
| MSE + Penalty | 3.748 | 0.298 | 0.039 | 0.000 | 0.000 |
| LD + MAE | 3.709 | 0.221 | 0.033 | 0.000 | 0.000 |

To demonstrate the effectiveness of the proposed BNN learning methods, we conduct simulation studies on both our models and the ML4OPF models using an M1 Max CPU with 32 GB RAM, without any GPU. This setup highlights performance improvements due to the learning mechanism rather than computational power. We propose two classes of different models as:

- •
Supervised BNN and Supervised BNN SvP: Standard BNN learning with labeled data, utilizing mean prediction and Selection via Posterior (SvP), respectively. The network uses ReLU activation and no *bound repair layer*.
- •
Sandwich BNN and Sandwich BNN SvP: The proposed Sandwich BNN trained with labeled and unlabeled data as discussed in Section 3. Unsupervised training utilizes four times the number of labeled data samples. The network uses ReLU activation and no *bound repair layer*.

Importantly, we intentionally keep the BNN architecture unoptimized, constructing it using four different sub-networks (one for each of the four ACOPF outputs: real power generation, reactive power generation, voltage magnitude, and voltage angle). Each sub-network has two hidden layers with the number of hidden neurons equal to 2 $\times$ input size. In the Sup stage of the Sandwich BNN, both weights and biases are updated, whereas in the UnSup stage, only the weights are modified via SVI. Best model out of five random trials is selected.

Tables 1 and 2 present the comparative performance of different methods for solving the ACOPF problem on the ‘case57’ and ‘case118’ test cases, containing 57 and 118 nodes, respectively. For ’case57’, the Sandwich BNN SvP method achieves the best Max Eq. performance (0.027), outperforming all other methods, including the standard Sandwich BNN, without compromising other metrics. All the methods proposed in this paper outperform the best DNN results, typically achieved with the LD+MAE model (last row in the tables). Similar trends are observed for the ‘case118’ as well.(^6^66See Appendix C for similar tabular results and discussion of trends on larger test instances.)

It is important to contextualize the significance of these numerical improvements. In ACOPF problems, cost values are in USD, with the mean cost for ‘case118’ being $97,000 or 9.7 in the per-unit system. Therefore, a 1% Gap corresponds to an expected difference of $970 across the testing instances. A ‘Max Eq.’ value of 0.08 implies a maximum expected power imbalance of 8.0 Megawatts among all 118 nodes of ‘case118’. Thus, reducing the Max Eq. from 1.284 with the LD+MAE model to 0.089 with our Sandwich BNN SvP model represents a significant improvement.

**Table 2: Comparative performance results for the ACOPF Problem for ‘case118’ with 512 labeled training samples, 2048 unlabeled samples, and $T_{\text{max}}=600$ sec.**
| Method | Gap% | Max Eq. | Mean Eq. | Max Ineq. | Mean Ineq. |
| --- | --- | --- | --- | --- | --- |
| Sandwich BNN SvP (Ours) | 1.484 | 0.089 | 0.018 | 0.008 | 0.000 |
| Sandwich BNN (Ours) | 1.485 | 0.100 | 0.016 | 0.008 | 0.000 |
| Supervised BNN SvP (Ours) | 1.568 | 0.147 | 0.022 | 0.013 | 0.000 |
| Supervised BNN (Ours) | 1.567 | 0.205 | 0.020 | 0.013 | 0.000 |
| Naïve MAE | 1.638 | 2.166 | 0.187 | 0.000 | 0.000 |
| Naïve MSE | 1.622 | 3.780 | 0.242 | 0.000 | 0.000 |
| MAE + Penalty | 1.577 | 1.463 | 0.102 | 0.000 | 0.000 |
| MSE + Penalty | 1.563 | 2.637 | 0.125 | 0.000 | 0.000 |
| LD + MAE | 1.565 | 1.284 | 0.083 | 0.000 | 0.000 |

Figure: Figure 2: Growth trajectories of performance metrics for ACOPF across system sizes for different methods. Detailed results for ‘case500’ and ‘case2000’ are given with Table 6 and Table 7 respectively, in Appendix C.
Refer to caption: x1.png

Figure: Figure 3: Empirical study comparing total variance in error $\widehat{\mathbb{V}}_{e}$ with $2\times\text{MPV}$ across different cases of ACOPF and the proposed learning mechanisms.
Refer to caption: x2.png

Figure: Figure 4: Comparison of voltage magnitude and voltage angle error bounds (in logarithmic scale) across bus indices for ‘case57’ (top row) and ‘case118’ (bottom row). The plot illustrates that PCBs using theoretical Bernstein bounds with 2 $\times$ MPV from hypothesis equation 5 are tightest among all PCBs. We consider $\delta=0.95$ and 1000 *out-of-sample* testing data points i.e. $M=1000$.
Refer to caption: x3.png

Figure 2 illustrates the growth of various metrics with increasing system size while keeping training resources constant. The proposed BNN methods exhibit significantly lower scaling in the expected maximum equality gap. Although Sandwich BNN SvP shows slightly higher scaling inequality gaps, the relative improvement in Max Eq. far outweighs these minor drawbacks. Notably, for ‘case500’ (see Table 6 in the Appendix C), the expected maximum power imbalance is below 0.5% of the mean real power demand ($1.7\times 10^{4}$ MW) and power grids already are equipped with spinning reserves (see Ela et al. ()) that have reserve capacity to handle these imbalances. This is a significant improvement over the DNN models which have much higher ‘Mean Eq.’ values. Moreover, the ‘Max Ineq.’ growth could be easily suppressed by incorporating *bound repair* layers, as used in DNN models in ML4OPF AI4OPT ().

Next, we present results for Proabilistic Confidence bounds, described in Section 4. Figure 3 shows that $2\cdot\text{MPV}$ consistently serves as an empirical upper bound for total variance in error, validating its robustness across models and system configurations(^7^77Total variance in error is assumed to stabilizing with 1000 testing samples (see Figure 6 in Appendix C).). In Figure 4, the theoretical Bernstein bounds using $\mathbb{V}_{e}=2\cdot\text{MPV}$ provide tight, practical bounds, whereas Hoeffding’s bounds are overly conservative and not useful for grid operations. For example, in ‘case118’, the Bernstein bound ensures a probabilistic guarantee on voltage constraint satisfaction, such that a predicted voltage between 0.91–1.09 pu guarantees no violations within the ACOPF limits of 0.90–1.10 pu, i.e., the maximum value of the Bernstein bound on the error is 0.010 pu across all nodes. Compared to Hoeffding’s bound ($0.064$ pu) or the empirical Bernstein bound ($0.018$ pu), the Bernstein bound ($0.010$ pu) is far tighter and more practical, highlighting the benefits of BNNs for optimization proxies. The error bounds for the ‘case500’ is provided in Fig. 6 of the Appendix C. Finally, we note that both Hoeffding’s and empirical Bernstein bounds can also be obtained by testing DNN models across $M$ samples(^8^88A form of MPV can be obtained via Ensemble DNNs Ganaie et al. (), however, it will lead to very high computational requirement compared to the BNN.).

## 6 Conclusions

In conclusion, this paper introduces a semi-supervised Bayesian Neural Network (BNN) approach to address the challenges of high labeled data requirements and limited training time in learning input-to-output maps for constrained optimization problems. The proposed Sandwich BNN method incorporates unlabeled data through input data augmentation, ensuring constraint feasibility without relying on a large number of labeled instances. We provide tight confidence bounds by utilizing Bernstein’s inequality, enhancing the method’s practical applicability. Results show that BNNs outperform DNNs in low-data, low-compute settings, and the Sandwich BNN more effectively enforces feasibility without additional computational costs compared to supervised BNNs.

#### Acknowledgments

Use unnumbered third level headings for the acknowledgments. All
acknowledgments, including those to funding agencies, go at the end of the paper.

## Appendix A ACOPF Problem: Modeling and Dataset

The alternating current optimal power flow (ACOPF) problem is essential for power grid operations and planning across various time scales. It determines generator set-points for real and reactive power that minimize generation costs while meeting power demand and satisfying physical and operational constraints on output variables. We follow the ACOPF model given in PowerModels Coffrin et al. (), and take the dataset from Torch_geometric (Lovett et al. ()).

**Table 3: Sets for ACOPF**
| $N$: buses | $G$: generators, generators at bus $i$ ($G_{i}$) |
| --- | --- |
| $E$: branches (forward and reverse orientation, $E_{R}$) | $S$: shunts, shunts at bus $i$ ($S_{i}$) |
| $L$: loads, loads at bus $i$ ($L_{i}$) | $R$: reference buses |

## Appendix B Experimental Setting Details

### B.1 Architectures

Supervised BNN and Sandwich BNN: For ACOPF problem we use real and reactive power demands as input while predicting all decision and state variables using separate networks

- –
Input real and reactive power demands: Two times the number of nodes having non-zero load
- –
Output real and reactive power generation setpoints, voltage magnitude and voltage angle at each node: Two times the number of generators + Two times the number of nodes

### B.2 Hyper-parameters

#### B.2.1 ML4OPF :

config: The optimization is performed using the Adam optimizer with a learning rate of $1\times 10^{-4}$ (taken from Park & Van Hentenryck ()). All networks have two hidden layers, each with a size of $2\times\text{Number of outputs}$. For hidden layers, ReLU activation function is selected while, the bound repair mechanism is handled using a Sigmoid function Park & Van Hentenryck ().

penalty_config: Multiplier of $1\times 10^{-2}$, with no excluded keys.

ldf_config: Step size $1\times 10^{-2}$, and a kick-in value of 0. The LDF update frequency is set to 1, and no keys are excluded from the updates.

All other hyperparameters are set at default ML4OPF values AI4OPT ().

#### B.2.2 Proposed BNNs

For all simulations, maximum training time 10 min., $T_{max}=600$ seconds, per round time $T_{r}$ is 200 seconds with 40-60 split between Sup and UnSup models ($T_{s}=80$ amd $T_{u}=120$ seconds). For VI, we use MeanFieldELBO loss function from Numpyro and Adam optimizer with a initial learning rate rate of $1\times 10^{-3}$, and decay rate of $1\times 10^{-4}$, both of which are selected via grid search.

We use decay schedule as $\text{initial learning rate}/(1+\text{decay rate}\times\text{step})$ within a Sup or UnSup model, while simple step decay among different rounds of Sandwich BNN. Learning is done with single batch for all models and each of the BNN weight and bias is parameterized using mean and variance parameters of a Gaussian distribution, with mean field assumption i.e. independent from each other. The prior for each weight and bias has zero mean and $10^{-2}$ variance while likelihood noise variance is initialized with $10^{-5}$ mean and $10^{-6}$ variance for Sup and fixed at $10^{-10}$ for UnSup.

### B.3 Structure of posterior prediction matrix (PPM)

$$ $\displaystyle\mathbf{Y}\equiv\begin{bmatrix}y_{11}&\cdots\cdots\cdots&y_{1H}\\ \vdots&\ddots\ddots\ddots&\vdots\\ \cdots&y_{\texttt{Variable,Sample}}&\cdots\\ \vdots&\ddots\ddots\ddots&\vdots\\ y_{O1}&\cdots\cdots\cdots&y_{OH}\end{bmatrix}\quad\begin{bmatrix}O:\text{ Number of variables}\\ H:\text{Number of posterior samples}\end{bmatrix}$ $$

## Appendix C Additional Results

Here, we present the complete results analogous to Tables 1 and 2 for the larger test cases: ‘case500’ and ‘case2000’ with 500 and 2000 nodes, respectively, in Tables 6 and 7. In general, it is clear from the tables that the approaches presented in this paper outperform the state-of-the-art DNN-based approaches when limited training time and compute resources are provided. Between the supervised and the sandwich BNN models, it appears that there is no clear winner. Table 6 indicates that the supervised BNN outperforms the sandwich BNN model on the Max. Ineq. gap metric for the ‘case500’ whereas the trend is reversed for the case2000’ in Table 7. This minor variation is attributed to the fact that the maximum training time of $T_{\max}=600$ sec. is insufficient for both cases, and with more training time, these trends should look similar to the ones in Tables 1 and 2.

Another critical observation for both cases is that the ‘Max. Eq.’ value is substantially high even for the proposed best model (in Tables 6 and 7, respectively). However, it is clear that the proposed BNN proxies are better than standard DNN models with an order-of-magnitude difference. On the surface, readers may dismiss the efficacy of the proposed BNN models due to these large values. Still, these values have to be examined in conjunction with the error bounds on the voltage magnitude and voltage angles in Fig. 6. Examined together, the BNN model predictions have very low errors for the voltage magnitude and angles, and it is very much possible to develop a computationally inexpensive projection algorithm that projects this prediction onto the feasibility set of the ACOPF problem on lines of Zamzam & Baker (). This procedure would further reduce the Max. Eq. values and makes the projected solution usable. This is a minor detail regularly tackled in a power grid context and is not dealt with extensively in this paper.

Finally, Figure 6 shows the stabilization of the mean errors and the MPV with varying testing samples and posterior samples, respectively. These plots validate the number of testing and posterior samples chosen for generating the results presented in Section 5.

**Table 6: Comparative performance results for the ACOPF Problem for case500 with 512 labeled training samples, 2048 unlabeled samples and $T_{max}=$ 600 sec.**
| Method | Gap% | Max Eq. | Mean Eq. | Max Ineq. | Mean Ineq. |
| --- | --- | --- | --- | --- | --- |
| Sandwich BNN SvP (Ours) | 2.009 | 0.770 | 0.066 | 0.190 | 0.000 |
| Sandwich BNN (Ours) | 2.002 | 0.781 | 0.056 | 0.191 | 0.000 |
| Supervised BNN SvP (Ours) | 1.191 | 2.204 | 0.088 | 0.141 | 0.000 |
| Supervised BNN (Ours) | 1.191 | 2.401 | 0.072 | 0.140 | 0.000 |
| Naïve MAE | 1.208 | 20.818 | 0.905 | 0.000 | 0.000 |
| Naïve MSE | 1.201 | 24.089 | 1.031 | 0.000 | 0.000 |
| MAE + Penalty | 1.205 | 11.833 | 0.580 | 0.000 | 0.000 |
| MSE + Penalty | 1.215 | 10.314 | 0.475 | 0.000 | 0.000 |
| LD + MAE | 1.279 | 11.166 | 0.532 | 0.000 | 0.000 |

**Table 7: Comparative performance results for the ACOPF Problem for case2000 with 512 labeled training samples, 2048 unlabeled samples and $T_{max}=$ 600 sec.**
| Method | Gap% | Max Eq. | Mean Eq. | Max Ineq. | Mean Ineq. |
| --- | --- | --- | --- | --- | --- |
| Sandwich BNN SvP (Ours) | 0.514 | 5.114 | 0.324 | 0.196 | 0.000 |
| Sandwich BNN (Ours) | 0.503 | 5.409 | 0.262 | 0.187 | 0.000 |
| Supervised BNN SvP (Ours) | 0.461 | 4.107 | 0.238 | 0.917 | 0.000 |
| Supervised BNN (Ours) | 0.451 | 4.225 | 0.193 | 0.922 | 0.000 |
| Naïve MAE | 56.365 | 43.529 | 4.392 | 0.000 | 0.000 |
| Naïve MSE | 56.366 | 43.085 | 4.261 | 0.000 | 0.000 |
| MAE + Penalty | 56.349 | 19.591 | 1.055 | 0.000 | 0.000 |
| MSE + Penalty | 56.377 | 12.592 | 0.682 | 0.000 | 0.000 |
| LD + MAE | 57.257 | 18.870 | 0.647 | 0.000 | 0.000 |

Figure: Figure 5: Convergence of mean error, MPV, total variance in error with respect to number of testing samples, and number of posterior samples.
Refer to caption: x5.png

## Appendix D Concentration Bounds

This section presents three bounds used in Section 4. Here, $X$ represents a generic random variable and is not related to the optimization proxy variables. Since these are well-known inequalities, we omit the proofs, which can be found in the respective references.

###### Theorem 1 (Hoeffding’s) .

Hoeffding ()
Let $X_{1},\ldots,X_{M}$ i.i.d. random variables and suppose that $|X_{i}|\leqslant R$ with expectation $\mathbb{E}(X_{i})$, and let $\bar{X}_{M}=\frac{1}{M}\sum_{i=1}^{M}X_{i}$. Then, with probability at least $1-\delta$,

$$ $\big{|}\bar{X}_{M}-\mathbb{E}(X_{i})\big{|}\leqslant R\sqrt{\frac{\log(2/ \delta)}{2M}}.$ $$

.

###### Theorem 2 (Empirical Bernstein) .

Audibert et al. (); Mnih et al. ()
Let $X_{1},\ldots,X_{M}$ be i.i.d. and suppose that $|X_{i}|\leqslant R$, expectation $\mathbb{E}(X_{i})$ and let $\bar{X}_{M}=\frac{1}{M}\sum_{i=1}^{M}X_{i}$. With probability at least $1-\delta$,

$$ $\big{|}\bar{X}_{M}-\mathbb{E}(X_{i})\big{|}\leqslant\sqrt{\frac{2\widehat{ \mathbb{V}}\log(3/\delta)}{M}}+\frac{2R\log(3/\delta)}{M}$ $$

where, $\widehat{\mathbb{V}}=(1/M)\sum^{M}_{i=1}(X_{i}-\bar{X}_{M})^{2}$ is empirical variance.

###### Theorem 3 (Bernstein) .

Sridharan ()
Let $X_{1},\ldots,X_{M}$ be i.i.d. and suppose that $|X_{i}|\leqslant R$, mean $\mathbb{E}(X_{i})$ and ${\mathbb{V}}=\mathrm{Var}(X_{i})$. With probability at least $1-\delta$,

$$ $\big{|}\bar{X}_{M}-\mathbb{E}(X_{i})\big{|}\leqslant\sqrt{\frac{2\mathbb{V} \log(1/\delta)}{M}}+\frac{2R\log(1/\delta)}{3M}$ $$

**Table 8: Error bound $\varepsilon$ in PCBs, provided by different concentration inequalities.**
| Hoeffding’s | Empirical Bernstein | Bernstein |
| --- | --- | --- |
| $R\sqrt{\frac{\log\left({2}/{\delta}\right)}{2M}}$ | $\sqrt{\frac{2\widehat{\mathbb{V}}_{e}\log\left({3}/{\delta}\right)}{M}}+\frac{<br>3R\log\left({3}/{\delta}\right)}{M}$ | $\sqrt{\frac{2(2\times\text{MPV})\log\left({1}/{\delta}\right)}{M}}+\frac{2R<br>\log\left({1}/{\delta}\right)}{3M}$ |

###### Theorem 1 (Hoeffding’s) .

###### Theorem 2 (Empirical Bernstein) .

###### Theorem 3 (Bernstein) .

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: 1.	How many data points were used to train the DNN models?
2.	I wonder if the authors tried to use Sandwich learning to train DNNs. As it reduces the dependence on labelled datasets, it can be a good improvement to scale up to larger systems with limited observations. 
3.	Some papers in the literature call a shallow NN a model with two hidden layers i.e. input -> hidden and hidden -> output. Others refer shallow NNs as one layered models. If it is the case of the former, to name it as DNN it must have more than two layers. I wonder which is the case in this study. If a shallow NN was trained, then it deeper models can have better advantages.
soundness: 2
strengths: 1.	The main advantage is that the BNN seem to have some performance advantages over DNNs for small and large systems even with training datasets as small as 512 observations. While the optimality gaps are close among all methods, there are notable power balance gap differences between BNN and DNN methods.
2.	In case of large systems such as 2000-bus system, with 512 observations, while DNN training fails, all BNN models still able to perform well. This is particularly useful when it is computationally expensive to generate datasets.
3.	Another contribution is using unsupervised loss term in the training loop, which can work like regularization term, focusing the attention of training to model parameters that would generate more realistic (feasible) solutions. The sandwich training that alternates between supervised and unsupervised data is akin to Physics Informed Neural Network training proposed by [1]. While PINN training contains all KKT conditions in one loss function, the proposed Sandwich learning alternates between two loss functions.

[1] Nellikkath, Rahul, and Spyros Chatzivasileiadis. "Physics-informed neural networks for minimising worst-case violations in dc optimal power flow." 2021 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm). IEEE, 2021.
summary: The paper proposes Bayesian Neural Networks (BNNs) for solving the OPF problem under uncertain demand. It employs a novel semi supervised training procedure that reduces the dependence of the training to labelled input-output pairs. The study also suggests using Bernstein bound with Mean Predictive Variance to assess the BNN model performance on out-of-sample inputs.

The authors consider two BNN training methods: a supervised approach using labelled data and a semi-supervised approach, or sandwich learning, which alternates between supervised and unsupervised loss functions iteratively. The latter enforces feasibility by utilizing a function that takes the value 0 if the constraints are satisfied. 

The model was applied to 57-, 118-, 500- and 2000-bus test cases and compared against 5 alternative methods in the literature.
weaknesses: 1.	The optimality gaps among different methods are very close for 118- and 500-bus cases. The only exception is 57-bus system where sandwich learning shows some improvement. It looks the only consistent advantage is less power balance violations.
2.	It is not clear if sandwich learning improves BNN model performance. The differences among proposed BNN approaches seem marginal and is not consistent among all test cases.
3.	If I understand correctly, the DNNs in the study are also trained with 512 data points and the training was terminated after 600 seconds (please correct me at this point if I missed the relevant part of the script). My main concern is that training DNNs (with two hidden layers and n_hidden = 2 x input size) with 512 observations is not fair. DNNs are universal function approximators with number of linear facets are determined by hidden layer depth and width [2] and the training must be carried out with dense enough datasets for a good estimation especially in case of larger systems. I understand this is one of the advantages of using BNNs but larger training datasets can still be feasible to construct. 
4.	It is not surprising that in case of 2000-bus system, the training would fail with a small dataset and short training time because the NN model (with hidden layer width of 2 x input size) has too many trainable parameters. The authors could find the required training size for DNNs to perform as well as BNNs as a proof of scalability of the latter.

[2] Montufar, G. F., Pascanu, R., Cho, K., & Bengio, Y. (2014). On the number of linear regions of deep neural networks. Advances in neural information processing systems, 27.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: See above
soundness: 2
strengths: - The paper is relatively well-organised and easy to understand;
- The paper offers a fairly well-executed application/combination of Bayesian semi-supervised learning techniques to a realistic real-world problem in constrained optimization;
- The Sandwich BNN framework idea is fairly interesting and seems to have direct applicability to general time-sensitive and resource-limited engineering optimization problems
summary: This paper proposes a Bayesian neural network-based semi-supervised learning framework for tackling constrained optimization problems. The authors target engineering problems where labelled data and computational resources are limited. The proposed approach alternates between a supervised learning step which minimizes the cost of the optimization problem and an unsupervised learning step which enforces the related constraints. The authors claim that their approach outperforms conventional methods while significantly reducing equality and feasibility gaps.
weaknesses: - As far as I can tell, the work is primarily applied and there is limited technical novelty, which could reduce the community's interest in this paper;
- The paper emphasizes the benefits of Bayesian neural networks for uncertainty estimation but offers no uncertainty calibration analysis or comparisons with other viable uncertainty estimation methods. It is important to note that Bayesian neural networks tend to underestimate variance (partly) due to the mean-field assumption;
- The assumption that the problem has a feasible solution strikes me as quite strong in the general case, for example in complex highly non-convex settings. I would be interested in hearing the author's thoughts on this;
- There is very limited discussion about the drawbacks and/or limitations of the approach.

In my view, the paper could be improved by broadening comparisons with other Bayesian and non-Bayesian uncertainty methods and clarifying and testing the feasibility assumptions. The apparent lack of significant innovation could reduce the impact of the paper, as similar methods (e.g., ensembles or conventional semi-supervised frameworks) could potentially achieve comparable results without the additional complexity of Bayesian inference.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. How does the number of labeled or unlabeled training data affect the BNN performance?
2. How does the number of posterior samples affect the constraint violation?
soundness: 2
strengths: 1. The work considers a practical and important scenario of solving constrained optimization problems when labeled data is limited and model training times are restricted.
2. The authors derive tight confidence bounds for the testing error by utilizing Bernstein's inequality.
3. The experiments conducted on the case 2000 demonstrate the scalability of the proposed approach.
summary: The paper introduces a semi-supervised learning framework based on BNN to solve constrained optimization problems when labeled data is limited and model training times are restricted. 
The authors propose a sandwich-style training approach alternating between supervised and unsupervised learning. Additionally, they employ BNNs to generate multiple predictions and improve feasibility through a selection process. Experimental results demonstrate that BNNs outperform DNNs in settings with low data and limited computational resources.
weaknesses: 1. While the main challenges addressed in this work are limited labeled data and restricted model training times, it is unclear how BNNs specifically contribute to addressing these challenges. The sandwich-style semi-supervised training approach proposed by the authors can be applied to any NN structure. 
    - Training BNNs is computationally more expensive than regular NNs. The authors should discuss it and present the complexity explicitly.
    - In section 3.1., the authors propose a selection via posterior strategy to reduce the constraint violation. It is unclear what the benefits of BNN are in this strategy since one can add Gaussian noise to regular DNN prediction and select the one with minimum constraint violation. Theoretically, how does the equality constraint violation decrease with the increased generated samples?
    - In Section 4, the authors derive confidence bound for the testing error using the MPV as a proxy for the TVE in the Bernstein bound. They hypothesize an inequality (line 286) without providing sufficient justification. The authors should explicitly state and discuss the assumptions made in deriving this confidence bound.
    - The authors should discuss and compare their confidence bound with existing generalization analyses or confidence bounds for DNNs based on the number of training samples, such as those presented in [1].

2. The authors propose an alternated training approach where supervised and unsupervised learning are performed in separate iterations. However, the benefits of this approach are unclear. Since one can easily combine the supervised and unsupervised loss together at each iteration, it may save more training time. 

3. Overall, this work combines sandwich training, BNN, and SvP, so a comprehensive ablation study is needed to evaluate the individual contributions and importance of each component. 

4. In experiments, the authors only include supervised training methods as baselines, excluding self-supervised and primal-dual learning approaches due to their higher training times and computational demands.
    - The sandwich training approach can also be applied to regular NNs, serving as a meaningful baseline.
    - There are efficient unsupervised approaches for solving AC-OPF problems that the authors did not discuss in the related work section or compare against in their experiments [2].

[1] Kawaguchi, K., Kaelbling, L. P., & Bengio, Y. (2017). Generalization in deep learning. arXiv preprint arXiv:1710.05468, 1(8).

[2] Huang, W., & Chen, M. (2021). DeepOPF-NGT: Fast no ground truth deep learning-based approach for AC-OPF problems. In ICML 2021 Workshop Tackling Climate Change with Machine Learning.


I will adjust my scores if the concerns are addressed.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: Please refer to the section on weaknesses.
soundness: 2
strengths: The strengths of this paper lie in its development of a semi-supervised Bayesian Neural Network (BNN) approach that addresses constrained optimization challenges under strict time and data constraints. First, the choice of BNNs over DNNs enhances uncertainty quantification, facilitating more reliable predictions by integrating prior beliefs, and is novel. The introduction of a novel Sandwich learning method further strengthens the model by enforcing feasibility through unlabeled data, bypassing the need for additional labeled instances. Moreover, the use of predictive variance within the BNN framework enables the formulation of tight expected error bounds.
summary: The authors introduces a semi-supervised Bayesian Neural Network (BNN) framework to efficiently solve constrained optimization problems common in engineering applications like energy networks, where uncertain parameters and limited labeled data pose computational challenges. By alternating between supervised cost-minimization with labeled data and unsupervised constraint satisfaction with unlabeled data, the model achieves a tenfold reduction in equality gaps and halves feasibility and optimality gaps compared to standard BNNs and deep neural networks. Additionally, a novel Selection via Posterior (SvP) scheme leverages BNN uncertainty estimates to further minimize errors, while the framework’s probabilistic confidence bounds offer a scalable solution adaptable to various high-stakes optimization problems with minimal labeled data requirements.
weaknesses: - The authors should correct the citation formatting throughout the paper, particularly the use of \citet and \citep. For example, the citation of Ibrahim et al. (2020) should be formatted as (Ibrahim et al. 2020) rather than including it in the narrative.
- The overall presentation of the study is weak and requires enhancement for improved clarity and impact.
- A comparison against methods that require more training time is necessary. Since the neural networks proposed by Park & Van Hentenryck (2023) and others do not necessitate extensive training time, it is crucial to understand the implications of low computational requirements. Although low compute restrictions during test time are typically essential for inference on new examples, we need to train fewer models and thus it may render such restrictions (e.g., a hard limit of 10 minutes of training time on a single CPU core) unnecessary.
- What is the justification for including equation 1d as part of the constraints?
- The assertion that constructing the feasibility dataset Df​ incurs no additional computational cost should be revisited. While input sampling may indeed be inexpensive, obtaining feasible solutions can be resource-intensive, necessitating a more accurate representation.
- The authors should clarify why they did not compare their approach with the primal-dual self-supervised learning method proposed by Park & Van Hentenryck (2023). Additionally, the methods of DC3, and Zamzam & Baker (2020) should be included in the comparison. A new table could have been created to compare the training times of these methods, as it is possible to obtain outputs from these models without fitting them completely.
- Completely omitting state-of-the-art baselines does not appear to be a prudent choice. The authors should include the results of these methods while indicating that they require more time and thus cannot be used for direct comparison.
- Additionally, the authors have not accounted for the supervised dataset creation time within the 10-minute limit. Since self-supervised methods do not necessitate this step, the current comparison may be considered unfair. The authors should revise the training time to reflect a total of 10 minutes for training plus T minutes for generating true labels for the supervised dataset. This revised training time should then be applied uniformly to both the supervised and self-supervised methods, even when the latter do not require true labels.
- Is the statement, “Moreover, unsupervised methods still require testing data and consequently the associated data generation time in order to perform validation and provide confidence bounds on error with respect to true solution,” not applicable to the proposed method? Although the proposed method does not require data generation for confidence bounds, it does require labeled training data, which is not needed for self-supervised methods. Additionally, to compute the gap and constraint violations, wouldn’t the proposed method also need true labels?
- The manuscript contains several grammatical mistakes that should be addressed, including but not limited to:
    - "The fundamental idea of this training method is to update the network weights and biases through multiple rounds of training in which each round alternates between using the labeled dataset."
    - "Next, we present results for Probabilistic Confidence bounds, described in Section 4. Figure 3 shows that..."
    - "In Bayesian Neural Network (BNN) literature, the standard approach is to use the mean posterior prediction Ep​(w∣D)\[fw​(xt​)] for a test input xt​." Please clarify the context for "a standard approach."

Meta-review: **Summary**


The paper introduces a semi-supervised Bayesian Neural Network (BNN) framework designed to tackle constrained optimization problems, particularly the  optimal power flow (OPF) problem with uncertain demands and limited labeled data. The approach alternates between supervised learning, which minimizes the optimization problem's cost, and unsupervised learning, which enforces related constraints, referred to as sandwich learning. This method significantly reduces feasibility and optimality gaps by employing a novel training procedure that decreases reliance on labeled data. The authors also implement a Selection via Posterior (SvP) scheme that uses BNN uncertainty estimates to enhance model performance and ensure feasibility. Experimental results across various test cases, including 57-, 118-, 500-, and 2000-bus networks, demonstrate that this BNN framework outperforms traditional deep neural networks (DNNs) and other conventional methods, especially in scenarios characterized by sparse data and computational resource constraints.

**Strengths**

The reviewers unanimously highlighted several strengths of the proposed framework:
* The paper is well-organised and easy to follow.
* The use of BNN seems to have some performance advantages over DNNs for small and large systems even with training datasets as small as 512 observations.
* The proposed unsupervised loss term in the training loop, and the sandwich training scheme used in the paper are interesting and add practical value to the proposed solution.


**Weaknesses**

Several core weaknesses was brought up by the reviewers. These include: 
* The paper's representation could be enhanced. 
* The lack of comparison with more recent relative baselines, including self-supervised and primal-dual learning approaches due to their higher training times and computational demands. The paper could benefit from reporting such baselines and their corresponding training times.
* While this work focuses on addressing the challenges of limited labeled data and restricted model training times, it is not clearly explained how Bayesian Neural Networks (BNNs) specifically contribute to overcoming these challenges.
* Limited discussions on limitations of the proposed solution.

**Conclusion** 

The majority of reviewers recognize the importance of the problem addressed by the paper but criticize the experimental setup and the paper's inadequate positioning, including a lack of rigorous comparisons with recent baselines. Despite a substantial rebuttal from the authors, this did not alter the predominantly negative perception held by the reviewers. I agree with the reviewers and vote for rejection of the paper.

Decision: Reject