SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ExploRation: To simulate the exploration behaviour, a state that does not appear optimal but has the potential to complement the surrogate error model should be selected. We use an exploration generator  $\mathbf{G}_{\mathrm{R}}$  to generate candidates. To encourage diversity so as to facilitate exploration, we assign the generator random weights sampled from a simple distribution, e.g.,
$$
\boldsymbol {\theta} _ {\mathbf {G} _ {\mathrm {R}}} ^ {(t)} \sim N \left(0, \mathcal {I} ^ {\left| \theta_ {\mathbf {G} _ {\mathrm {R}}} \right|}\right). \tag {8}
$$
We do not intend to train the exploration generator  $\mathbf{G}_{\mathrm{R}}$ , because any training loss that encourages exploration and diversity can overly drive the base networks to shift focus in the state space and cause instability in the integrated algorithm. Such an instability phenomenon, caused by training  $\mathbf{G}_{\mathrm{R}}$ , is demonstrated in the ablation study (2) in Table 2.
By adopting the idea of active exploration via disagreement [56, 57], we consider the state, for which the base networks are the least confident about to estimate the implicit errors, as more informative. Since we use an ensemble of base neural networks to estimate the error, the standard deviations of the base network predictions serve as natural confidence measures [56], which are stored in a  $k$ -dimensional vector:
$$
\boldsymbol {\sigma} \left(\hat {\mathbf {x}}, \left\{\mathbf {w} _ {i} ^ {(t - 1)} \right\} _ {i = 1} ^ {L}\right) = \left[ \sigma_ {1} \left(\left\{\phi_ {1} (\hat {\mathbf {x}}, \mathbf {w} _ {i}) \right\} _ {i = 1} ^ {L}\right), \dots , \sigma_ {k} \left(\left\{\phi_ {k} (\hat {\mathbf {x}}, \mathbf {w} _ {i}) \right\} _ {i = 1} ^ {L}\right) \right]. \tag {9}
$$
The state maximizing disagreement, i.e., an accumulated standard deviation, between the base networks, is selected given as
$$
\hat {\mathbf {x}} _ {\mathrm {R}} ^ {(t)} = \arg \max  _ {\hat {\mathbf {x}} \sim p (\hat {\mathbf {x}} | \boldsymbol {\theta} _ {\mathbf {G} _ {\mathrm {R}}} ^ {(t)})} \boldsymbol {\sigma} (\hat {\mathbf {x}}, \left\{\mathbf {w} _ {i} ^ {(t - 1)} \right\} _ {i = 1} ^ {L}) \mathbf {w} _ {k} ^ {T}, \tag {10}
$$
where the row vector  $\mathbf{w}_k = [w_1, w_2, \ldots, w_k]$  stores the implicit error weights. The state-error pair  $\left(\hat{\mathbf{x}}_{\mathrm{R}}^{(t)}, \mathbf{e}_{\mathrm{R}}^{(t)}\right)$  is obtained after error collection.
Surrogate Model Update: To initialize the algorithm, we priorly collect a set of state-error pairs  $D_0 = \{\mathbf{x}_i,\mathbf{e}_i\}_{i = 1}^N$  for randomly selected states. Next, at each iteration  $t$ , two new states are selected and their physical errors are calculated, thus resulting in two new training examples to update the surrogate error model, and an expanded training set  $D_{t} = D_{t - 1}\cup \left(\hat{\mathbf{x}}_{\mathrm{IT}}^{(t)},\mathbf{e}_{\mathrm{IT}}^{(t)}\right)\cup \left(\hat{\mathbf{x}}_{\mathrm{R}}^{(t)},\mathbf{e}_{\mathrm{R}}^{(t)}\right)$ . In our implementation, the base neural network weights  $\mathbf{w}_i^{(t - 1)}$  obtained from the previous iteration are further fine tuned using the two added examples  $\left(\hat{\mathbf{x}}_{\mathrm{IT}}^{(t)},\mathbf{e}_{\mathrm{IT}}^{(t)}\right)$  and  $\left(\hat{\mathbf{x}}_{\mathrm{R}}^{(t)},\mathbf{e}_{\mathrm{R}}^{(t)}\right)$ , as well as the  $N$  examples sampled from the previous training set  $D_{t - 1}$ .
3.3 Remediation System and Implementation
Given an ML estimation  $\hat{\mathbf{x}}$ , the remediation system collects its physical error vector as in Eq. (1), then calculates the accumulated error from the objective function of Eq. (2) and compares it to the feasibility threshold  $\epsilon > 0$ . When the error exceeds the threshold, the GEESE algorithm is activated to search a feasible estimation  $\hat{\mathbf{x}}^*$  such that  $e(\hat{\mathbf{x}}^*) \leq \epsilon$  by querying the physical error as few times as possible. Algorithm 2 outlines the pseudocode of GEESE<sup>2</sup>, while Fig.1 illustrates its system architecture. Our key implementation practice is summarized below.
$^{2}$ project repo: https://github.com/RalphKang/GEESE
Algorithm 2 GEESE
Require: Physical error model (and or metrics)  $\{P_i\}_{i=1}^h$  and error weights  $\{w_i\}_{i=1}^h$ , feasibility threshold  $\epsilon > 0$ , training frequency coefficient  $\delta_G$ , focus coefficient  $c > 0$ , maximum iteration numbers  $T$  and  $T_e$ , early stopping threshold  $\epsilon_e > 0$ , initial training data size  $N$
Ensure: An acceptable state  $\hat{\mathbf{x}}^*$  with  $e(\hat{\mathbf{x}}^{*})\leq \epsilon$
1: Initialize: iteration index  $t = 0$ , initial base neural network weights  $\left\{\mathbf{w}_i^{(0)}\right\}_{i=1}^L$ , number of early stopped base neural networks  $n_e = 0$ , initial exploitation neural network weights  $\theta_{\mathbf{G}_{\mathrm{IT}}}^{(0)}$
2: Sample  $Z_{\mathrm{IT}}$  for exploitation generator  $\mathbf{G}_{\mathrm{IT}}$
3: Randomly select  $N$  states to query their physical errors to obtain  $D_0 = \{\mathbf{x}_i, \mathbf{e}_i\}_{i=1}^N$
4: for  $t < T$  do
5: Set the added training dataset as  $\Delta D_{t} = \emptyset$
6: Update  $\theta_{\mathbf{G}_{\mathrm{IT}}}^{(t)}$  by training with Eq. (6) approximated by  $Z_{\mathrm{IT}}$  for up to  $T_{G} = \delta_{G}\left\lfloor \frac{2n_{e}}{L} +1\right\rfloor$  iterations
7: Select exploitation query state  $\hat{\mathbf{x}}_{\mathrm{IT}}^{(t)}$  by Eq. (7) approximated by  $X_{\mathrm{IT}}^{(t)}$
8: if  $\hat{e}\left(\hat{\mathbf{x}}_{\mathrm{IT}}^{(t)},\left\{\pmb{\theta}_i^{(t)}\right\}_{i = 1}^L\right)\leq c\epsilon$  then
9: Collect the new state-error pair  $\Delta D_{t} = \Delta D_{t}\cup \left(\hat{\mathbf{x}}_{\mathrm{IT}}^{(t)},\mathbf{e}_{\mathrm{IT}}^{(t)}\right)$  , set  $\hat{\mathbf{x}}^* = \hat{\mathbf{x}}_{\mathrm{IT}}^{(t)}$
10: end if
1: if  $e = \sum_{i=1}^{h} w_i E_{P_i}(\hat{\mathbf{x}}^*) \leq \epsilon$  then
2: Stop the algorithm
3: end if
4: Sample  $\pmb{\theta}_{\mathbf{G}_{\mathbf{R}}}^{(t)}$  by Eq. (8)
15: Select exploration query state  $\hat{\mathbf{x}}_{\mathrm{R}}^{(t)}$  by Eq. (10) approximated by  $X_{\mathrm{R}}^{(t)}$
16: Collect the new state-error pair  $\Delta D_{t} = \Delta D_{t}\cup \left(\hat{\mathbf{x}}_{\mathbb{R}}^{(t)},\mathbf{e}_{\mathbb{R}}^{(t)}\right)$  , update the training data  $D_{t} =$ $D_{t - 1}\cup \Delta D_t$
17: Obtain  $\tilde{D}_i$  by sampling randomly  $N$  state-error pairs from  $D_{t-1}$  for each base neural network. Prepare training datasets  $\{D_i\}_{i=1}^L$  where  $D_i = \Delta D_t \cup \tilde{D}_i$
18: Update  $\left\{\mathbf{w}_i^{(t)}\right\}_{i = 1}^L$  by training each base neural network using  $D_{i}$  by Eq. (5) for up to  $T_{e}$  iterations, and count the number of early stopped base neural networks  $n_e$
19: end for
Empirical Estimation: Eqs. (6), (7) and (10) require operations performed over probability distributions. In practice, we approximate these by Monte Carlo sampling. For Eq. (6), we minimize instead the average over the sampled latent variables  $Z_{\mathrm{IT}} = \{\mathbf{z}_i\}_{i=1}^{N_{\mathrm{IT}}}$  with  $\mathbf{z}_i \sim U\left([-a_{\mathrm{IT}}, a_{\mathrm{IT}}]^d\right)$ , and this is fixed in all iterations. The search space of Eq. (7) is approximated by a state set computed from  $Z_{\mathrm{IT}}$  using the trained generator, i.e.,  $X_{\mathrm{IT}}^{(t)} = \left\{\mathbf{G}_{\mathrm{IT}}(\mathbf{z}_i, \boldsymbol{\theta}_{\mathbf{G}_{\mathrm{IT}}}^{(t)})\right\}_{i=1}^{N_{\mathrm{IT}}}$ . Similarly, the search space of Eq. (10) is approximated by a state sample  $X_{\mathrm{R}}^{(t)} = \left\{\mathbf{G}_{\mathrm{R}}(\mathbf{z}_i, \boldsymbol{\theta}_{\mathbf{G}_{\mathrm{R}}}^{(t)})\right\}_{i=1}^{N_{\mathrm{R}}}$  where  $\mathbf{z}_i \sim U\left([-a_{\mathrm{R}}, a_{\mathrm{R}}]^d\right)$ .
Early Stopping: When training the base neural networks for implicit error estimation, in addition to the maximum iteration number  $T_{e}$ , early stopping of the training is enforced when the training loss in Eq. (5) is smaller than a preidentified threshold  $\epsilon_{e}$ . As a result, a higher number  $n_e$  of early stopped base neural networks indicates a potentially more accurate error estimation. This strengthens the confidence in training the generator  $\mathbf{G}_{\mathrm{IT}}$  by Eq. (6) that uses the trained base neural network from the previous iteration. In other words, when the base neural network are not sufficiently well trained, it is not recommended to put much effort in training the generator, which relies on the estimation quality. Therefore, we set the maximum iteration number  $T_{G}$  for training  $\mathbf{G}_{\mathrm{IT}}$  in proportional to  $n_e$ , i.e.,  $T_{G} = \delta_{G}\lfloor \frac{2n_{e}}{L} +1\rfloor$ , where  $\delta_G$  is training frequency coefficient.
Failed Exploitation Exclusion: The state selection motivated by exploitation aims at choosing an  $\hat{\mathbf{x}}_{\Pi}^{(t)}$  with comparatively low physical error. To encourage this, a focus coefficient  $c$  is introduced,
Figure 1: The workflow of GEESE: If the estimation from an ML estimator fails the physical evaluation  $E_P$ , GEESE is activated. The error estimated by hybrid surrogate error model is used to train the exploitation generator  $\mathbf{G}_{\mathrm{IT}}$ . Two candidate state sets are generated by  $\mathbf{G}_{\mathrm{IT}}$  and exploration generator  $\mathbf{G}_{\mathrm{R}}$ , and finally, two states  $\hat{\mathbf{x}}^* = \hat{\mathbf{x}}_{\mathrm{IT}}$  and  $\hat{\mathbf{x}}_{\mathrm{R}}$  are selected by the surrogate error model and fed into  $E_P$  for evaluation and data collection. The process is terminated till  $e(\hat{\mathbf{x}}^*) \leq \epsilon$ .
which, together with the feasibility error threshold  $\epsilon > 0$ , is used to exclude a potentially failed state with a high estimated error, i.e.,  $\hat{e}\left(\hat{\mathbf{x}}, \{\mathbf{w}_i\}_{i=1}^L\right) > c\epsilon$ , to avoid an unnecessary query.
We test the proposed approach GEESE on three real-world engineering inverse problems, including aero engine design [42], electro-mechanical actuator design [58] and pulse-width modulation of 13-level inverters [59]. The first problem is to find eleven design parameters (state) of an aero engine to satisfy the thrust and fuel consumption requirement (observation). The second problem is to find 20 design parameters (state) of an electro-mechanical actuator to satisfy requirements for overall cost and safety factor (observation). And the third problem is to find a group of 30 control parameters (state) of a 13-level inverter to satisfy the requirements for distortion factor and nonlinear factor (observation). Details of these problems along with their physical models and metrics for evaluation are explained in supplementary material (Section A). We compare it with a set of classical and state-of-the-art black-box optimization techniques, including Bayesian Optimization with Gaussian Process (BOGP), GA [21], PSO [22], CMAES [60], ISRES [61], NSGA2 [62], and UNSGA3 [63], as well as the recently proposed work SVPEN [42], which employs RL in solving SAE inverse problems. These techniques are chosen because they are effective at seeking solutions with the assist of actual physical evaluations.
In practice, different types of simulators exist for the same problem. For instance, for problem 1, a simulator mentioned in [64] with high fidelity takes 10.3 seconds  $\times$  309 iterations  $= 53$  minutes to obtain a converged simulation, while another simulator in [65] with a much lower fidelity can realize real-time simulation. Since we aim at a research scenario that attempts to search a feasible state without querying too much data and without setting a high standard on data quality, we choose to use faster simulators with lower but reasonable fidelity in our experiments. For the three studied problems, each simulator takes no more than five seconds to run. Since the computing time varies when changing simulators, we report the performance by query times instead of actual computing time. Accordingly, we adopt two metrics to compare performance. First, we set a maximum budget of  $T = 1,000$  query times for all studied problems and compared methods, and test each method on each problem individually with 100 experimental cases, each case corresponds to a concerned ML state estimation. The setup of the experimental cases is described in Appendix A of supplementary material. We measure the number of experiments out of 100 where a method fails to correct the concerned estimation when reaching the maximum query budget, and refer to it as the failure times  $N_{\mathrm{failure}}$ . Also, the average number of queries that a method requires before finding a feasible state in an experiment, is reported over 100 experiments, and referred to as average query times  $N_{\mathrm{query}}$ . A more competitive algorithm expects smaller  $N_{\mathrm{failure}}$  and  $N_{\mathrm{query}}$ .
We report the adopted hyper-parameter and model setting for GEESE: The common hyperparameter settings shared between all three studied problems include  $T_{e} = 40$ ,  $\epsilon_{e} = 1e^{-4}$  and  $N = 64$ , and the learning rates of  $1e^{-2}$  and  $1e^{-4}$ , for training the exploitation generator and base neural networks, respectively. Different focus coefficients of  $c = 1.5, 2$  and 5 (set in an increasing fashion) are used for problems 1, 2 and 3, respectively, due to an increased problem complexity in relation to their
Table 1: Performance comparison of the compared methods, where the best is shown in bold, while the second best is underlined
Algorithm	Problem 1
State Dimension:11	Problem 2
State Dimension:20	Problem 3
State Dimension:30
Failure times	Query times	Failure times	Query times	Failure times	Query times
BOGP	0	3.29 ±1.51	97	973.76 ±144.28	4	112.66 ±229.98
GA	0	64.00 ±0.00	0	130.56 ±63.31	13	231.76 ±339.71
PSO	0	64.00 ±0.00	0	64.00 ±0.00	12	244.16±343.71
CMAES	0	55.67 ±3.28	0	119.44 ±41.80	12	227.42 ±312.17
ISRES	0	65.00±0.00	0	177.64 ±80.51	16	250.05 ±350.16
NSGA2	0	64.00 ±0.00	0	139.52 ±68.56	13	232.40 335.94
UNSGA3	0	64.00 ±0.00	0	140.80 ±79.94	12	227.52 ±330.07
SVPEN	100	1000.00±0.00	100	1000.00±0.00	100	1000.00±0.00
GEESE (Ours)	0	3.18 ±1.98	0	51.65 ±33.01	0	43.56 ±65.28
increasing dimensions of the state space. Similarly, an increasing training frequency coefficient  $\delta_G = 1, 1$  and 7 is used for problems 1, 2 and 3, respectively, because the problem requires more training iterations as it involves more complex patterns from higher-dimensional state space. The ensemble surrogate model for estimating the implicit errors is constructed as an average of 4 multi-layer perceptrons (MLPs) each with three hidden layers consisting of 1024, 2028 and 1024 hidden neurons. The exploration generator  $\mathbf{G}_{\mathrm{R}}$  is constructed as a single layer perceptron (SLP) and its one-dimensional input is sampled from  $U([-5,5])$ . For problems 1 and 2 that are relatively less complex from an engineering point of view, we directly set the latent space  $\mathcal{Z}$  as the state space  $\mathcal{X}$  without using any neural network to transform in between. Then, we directly sample initial state set  $X_{\mathrm{IT}}^{(0)}$ . The exploitation state is directly optimized iteratively, e.g., by a gradient descent approach based on Eq.(7) to obtain state set  $X_{\mathrm{IT}}^{(t)}$ , as shown in Eq. (11). The one with the smallest objective function value is selected as the exploitation state, i.e.,
$$
\hat {\mathbf {x}} _ {\mathrm {I T}} ^ {(t)} = \underset {\hat {\mathbf {x}} \in \mathcal {X}} {\arg \min } \hat {e} \left(\hat {\mathbf {x}}, \left\{\mathbf {w} _ {i} ^ {(t - 1)} \right\} _ {i = 1} ^ {L}\right). \tag {11}
$$
Problem 3 involves a special state pattern, requiring an increasing state value over the dimension, i.e.,  $\mathbf{x}_i - \mathbf{x}_{i + 1} < 0$ . To enable the latent variables to capture this, we construct the exploitation generator  $\mathbf{G}_{\mathrm{IT}}$  as an MLP with three hidden layers consisting of 256, 512 and 256 hidden neurons. Also, to avoid generation collapse [66] in problem 3, a regularization term has been added to the training loss in Eq. (6), resulting in the following revised training to encourage state diversity, as
$$
\boldsymbol {\theta} _ {\mathbf {G} _ {\mathrm {I I}}} ^ {(t)} = \arg \min  _ {\boldsymbol {\theta} \in \mathcal {R} ^ {3 0}} \mathbb {E} _ {\mathbf {z} \sim U \left[ - 5, 5 \right] ^ {3 0}} \left[ \hat {e} \left(\mathbf {G} _ {\mathrm {I I}} (\mathbf {z}, \boldsymbol {\theta}), \left\{\mathbf {w} _ {i} ^ {(t - 1)} \right\} _ {i = 1} ^ {L}\right) + \max  (0. 0 2 8 8 - \sigma_ {1} (\mathbf {z}, \boldsymbol {\theta}), 0) \right], \tag {12}
$$
where  $\sigma_{1}(\mathbf{z},\boldsymbol {\theta})$  denotes the standard deviation of the first state element generated by  $\mathbf{G}_{\mathrm{IT}}$ . We encourage it to shift away from the collapsed point but not overly spread, by bounding  $\sigma_{1}$  with a portion of the standard deviation of a uniform distribution, e.g., 0.288, and the portion  $\frac{0.288}{10} = 0.0288$  is observed empirically effective. The spread control is only needed for the first state as the remaining states follow by  $\mathbf{x}_i - \mathbf{x}_{i + 1} < 0$ . Configurations of the competing methods, together with extra information on GEESE, are provided in Appendix B of supplementary material.
4.1 Results and Comparative Analysis
Table 1 summarizes the results of the compared methods for the three problems, obtained with a feasibility threshold of  $\epsilon = 0.075$ , which reflects high challenge with low error tolerance. It can be observed that GEESE has the least failure times  $N_{\mathrm{failure}}$  on all three problems. In problem 3, especially, GEESE succeeds with no failure while most other methods have more than 10 failures. This is a highly desired characteristic for a remediation system with low error tolerance. In addition, GEESE also has the least query times  $N_{\mathrm{query}}$  in all three problems, indicating the best efficiency. We report additional results in Appendix C of supplementary material by varying the feasibility threshold  $\epsilon$  and the initial sample size  $N$ , where GEESE also achieves satisfactory performance in general, while outperforming other methods in handling higher-dimensional problems with lower error tolerance. SVPEN [42] cannot return a feasible correction in 1000 queries in all experiments, as its core supporting RL requires a lot more queries than other optimization based techniques.
4.2 Ablation Studies and Sensitivity Analysis
To examine the effectiveness of the key design elements of GEESEE, we perform a set of ablation studies and report the results in Table 2 using problem 1 with a small feasibility threshold  $\epsilon = 0.05$  indicating low error tolerance. The studies include the following altered designs: (1) Estimate directly the implicit error sum using an MLP with the same hidden layers but one single output neuron. (2) Train the exploration generator  $\mathbf{G}_{\mathrm{R}}$  by using an approach suggested by [57]. (3) Remove the early stopping design. (4) Remove the focus coefficient.
Results show that estimating the implicit error sum worsens the performance. As explained earlier in Section 3.1, this is because the structural information in gradient is lost in error sum estimation, causing ambiguous update when training  $\mathbf{G}_{\mathrm{IT}}$ , and consequently requires GEESE to make more error queries. Also training  $\mathbf{G}_{\mathrm{R}}$  worsens the performance as compared to just assigning random network weights to  $\mathbf{G}_{\mathrm{R}}$  without training. As previously explained in Section 3.2, this is because training  $\mathbf{G}_{\mathrm{R}}$  can frequently shift the focus of the surrogate error model and, thus, impact on the stability of the optimization process. Both early stopping and focus coefficient play an important role in GEESE, where the former prevents GEESE from overfitting and the latter helps avoid unnecessary queries. Additional results on hyperparameter sensitivity analysis for GEESE are provided in Appendix D of supplementary material. The results show that GEESE is not very sensitive to hyperparameter changes and allows a wide range of values with satisfactory performance, which makes GEESE easy to be tuned and used in practice.
(1): Individual vs Sum Error Estimation
Surrogate Error Model	Query times	Standard deviation
Estimate error elements	20.20	16.37
Estimate error sum	23.26	21.18
(3): Effect of Early stopping
Schedule	Query times	Standard deviation
with earlystop	20.20	16.37
w/o earlystop	32.80	17.84
Table 2: Results of ablation studies reported on problem 1, where a better performance is highlighted in bold.
(2): Effect of Exploration Training
Exploration style	Query times	Standard deviation
w/o training	32.64	22.82
with training	41.32	97.15
(4): Effect of Focus Coefficient
Schedule	Query times	Standard deviation
with focus coefficient	20.20	16.37
w/o focus coefficient	27.19	19.36
We have proposed a novel physics-driven optimization algorithm GEESE to correct ML estimation failures in SAE inverse problems. To query less frequently expensive physical evaluations, GEESE uses a cheaper hybrid surrogate error model, mixing an ensemble of base neural networks for implicit error approximation and analytical expressions of exact explicit errors. To effectively model the probability distribution of candidate states, two generative neural networks are constructed to simulate the exploration and exploitation behaviours. In each iteration, the exploitation generator is trained to find the most promising state with the smallest error, while the exploration generator is randomly sampled to find the most informative state to improve the surrogate error model. These two types of selection are separately guided by the approximated error by the ensemble and the disagreement between its base neural networks. The element-wise error approximation promotes a more effective interaction between the surrogate error model and the two generators. Being tested on three real-world engineering inverse problems, GEESE outperforms all the compared methods, finding a feasible state with the least query number with no failure under the low error tolerance setup.
However, there are still challenges to address in the future, particularly for very high-dimensional inverse problems. Such problems are in need of larger and more complex model architecture to accommodate their more complex underlying patterns, and thus impose challenge on training time and data requirement. Computation expense should not only consider the query cost of physical evaluations but also the learning cost of such models. Flexible neural network architectures that allow for embedding domain specific or induced knowledge in addition to simulation data and their training, as well as their interaction with the main solution model, e.g., an ML estimator for inverse problems, are interesting directions to pursue.