Bayesian Modeling and Uncertainty Quantification for Learning to Optimize: What, Why, and How
Authors: Yuning You
Yue Cao
Tianlong Chen
Zhangyang Wang
Yang Shen
Venue: ICLR 2022 Poster (2022)

Abstract: Optimizing an objective function with uncertainty awareness is well-known to improve the accuracy and confidence of optimization solutions. Meanwhile, another relevant but very different question remains yet open: how to model and quantify the uncertainty of an optimization algorithm (a.k.a., optimizer) itself? To close such a gap, the prerequisite is to consider the optimizers as sampled from a distribution, rather than a few prefabricated and fixed update rules. We first take the novel angle to consider the algorithmic space of optimizers, and provide definitions for the optimizer prior and likelihood, that intrinsically determine the posterior and therefore uncertainty. We then leverage the recent advance of learning to optimize (L2O) for the space parameterization, with the end-to-end training pipeline built via variational inference, referred to as uncertainty-aware L2O (UA-L2O). Our study represents the first effort to recognize and quantify the uncertainty of the optimization algorithm. The extensive numerical results show that, UA-L2O achieves superior uncertainty calibration with accurate confidence estimation and tight confidence intervals, suggesting the improved posterior estimation thanks to considering optimizer uncertainty. Intriguingly, UA-L2O even improves optimization performances for two out of three test functions, the loss function in data privacy attack, and four of five cases of the energy function in protein docking. Our codes are released at https://github.com/Shen-Lab/Bayesian-L2O.

Published as a conference paper at ICLR 2022
BAYESIAN MODELING AND UNCERTAINTY QUANTIFI- CATION FOR LEARNING TO OPTIMIZE: WHAT, WHY AND HOW
Yuning You', Yue Cao', Tianlong Chen², Zhangyang Wang', Yang 1.2
l Department of Electrical and Computer Engineering, Texas A&M University
Department of Computer Science and Engineering. Texas A&M University "Department of Electrical and Computer Engineering, University Texas at Austin
{yuning. you, you, cyppsp, yshen) @tatun edu, (tianlong chen, atlaswang] Sutexas. edu
ABSTRACT
Optimizing an objective function with uncertainty awareness well-known improve the accuracy and confidence of optimization solutions. Meanwhile, another relevant but very different question remains yet open: how to model and quantify the uncertainty of optimization algorithm (a.k.a,, optimizer) it- consider the optimizers as
self? To close such a gap, the prerequisite sampled from distribution, rather than few prefabricated and fixed update rules. We first take the novel angle to consider the algorithmic space of opti- mizers, and provide definitions for the optimizer prior and likelihood, that in- trinsically determine the posterior and therefore uncertainty. We then leverage the recent advance of learning optimize (L20) for the space parameteriza- tion, with the end-to-end training pipeline built via variational inference, referred as uncertainty-awwe
L20 (TA-L-20). Our study represents the first effort recognize and quantify the uncertainty of the optimization algorithm. The ex- tensive numerical results show that, UA-L2O achieves superior uncertainty cal- ibration with accurate confidence estimation and tight confidence intervals, sug- gesting the improved posterior estimation thanks to considering optimizer uncer- tainty. Intriguingly, UA-L20 out of three test functions, the loss function in data privacy attack, and four five cases of the energy function in protein docking. Our codes released htttps://qithii com Shen Sttp:/heeennaaaaaaaa Laaeee-L-a/aaaain--22222
improves optimization performances for two
INTRODUCTION
Computational models of many real-world applications involve optimizing non-convex objective functions. As the non-convex optimization problem is NP-hard, no optimization algorithm (or opti- mizer) could guarantee the global optima in general. Instead their solutions' usefulness (sometimes based on their proximity to the optima), when the optima are unknown, can be very uncertain. Be- ing able quantify solution uncertainty directly provides calibration with ensured awareness of the solution quality and usefulness (and another potential benefit is in optimization performance by enhancing the search efficiency). For instance, reliable and trustworthy machine learning models demand uncertainty awareness and quantification (UQ) during training (optimizing) such models, whereas in reality deep neural networks without proper modelling of uncertainty suffer from over- confidence and miscalibration (Guo et al. 2017). In another example of 3D prediction of protein- protein interactions, even though there exists the model uncertainty of the objective function and the data uncertainty of the protein structure data (Cao & Shen, 2020), Mtate-of-thh-ar methods only predict several ranked solutions (Porter et al. 2019) without any associated uncertainty, which is hard for biologists to interpret.
Despite progress in optimization with envertaintt--waareeee fundamental open questions remain: existing methods consider uncertainty either within the data the model (including objective func- tions) (Kendall Gal, 2017; Ortega et al.) 2012) Lyu et al.) [021]] Cao Shen 2020). whereas inconspicuous attention paid to the uncertainty arising from the optimizer that is directly

---

Published as a conference paper at ICLR 2022
sponsible for deriving the end solutions. The optimizer is usually prefabricated and fixed. For instance, there are several popular update rules in Bayesian optimization, such as expected improve- ment (Vazquez & Bect 2010) and upper confidence bound (Srinivas et al.) 2009), that chosen and unaltered for the entire process. For Bayesian neural networks training, the update rule of the iterative optimizer is usually chosen off-the-shelf, such as Adam, SGD, and RMSDrop. In principle, it is important to recognize the existence of an optimization algorithm space where a specific opti- mizer lies as well the importance of the optimizer uncertainty, intrinsically defined in the space to the optimization and UQ solutions. In practice, such uncertainty is unwittingly omitted in the current context where an optimizer is treated as a de facto specified sample in the space.
A naïve characterization for a hand-built optimizer is to rely on a few of its hyper-paun parameters Wen- zel et al. 2020; Lorraine & Duvenaud, 2018), whereas such a parameterization space could be too biased and restricted to span a broader and more complicated optimizer manifold, potentially result- ing in inaccurate UQ results (see Section 4 confidence estimation for test functions). Fortunately,
the recent surge of learning to optimize (L20) (Andrychowicz al. 2016aj Li & Malik 2016 Cao
ct al.
2019a You et al. 2020 Li et al. 2020 Chen et al.] 2020b Shen et al.) [021]] Chen et al.
2021 which another prominent optimization paradigm, parameterized by neural network using high-dimensional weights and following the data-driven ethos to learn the update rule adaptively, therefore introducing the less inductive bias and accompanied with the universal approximation ability, indicates a plausible solution to depict the sophisticated and more complete optimizer space.
Contributions. The aforementioned introduction reveals and discusses the fundamental question of why modelling the optimizer uncertainty, and our contingent questions would be, what defines the optimizer uncertainty and how enable UQ during optimization. We give the following answers. (i) We define the prior and likelihood of the optimizer (Neal (012), which routinely determine the optimizer posterior with the general product rule and Bayes' theorem. The optimizer uncertainty is therefore well-defined. (ii) To unseal UQ within optimization, we treat an optimizer as a random sample from an algorithmic space of iterative optimizers. We leverage the surging 1.20 technique (Andrychowicz et al. 2016a Li & Malik] 2016) with high-dimensional weights for the optimizer space parameterization. (iii) We further construct the end-to-end training pipeline via variational
inference (Kingma & Welling 2013 Higgins et al. 2016) to avoid the expensive computational cost of Markov chain Monte Carlo (MCMC) and degenerated posterior. The newly trained L20 is referred to uncertainty-awwrt work is, for the first time, we recognize and quantify a novel form of uncertainty that lies in the optimization algorithm space parameterized by L20, apart from the classical data- model-based uncertainties (also known as epistemic and aleatoric uncertainties (Fox & Ülkümen 2011
L20 (UA-L20). In summary, the core intellectual value of this
Extensive experiments show that UA-L.20 had superior capability in uncertainty calibration thanks to the accurate estimation of solution posteriors. It confidence levels well matched the chance for the global optimum falling in the corresponding intervals. Intriguingly, although not directly targeted, UA-L20 also outperformed competing methods in optimization performance, as seen in optimizing two out of three test functions in high dimensions, the loss function in data privacy attack, energy functions in four out of five protein-docking cases.
2 PRELIMINARIES
2.1 LEARNING TO OPTIMIZE
Let us consider an optimization problem: ming fxx where x Rd. A classic optimizer often iteratively updates x based on handcrafted rule. For example, the first-order gradient descent algorithm takes an update at iteration based on the local gradient at the instantaneous point x,: X++1 = X, - (xx), where is the step size.
Learning to Optimize (I.2O) has lots of freedom to use the available information. We define z, as optimization trajectories' historical information up to time t, e.g., the existing iterates Xp.... xp...., Xx, and/or their gradients V fx(x)).... fxx).. L20 models an update rule by a predictor function of Z₁: X X, g(zz; Φ). where the mapping of is parameterized by Φ. Finding an optimal update rule can be formulated mathematically as searching for a good & over the parameter space of g. Practically, g is often neural network. Since neural networks are universal approximators, L20 has the potential to discover completely new update rules without relying on existing rules. In order to find a desired o associated with a fast optimizer, (Andrychowicz et al. 2016b) proposed to
2

---

Published as a conference paper at ICLR 2022
minimize the weighted sum of the objective function fxx) over time span T:
min wo (xx); with X1+1 Xt g(2,5 Φ), L O, ..... -
where w...... WT are the weights whose choices vary from case to case and depend on empirical settings. Note that & determines the objective value through determining the iterates xx. L20 solves the problem I for desirable & and correspondingly the update rule g xb.
typical 1.20 workflow is divided into two stages (Chen et al. 202T): a meta-training stage that learns the optimizer with set of similar optimizees from the task distribution; and a meta-testing stage that applies the learned optimizer to new unseen optimizees. The meta-training process often occurs offline and is time consuming. However, the online application of the method, meta testing, is (aimed to be) time saving.
2.2 THE UNCERTAINTY OF OPTIMIZATION
The choice of optimizers is recognized to remarkably impact the solution quality, especially for non-convex and rugged objectives such various loss functions for training deep networks. While non-convex optimization is NP-hard and the global optimum is never guaranteed, domain users usually can only blindly take the solutions given by particular optimizer, or choose from several well-known algorithms based on human experiences or on an ad hoc basis. Such trial-and-error se- lections are biased and restricted span a more complete optimizer manifold, and computationally expensive, even intractable when new algorithms are to be discovered by L20, whose "fitness" or confidence on certain problem instances (especially instances with shifts from the target task distri- butions) is never known. Given a specific problem instance, if we can provide a confidence score or quantify the uncertainty of each candidate optimizer, it will certainly improve the selection and calibration of optimization algorithms, in particular the learned optimizers since they are essentially all "new", and hence enhancing the reliability L20.
As for non-convex objectives, different choices of optimizers, even each algorithm's hyper- parameter or initialization variations, can lead to vastly different solutions, implying new type of uncertainty arising from choosing optimizers. Stochastic optimization methods like random search (Zigljavsky 2012). simulated annealing (Kirkpatrick et al.) 1983) and genetic algorithms (Gold-) enberg 1989) inject the randomness into algorithms reduce solution uncertainty. However, they cannot reliably quantify its effect on the final solutions. Several works explored uncertainty quan- tification during optimization, by Bayesian optimization (Hennig & Schuler 2012] Wang & Jegelka 2017) or Monte Carlo sampling (Ortega et al. (0012). Several works in Bayesian optimization ex- plored uncertainty quantification during optimization, by modeling the distributions over the func- tional space (Hennig& & Schuler 2012] Wang & Jegelka (017) or directly over the optimum (Ortega al. 20122 Cao & Shen 20200). However, those methods are often computationally too expensive, not to mention applied within the already costly 1.20 process. More importantly, they neither ex- plicitly consider the uncertainty within the optimizer itself, nor concern the automatic learning and discovery of the most suitable optimizer.
TECHNICAL APPROACH
To quantify and address such "fitness" especially for the L20 parameterization in a principled way, we will introduce this new optimizer uncertainty arising from the choice of optimization algo- rithms, which is different from the classical data- or masel-based uncertainties (also known as epis- temic and aleatoric uncertainties (Fox & Ülkümen 2).1]). We will explore: how to quantify opti- mizer uncertainty? how to design a learned optimizer so that its uncertainty can be more accurately quantified? what is the best calibrated optimizer
can choose or learn for given problem?
The core innovation is to treat an optimizer as a random sample from a continuous algorithmic space, rather than one of few hand-crafted update rules, as to model the intrinsic uncertainty within the optimizer. This novel view of algorithmic space is particularly enabled by L20. Note that an L20 update rule is typically parameterized by neural network g. with its inputs and learnable parameters Φ. Its learning capacity allows us to "sample" optimizers by taking g(: Φ) with different weights Φb That is sufficiently versatile and flexible in practice.
3

---

Published as a conference paper at ICLR 2022
3.1 CONCEPTS FOR OPTIMIZER UNCERTAINTY
Without loss of generality, we define iterative algorithmic space 9, where each point g E G maps to an update g(Z,) dependent on the current/past zero-th order and/or first-order information 2,. We treat ! as random vector from 9, that leads to defining the optimizer uncertainty follows:
Definition (Optimizer Uncertainty) Let G be the algorithmic space, where each point g E G is an optimizer (omitting & parameterization). We assume that
1. 9 has a prior distribution p(g);
2. Its likelihood can be interpreted p(z, Zto 9) - p(x;/ Vto t.
The likelihood factorization is based on the causality of iterative optimizers, i.e., (x,, 2;) depends solely on {(x.,> Z B,)))isss given g. Thereby, we define the optimizer uncertainty at step t as the posterior of g, that is conditional distribution on the steps of the trajectory, using Bayes" theorem:
p(glzz) x p(g) II p(x
Intuitively, the prior p(g) represents the belief about well-performing optimizers, the likelihood p(z, ( g) represents the probability of observing an optimization trajectory (data) under given optimizer g. and the posterior p(g (z) represents the probability for being the optimizer generating the observed data.
Prior work on hyper- aaper-parameter optimization (HPO) for classical algorithms (Feurer & Hutter 2019) special case of L20 (e.g.. only few hyper-parameters
could be viewed results in a much biased and restricted prior p(g)). The seminal work Bergstra et al. (2011) optimized
"learnable" which
hyper-parameters using costerior-baset fitness modeling. Recently, /enzel et al. (020) proposed non-Bag Non-Bayesian approach of prper-parameter ensembling that can estimate the predictive uncertainty of an optimization algorithm by varying syper-parameters Our work generalizes from tuning a few hyper-paraa syper-parameters to learning the entire optimizer space, and extends a Bayesian treatment. Since L20 involves much higher-dimensional parameters compared HPO, its uncertainty quantification calls for computationally tractable and scalable approaches, which is detailed next.
3.2 L20 WITH BAYESIAN UNCERTAINTY QUANTIFICATION
Now that we have the optimizer space & parameterized in Φ an optimizer g is modeled as a random vector in the space. That allows for reducing the modeling of optimizer uncertainty p(g (z)) to mod- eling the posterior p( Φ 2.). For simplicity, we use Gaussian prior of &: p(Φ) exp
Φ
where À is constant. We also follow the hidely-adopter idea in (Gal & Ghahramani 2016 Bishop
Tipping, 2003: Ortega et al. 2012 Cao & Shen 2020), to view the loss function value as propor- tional to the negative logarithmic likelihood, i.e., (x = p(x,
Φ) x exp(-f(x?))
We can then calculate the posterior of the optimizer based upon equation
T
T
p(Φ)ZT) x exp(
1 2X
(
II t=1
5(()) 21 = exp (- - 1
(3)
Training. Maximum a posteriori (MAP) estimation of the optimizer parameters leads to a de- generate distribution and point estimation without uncertainty. Moreover, directly maximizing the optimizer posterior in Eq. 3 via Markov chain Monte Carlo (MCMC) would encounter computa- tional intractability considering the high-dimensional space of the optimizer space parameterized in L20 by neural network weights. Instead, we introduce a variational distribution q(Φ:; 0) (Kingmal & Welling (003)) with the variational parameter 0 and accordingly we introduce the following ob- jective function whose minimization is equivalent to maximizing the evidence lower bound (ELBO) (see Appendix for the derivation):
T
KL KLig d: 0)llp((lZ))) = =- E (b:0)
Σ f f(x)) (L(g(b:0)llp(0))
(4)
1=1
We refer to the L20 under the new meta- meta-training loss L200), which can be trained end-to-end with the Gaussian parametrization and the reparametrization
4 w.r.t 0 as uncertainty L20 (UA-
4

---

Published as a conference paper at ICLR 2022
X
X
X
tong
sampling
p(xe(2 1,
Function Instance
p(X Re,
Instance
2(610)
Φ g(d)0)
[x(-11) 2
(x(e))
Figure 1: Feed-forward computational chart for training UA-L20 Notations are as in the main text.
trick (Kingma & Welling 2013 Higgins et al. 2016). The feed-forward computational chart for training UA-L20 is depicted in Figure
Deployment. With the well-trained UA-1.20 and the found 0" we ideally assume the function values of the trajectory are submartingales that [(x] S f(x)). [0=g(o::=)) <
00. Then based upon martingale convergence theorem (Doob) 1953) lim;- f(x;) fxxx) exists almost surely, i.e. it converges with infinite optimization iterations. Within the finite steps, UA-L.20 has well-defined approximated posterior g( 0²), and hence the baked-in ability to quantify optimizer uncertainty at the solution. Therefore, the posterior of solutions is calculated by the integral with the variational distribution g( 0-) as:
p(x'IzT)= / p(x lr, (){(0; 0")d
(5)
which can be estimated by Monte Carlo sampling. Specifically, for estimating the solution posterior, the learned model is run 10,000 times with random initialization and different trajectories z, accord- ingly. The optimizer parameter is sampled from q( (0) once per iteration, following variational
inference in Kingma & Welling (203)) and Higgins al. (2016).
EXPERIMENTS
We examine our proposed UA-L20 on optimizing: (i) non-convex test functions, (ii) loss functions data privacy attack, and (ii) energy functions in predicting 3D protein-protein interactions (pro- tein docking). We compare UA-L20 with three non-Bayesian methods: manually-designed Adam (Kingma & Ba 2014) and particle swarm optimization (PSO) (Kennedy & Eberhart, 1995) as well as LSTM (Andrychowicz et al. 6a), an L20 method. We also compare to Adam and PSO with hyper-parameter Bayesian optimization (Adam-BO and PSO-BO), Adam with learning rate en- sembles (Adam_Ir (anaelble) and Adam with stochastic gradient MCMC (Adam_Noisy Gradient) Lastly we compare to Bayesian active learning (BAL) (Cao & Shen 2020).
For Adam and PSO, make their haper-yarameee parameters probabilistic to caleulate the posterior as shown in Table Appendix[[] All algorithms are run for 10,000 times with random initialization obtain the empirical posterior distributions p(x" (zy), following Eq. 5 UA-L20 and its counterparts for others. We access the optimization uncertainty with the following expression (Ortega et al.) 2012
Cao & Shen 2020):
P [b. < 1xx &112 S ubb)) = 1
(6)
Solution sampling and estimation
Posterior estimation
Calculating bounds
Assnying posterior
p(x*)
x ~ p(x"), argmin f(x")
ub, lb,
count count confidence interval
count
Running for 10,000 times
6. count/100, 000
Ground truth Xenin
Estimation
Figure 2: Computational procedure for uncertainty quantification: given designated confidence o, calculating the length of confidence interval ro and its corresponding estimated confidence 6..
5

---

Although UA-L20 is designed for the uncertainty awareness rather than the quality of optimization solutions, we still assess potential benefit in optimization performance, using the expected distance The lower Published as a conference paper at ICLR 2022
where lb, and ub, are the lower and upper bounds at the confidence level of σ, and choose the one with the lowest function value (we also study the mean value in Appendix E to be the estimated solution &. Ablation different runs and comparison with competitors
wall-clock time are presented in Appendix H For the choice of assessing the proximity of solutions rather than function values [f(x)) f(x). the reason is that in many optimization applications (e.g. privacy attack and protein docking), the objective function actually plays the role of the "tool" instead of the "goal", thus the function value not the key metric assess the quality of solutions in particular when the objective function is non-convex and noisy.
Evaluation metrics. UA-L20 is designed for uncertainty awareness in the optimizer. To assess the calibration of uncertainty quantification (UQ), we use the accuracy of confidence, following (Ortega et al. 2012; Cao Shen 2020). In other words, we compare expected confidence levels as in Eq. 6 (0.8 and 0.9 in this paper) and the empirical probability Eo of "success" when the corresponding confidence intervals indeed contain the global optima. When there tie in the major assessment metric Eg. the tightness of the confidence interval for the solution's 's proximity to the global optimum (Eq. o To ub, lb serves as a tie-breaker. A schematic illustration of the uncertainty assessment shown in Figure 2
between the optimized solutions and the global optimum <xxx"(xx 2T) ||x Xmmell2
distances indicate the better solution quality (optimization performance).
Two-stage training. Due to the extremely nigh-dimensional optimizer space and the rugged poste- rior landscape, over-challenging directly train the model through the equivalency of ELBO optimization in Equation (4). We therefore perform two-stage training where we first train our model in non--aane non-Bayesian way and then use the resulting f as the warm start for fine-tuning the mean vari-
ational parameters in the second stage. We examine the performance of the warm-start model in Appendix to show the benefit of the second-stage training.
Implementations. The model is implemented in Tensorflow 1.13 (Abadi et al. 2016) and optimized by Adam (Kingma & Ba, 2014). For the L20 architecture, we use the coordinate- wise LSTM from Andrychowicz et |l.] 20] containing 10,282 free parameters. The meta- meta-training and test data are described in Appendix [ For all experiments, the length of LSTM set to be 20, with 5,000 training epochs for both training stages. We tune the À values in equations (3) and (4) from {0.1, 1. 10), validated by validation performance in test functions and by the tightness of confidence intervals in privacy attack and protein docking. The ablation study on À is presented in Appendix C showing that UA-L20 is not sensitive to the value of X.
4.1 NON-CONVEX TEST FUNCTIONS
Setup. We first evaluate the performance on the test functions in the global optimization bench- mark (Jamil & Yang 2013). We choose three extremely rugged, non-convex functions, Rastrigin, Ackley and Griewank with analytical forms shown in Appendix B, paired with dimensionalities in {3.6, 6,.6,... 303.
Results. We compare UA-L20 with competitors in optimization and UQ performances shown in Figure 3 In the most relevant assessment metric, the accuracy of confidence levels (bottom two rows), UA-L20 performs the best in all but few cases for all the three functions, outperforming hand-engineered optimizers, non-Bayesian or Bayesian, and L20 without uncertainty awareness (DM_LTTM). The few exceptions, such as the Griewank function in 27 dimensions, could not be attributed to the higher dimensionality (as that was not the case for UA-L20 for Rastrigin and Ack- ley functions). DM_LSTM also showed overconfidence in such exceptions but the few exceptions cannot be attributed to L20 either.
Given that UA-L20 consistently outperforms competitors in confidence accuracy, comparing the tightness of confidence intervals alone (Figure 9 is not meaningful. Note that other methods, when having tighter confidence intervals, were often too overconfident to realize that the true global optima were inside the intervals much less than their confidence levels suggested.
We also examined potential but not intended benefits of UA-L20 in optimization performance (top row of Figure 3. Interestingly, UA-L20 even improved optimization performances for Ackley and Griewank functions when the dimensionality is above 12, while being slightly worse in Rastrigin.

---

Published as a conference paper at ICLR 2022
Rastrigin
Ackley
Griewank
Adam
Adam_Bo
PS0_80 80
BAL DILISTM
2
UA-120
12
15
18
21
27
30
12
18
21
24
30
12
15
21
0.02
0.0
0.00
0.00
-0.1
-0.02
-0.2
0.04
-0.04
0.3
12
15
18
21
24
27
30
9
12
18
21
24
30
12
15
18
21
24
27
0.05
0.05
0.0
0.8
0.00
-0.05
-0.05
0.3
6
12
15 Dimension
18
21
27
30
12
18 21 Dimension
24
30
12
15
21
24
27
Dimension
Figure 3: Optimization and uncertainty performance of different methods in three non-convex test functions. Different column represents different functions, each row stands for: (i) row for non-intended optimization performance, the lower the better; (ii) 2nd & 3rd rows, the most important metric for the intended uncertainty calibration, are for the precision of the estimated confidence, lower values indicating more accurate posterior estimation. The corresponding confidence intervals shown in Appendix C
These results echo our conjecture that, although central goal is not for better optimization lutions but for their better uncertainty awareness, better uncertainty calibration could sometimes improve optimization performances.
We further plot the confidence for solution regions segmented in ||x x versus the corresponding optimization performance ||x* Xtmeel ll2 (see Appendix [ for detailed procedure) in Figure 10 We find that when UA-L20 achieves better optimization performance (Ackley and Griewank as opposed Rastrigin), it generates solutions with higher confidence in the regions closer to the global optima and with narrower distributions (more certain). In contrast the competitors do not show such trends.
4.2 DATA PRIVACY ATTACK
Setup. We next apply model to application that critically needs UQ. As many machine learn- ing models are deployed publicly, it is important to avoid leaking private and sensitive information, such financial data and health data. Data privacy attack (Nasr al. 2018) studies this problem by playing the role of hackers and attacking machine learning models to quantify the risk of privacy leakage. Better attacks would help models to be better prepared for privacy defense.
We use the model and dataset in (Cao et al. 2019b), where each input has 9 features involving patient genetic information shown Appendix B and the output is the probability of the clinical significance (pathogenicity) for genetic variants in patients. We study the following model inversion attack (Fredrikson et al. 2015): given features x' E [o, 1 out of and the label y of each patient, the privacy attack aims to recover the rest 4 features Xime E | 1) (potentially sensitive patient information). Therefore, the optimization minxe[o.1] [g((''
y)² for all patients where
trained predictor. The closeness between the optimized and real input features quantify the risk information leakage and the quality of the attack. We evaluate our method for all test cases in (Cao et al. 2019b).
Results. We report the UQ and optimization performances of UA-L20 in Table UQ, UA-L20's
performance (the accuracy of estimated confidence) stands out against competitors, although not as dominant as it does to competitors test functions. Even in optimization, UA-L20 outperforms all optimization.
competitors, which again demonstrate the potential benefit of uncertainty-

---

Published as a conference paper at ICLR 2022
Table 1: Optimization and uncertainty performance of different methods in genetic data privacy attack.
Method
x Xyse |2
F0.9
0.9
<a.s
0.8)
To.9
ra.s
Adam
0.39
0.90
0.80
0.08
0.06
Adam_BO
0.38
0.90
0.80
0.08
0.05
Aram_Ir Adam_Ir.Ensemble EEnemble
0.35
0.90
0.80
0.01
0.009
Adam_Noiss Gradient
0.57
0.90
0.80
0.01
0.01
PSO
0.54
0.08
0.10
0.54
0.34
PSO
0.53
0.15
0.48
0.42
BAL
0.52
0.90
0.80
0.01
0.01
DM.LSTM UA-L20
0.34
0.09
0.77
0.09
0.02
0.30
0.25
0.27
0.06
0.04
Note that PSO had slightly more accurate confidence but much looser intervals, suggesting flat posteriors and leading to worse optimization.
We argue that, unlike pure optimization problems in test functions, the optimization for privacy attack involves the generalization issue in the machine learning paradigm: less generalizable samples pose challenges to not only optimization but also UQ. To verify this argument, we plot optimization performance for individual samples versus their estimated posterior precision in Figure 4 showing a positive correlation: samples easier to generalize are closer to the optima (more leftward in the r-axii), and more precise in estimated confidence (more downward in the y-axis). The tightness of confidence intervals is also well correlated with solution quality.
-Roueb
EDx
indiiduals
Figure 4: Optimization performance versus UQ results (Co.9, 6v.s, Pb.9 and ro.s) of UA-L20 for 318 test sam- ples in data privacy attack.
Similar to that in Section we also plot confidence versus optimization performance in Figure II to further demonstrate calibration capability of UA-L20. To demonstrate the practical use of developed UQ, conduct an experiment "failure" case detection, that we use assessed un- certainties to probe the optimization performance of individual runs. Results in Appendix show UA-L20 provides the best detection.
4.3 ENERGY FUNCTIONS FOR PROTEIN DOCKING
Setup. We lastly examine UA-L20 using a bioinformat- ics application: predicting the 3D structures of protein- complexes (Smith Sternberg 2002) or protein dock- ing. Ab initio protein docking be recast optimiz- ing a noisy and expensive energy function in a high- dimensional conformational space (Cao & Shen (020). While solving such optimization problems still remains difficult, quantifying the uncertainty of resulting solutions is even more challenging. The compared state-of-the-art method is BAL (Cao & Shen 2020).
200
Figure 5: Scatter plot of optimization per- formance versus the length of confidence in- terval for five cases in protein docking. The Pearson correlation coefficient achieved by BAL is -0.80, and that by UA-L20 is 0.59.
We calculate the energy function (objective function fxx)) in a CHARMM19 force field as in (Moal & Bates 2010). 25 protein-protein complexes are chosen from the protein docking benchmark set 4.0 (Hwang et al. 2010) as the training set, which is shown in Appendix B For each complex, we choose 5 starting points (top-5 structure predictions from ZDOCK (Pierce et al., 2014).. In total, our training set includes 125 samples. Moreover, parameterize the search space for flexible docking as IR 12 as in BAL. The resulting fxx is fully differentiable in the search space. We only consider 100 interface atoms due to the computational concern. The number of iterations for one training epoch is 600 and in total we have 5,000 training epochs. Both BAL and UA-L20 have 600 iterations during the testing stage.
8

---

Published as a conference paper at ICLR 2022
POB 3A64
3CRL
BAL 1HER_3
1.3
1.00
1.00
50% curd Viteval
0.75
0.75
0.50
0.50
0.0
3.00
0.00
1.5-
04-120 PDB 1AK
2.00
UM 7Dn L
1.00
W-120 PDB 1HER_7
1.0
0.75
2.50
0.50
0.0
2.00
0.00
pK"
Figure 6: Estimated posterior distributions, confidence intervals and ground truth solutions for cases 1AK4-7, 3CPH.7 and IHE8_3 in protein docking.
Table 2:
Optimization and uncertainty performance of BAL and UA-L20 in protein docking.
Target PDB model
E|[x* BAT.
Xmsol 2 (A)
El[x*
Xmeelz llz E [lb ubua]?
uby. BAT.
lby (A)
(docking difficulty)
UA-L20
BAL
UA-L20
UA-I.20
JAHW (easy)
1.89
1.11
No
Yes
2.20
0.79
1AK4.7 (easy)
2.45
1.13
Yes
Yes
1.93
1.11
3CPH
(medium)
3.89
3.11
No
No
1.70
1.62
IHE8 3 (medium)
3.05
1.42
Yes
Yes
2.24
1.32
IJMO_4 (difficult)
1.45
1.87
No
No
2.90
0.67
Results. We compare UA-L2O and BAL in Table 2 For UQ UA-L20 not only achieves a more accurate confidence shown in Figure 6 and Appendix C (three of five cases had ground truths located in 90% confidence intervals), but also has a clearer trend that the better solutions correspond tighter intervals (more certain) in Figure (with Pearson correlation coefficient of 0.59 versus -0.8 for BAL). Even for optimization, UA-L20 outperforms other methods in four out of five cases. The results show significant advantages of UA-L20 compared with the state-of-the-art BAL in both UQ and optimization for protein docking.
(with
4.4 RESULT SUMMARY
We briefly summarize the results as follows.
In general, UA-L20 achieves better optimization performance with a few exceptions. This
echos the hypothesis that the assessment of uncertainty is able enhance the search efficiency and effectiveness during optimization.
UA-L20 provides the most accurate posterior estimation most of the time. This empirical evi- dence verifies the effectiveness of our uncertainty-awar L,20 inspired from theory.
Accurate posterior estimation is a key for UA-L20's superior calibration capability, such that UA-L20 generates solutions closer to optima with higher confidence and tighter confidence inter- vals, whereas competitors suffer from overconfidence or miscalibration. Such calibration capabil- ity is critical to detect the failure cases in real-world applications.
5 CONCLUSIONS
Current optimization algorithms, even with umcertainty-awareness do not address the uncertainty arising within the optimizer itself. To close the gap, in this paper we attempt to ask and address three fundamental questions, why modelling the optimizer uncertainty, what defines the optimizer uncertainty, and how to enabling UQ during optimization. We first emphasize the uncertainty arising from the optimizer is crucial source that directly responses for the end solutions deriving, except for data- and model-uncertainty. Next, we define the prior and likelihood of the optimizer which determines the optimizer posterior and the optimizer uncertainty. We further leverage learning to optimize (L20) for the optimizer parameterization, treating an optimizer as a random sample from an algorithmic space of iterative optimizers, with the end-to-end training pipeline built via variational inference. Extensive experiments show UA-L20 achieves the optimization performance, superior in two out of three test functions with the high variable dimension, best in the loss function in data privacy attack, and exceeded in the energy function for protein docking in four out of five cases. Besides, it delivers the most accurate posterior estimation and calibration capability, with the solutions closer the ground truth of larger population and tighter confidence intervals. Our study represents the first effort to recognize and quantify the uncertainty of the optimization algorithm, with extensive support from numerical results and analysis.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENT
This project was in part supported by NSF (CCF-1943008 to Ys, CCSS-2113904 to ZW). Portions of this research were conducted with the advanced computing resources provided by Texas A&M High Performance Research Computing.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we use the Machine Learning Reproducibility Checklist v2.0, Apr. 2020 (Pineau al. 2021).
For all models and algorithms presented,
clear description of the mathematical settings, algorithm, and/or model. We clearly describe all of the settings, formulations, and algorithms in Section B
clear explanation of any assumptions. We state our assumptions clearly in Sec- tion
An analysis of the complexity (time, space, sample size) of any algorithm, We do not make the analysis.
For any theoretical claim,
clear statement of the claim. We provide clear statement in Section
complete proof of the claim. We do not make theoretical proofs.
For all datasets used, check if you include:
The relevant statistics, such as number of examples. We provide all the related statistics in Section and Appendix B.
The details of train/validationneest splits. We give this information in our repository. explanation of any data that were excluded, and all pre-processing step. We give this information in repository.
A link to downloadable version of the dataset or simulation environment. Our repository contains all of the instructions to run experiments on the datasets in this work. We put in supplementary materials.
For new data collected, a complete description of the data collection process, such as instructions to annotators and methods for quality control. We do not collect or release new datasets.
For all shared code related to this work, check you include:
Specification of dependencies. We give installation instructions in the README of our repository.
Training code. The training code is available in repository.
Evaluation code. The evaluation code is available our repository.
(Pre-)trained model(s). We do not release trained models.
README file includes table of results accompanied by precise command to run to produce those results. We include README with detailed instructions to repro- duce all of our experimental results.
For all reported experimental results, check if you include:
The range of typer-parameters considered, method to select the best hyper- parameter configuration, and specification all pyper-parameters used to gen- erate results. We provide all details of hyper-parameter tuning in Section 4
The exact number of training and evaluation runs. Ten thousand independent rep- etitions are conducted for each experiments.
A clear definition of the specific measure or statistics used to report results. We provide all details of evaluation metrics in Section
description of results with central tendency (e.g. mean) & variation (e.g. error bars). We report confidence intervals for all experiments in Section
10

---

Published as a conference paper at ICLR 2022
The average runtime for each result, estimated energy cost. We do not report the running time or energy cost.
A description of the computing infrastructure used. A clear description is pre- sented in Section
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large- scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Imple- meniation (OSDI 16), pp. 265-283, 2016.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, pp. 5981-3989, 2016a.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, pp. 3981-3989, 2016b.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for typer-parament optimization. Advances in neural information processing systems, 24, 2011.
Christopher M Bishop and Michael E Tipping. Bayesian regression and classification. Nato Science Series sub Series III Computer And Systems Sciences, 190:267-288, 2003.
Yue Cao and Yang Shen. Bayesian active learning for optimization and uncertainty quantification in protein docking. Journal of chemical theory and compulation, 16(8)55334-5347, 2020.
Yue Cao, Tianlong Chen, Zhangyang Wang, and Yang Shen. Learning optimize in swarms, In Advances in Neural Information Processing Systems, pp. 15018-15022 2019a.
Yue Cao, Yuanfei Sun, Mostafa Karimi, Haoran Chen, Oluwaseyi Moronfoye, and Yang Shen. Pre- dicting pathogenicity of missense variants with weakly supervised regression. Human mutation, 4009):159-11522, 2019b.
Tianlong Chen, Weiyi Zhang, Jingyang Zhou, Shiyu Chang. Sijia Liu, Lisa Amini, and Zhangyang Wang. Training stronger baselines for learning to optimize. arXiv preprint arXiv:2010. 0.09089, 2020a.
Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. Learning to optimize: A primer and a benchmark. arXiv preprint arXiv:2103. 12828, 2021.
Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang Wang. Self-pu: Self boosted and calibrated positive-unlabeled training. In International Confer- ence on Machine Learning, pp. 1510-1519. PMLR, 2020b.
Joseph Leo Doob. Stochastic processes, volume 10. New York Wiley, 1953.
Matthias Feurer and Frank Hutter. Hyperparameter optimization. In Automated Machine Learning, pp. 3-33. Springer, Cham, 2019.
Craig Fox and Gülden Ülkümen. Distinguishing two dimensions of uncertainty. Perspectives on thinking. judging, and decision making, 2011.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confi- dence information and basic countermeasures. Proceedings of the 22nd ACM SIGSAC Confer- ence Computer and Communications Security, pp. 1322-1333, 2015.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
11

---

Published as a conference paper at ICLR 2022
David E Goldenberg. Genetic algorithms in search, optimization and machine learning, 1989.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1321-1330. JMLR. org, 2017.
Philipp Hennig and Christian Schuler. Entropy search for information global optimiza- tion. The Journal of Machine Learning Research, 13(1):1809-1837 2012.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.
Howook Hwang, Thom Vreven, Joël Janin, and Zhiping Weng. Protein-Protein Docking Benchmark Version 4.0. Proteins, 78811):3111 1-1144, November 2010. ISSN 0887-3585 doi: 10. 1002/prot. 22830. URL httt:tt::: /www ncbi mlm. www.mmmmm...mm.mmmmmmmmmm..mm..m http:/ww..w..pp.maaaaiaeeeeeeeee /ttp//////////////////////////// 95805666
Momin Jamil and Xin-She Yang. A literature survey of benchmark functions for global optimisation problems. International Journal of Mathematical Modelling and Numerical Optimisation, 4(2): 150-194, 2013.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision?, 2017.
J Kennedy and R Eberhart. Particle swarm optimization, proceedings of ieee international confer- ence on neural networks (ienn'95 in, 1995.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412,6944, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 14, 2013.
Scott Kirkpatrick, Daniel Gelatt, and Mario Vecchi. Optimization by simulated annealing. science, 220(4598);671-669, 1983.
Chaojian Li, Tianlong Chen, Haoran You, Zhangyang Wang, and Yingyan Lin. Halo: Hardware- aware learning to optimize. In European Conference on Computer Vision, pp. 500-518. Springer, 2020.
Ke Li and Jitendra Malik. Learning to optimize. arXiv preprint arXiv:1606.0/885 2016.
Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hypernet- works. arXiv preprint arXiv: 1802.0091 2018.
Gangming Lyu, Yuning You, Ang Li, Xuewen Liao, and Christos Masouros. Probabilistic construc- tive interference precoding for imperfect csit. IEEE Transactions on Vehicular Technology, 70(4): 3932-3937, 2021.
lain Moal and Paul Bates. SwarmDock and the Use of Normal Modes in Protein-Protein Dock- ing. International Journal of Molecular Sciences, ((10))3623336444 September 2010. ISSN 1422-0067. doi: 10.3390/jms11110330 URL https::///// /tfp::://wwww.. www. ncbi. nlm. nih gov/pme//////oppoogooooooo articles/PMC299680
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Stand-alone and federated learning under passive and active white-box inference attacks, arXiv preprint arXiv:1812.009)) 2018.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Pedro Ortega, Jordi Grau-Moya, Tim Genewein, David Balduzzi, and Daniel Braun. A nonparamet- ric conjugate prior distribution for the maximizing argument of a noisy function. Advances in Neural Information Processing Systems, pp. 3005-3013, 2012.
12

---

Published as a conference paper at ICLR 2022
Vikas Pejaver, Jorge Urresti, Jose Migo-Martinez, Kymberleigh A Pagel, Guan Ning Lin, Hyun- Jun Nam, Matthew Mort, David N Cooper, Jonathan Sebat, Lilia M lakoucheva, et al. Mutpred2: inferring the molecular and phenotypic impact of amino acid variants. BioRxiv, pp. 134981, 2017.
Brian G. Pierce, Kevin Wiehe, Howook Hwang, Bong-Hyun Kim, Thom Vreven, and Zhip- ing Weng. ZDOCK server: interactive docking prediction of protein-protein complexes and symmetric multimers. Bioinformaties, 30(12):1771-1773, 02 2014. ISSN 1367-4803, doi: 00.100/bionfommaiicsbb000. URL https: btu097
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research. Journal of Machine Learning Research, 22:1 22,1-20, 2021.
K. A. Porter, I. Desta, D. Kozakov, and S. Vajda. What method to use for protein-protein docking? Curr. Opin. Siruct. Biol., 55:1-7, 04 2019.
Jiayi Shen, Xiaohan Chen, Howard Heaton, Tianlong Chen, Jialin Liu, Wotao Yin, and Zhangyang Wang. Learning a minimax optimizer: A pilot study. In International Conference on Learning Representations, 2021. URL
net htwpoppeereeeeeeeeeeeeeeeeeeeeee
Graham R Smith and Michael JE Sternberg. Prediction of protein-protein interactions by docking methods. Current opinion in structural biology, 12(1):28-355 -5, 2002.
Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian pro-
cess optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995, 2009.
Emmanuel Vazquez and Julien Bect. Convergence properties of the expected improvement algo- rithm with fixed mean and covariance functions. Journal of Statistical Planning and inference, 140(11):3088-3095 2010.
Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient bayesian optimization. In International Conference Machine Learning, pp. 3627-3635, 2017.
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. arXiv preprint arXiv:2006. 35770, 2020.
Mingzhang Yin and Mingyuan Zhou. Semi-implicit variational inference. In International Confer- ence on Machine Learning, pp. 5660-5669, PMLR, 2018
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. L2-gen: Layer-wise and learned efficient training of graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2127-2135, 2020.
Anatoly A Zhigljavsky. Theory of global random search, volume 65. Springer Science & Business Media, 2012.
13

---

Published as a conference paper at ICLR 2022
APPENDIX
A CONFIDENCE ESTIMATION PROCEDURES
In evaluation, for the confidence versus optimization performance scatter plot, we first perform Monte Carlo sampling on the solution posterior, and then bin the solutions to estimate their fraction and corresponding average distance to the optimum. Detailed procedures can be found in Algorithms
Algorithm 1: Confidence estimation with corresponding optimization performance
Input: Sampled solutions x" with size N, estimated solution x, known optimum Xine bin width W:.
Initialize: S = 0. 1. Calculate |x" - 8|12 for all sampled solutions, and bin them with w.
for each bin b do
2. Calculate the population fraction |6|/N.
3. Calculate average distance to Xtree Mean( {||x Xxmell2|x* E bJ).
4. S + S u (16//N, Mean( {||x* Xinee |||||| xx b))).
end for
Return: Scatter point set S.
B DATA DESCRIPTION
The analytical forms three test functions are shown in Table 3 The meta-training data are broad family of similar functions (Cao et al. 2019a). For Rastrigin, it is fD(x) = || bll EE 10 (2s.)) 10D where A, c sampled from i.i.d normal distributions; for Ackley, is fD(x) 20 cx. 0 Ax b1 > cxp(cos(2nz:)/D + exp(1) + 20; for Griewank, it is fo = De,11 Ax b||3/4000 c -II cos().)
For privacy attack, we use 9-dimension features of privacy attack data generated from MutPred2 (Pejaver et al.) 2017), which captures mutational impacts protein structure, dynamics, and func- tion, and grouped hierarchically into custom oncology based on their inherent relationships. The list of features can be found in Table The meta-training is performed on 1,593 patients and evaluate in 318 patients (Cao et al. 20195).
For protein docking, the PDB IDs of meta-training samples are shown in Table and we in proteins with PDB in Table 2
Table 3: The analytical forms of test functions fo (x) with the dimension D, and their corresponding flatness around global optima, estimated with the norm of Hessain matrix.
Function
Analytic form
Rastrigin
fn(x) = |x|
ΣD_ 10 + 10D
Ackley fD(x) 20exp(00xppppp00 -0..00002000000 /0.5|
De (xp(c)s(2Tz)) D) + exp(1) + 20
Griewank
fo(x) = 1 Σb | 1000
I cos(2;)
C MORE EVALUATION RESULTS
We provide more evaluation results follows. (i) The ablation of À values three test functions is shown in Figure| 7 (ii) The UQ results for test cases IAHW and 1JMO 4 in protein docking are plotted in Figure (iii) Confidence intervals corresponding results in Figure
14

---

Published as a conference paper at ICLR 2022
Table 4: 9-dimension feature description for data privacy attack experiment.
Feature index
Property/molccular impact
Relative solvent accessibility
Allosteric site
3
Catalytic site
4
Secondary structure
Stability and conformation flexibility
Special structural signatures
7
Macromolecular binding
Metal binding
PTM site
Table 5: 4-letter of proteins used in protein docking training
Difficulty Ievel
Protein Data Bank (PDB) code
INSO, 7CEI, IDFI, 1AVX, IBVN, HQQ,, ICGI, 1MAH, 1EZU, UPS. 1PPE, IROR, 2125, 2B42, TEAW, 2JHL, 18J1, 1KXQ, IEWY
Rigid
Medium
IXQS. IM10, IOK, IGRN IIBR, IATN
Flexible
Rastrigin
LEos 0.91
0.002
0.1
10
0,000
12
15
18
21
24
27
30
3
9
12
15
10
21
24
27
Iene 0.81
0.002
0.000
12
15
10
21
24
27
30
12
15
18
21
24
27
30
9
12
15
10
21
27
Ackley
1oa3 00.9
4.0
3.5
0.000
6
12
15
18
21
24
27
30
3
15
18
21
24
27
30
3.0
2.5
1Eo8 0.8]
0.02
2.0
12
18
24
30
15
21
Griewank
0.91 0.91
0.2
2.5
12
27
15
21
Fa 180
0.2
52.
12 18
12
21
27
Dimension
Figure 7: Optimization and uncertainty performance of UA-L20 with different À in three non-convex test functions. Different row represents different functions, and each column stands for: (i) 1st column is optimiza- tion performance. lower the better; (ii) 2nd column is precision of the estimated confidence, lower denoting more accurate posterior estimation; (iii) is the length of the confidence interval: only when with similar precision of the estimated contidence would a tighter interval indicating more certain solutions.
Table 6: The optimizer distributions over hyper- parameters in Adam and PSO.
Methods
Optimizer Distribution Settings
Adam
1og U[-2, -11, 8, ~UJ0,,, 1.0], 8, U]0.999, 1.0] U[O.5, 1.5], UJL.S, 2.5]. C2 U[I.5,
PSO
15

---

Published as a conference paper at ICLR 2022
BAL in PDB LAHW 3
BAL in PDB 1JMO_4
2.0
2.0
90% confidince interval
1.5
1.5
Density
1.0
0.5
0.5
0.0
0.0
2
5
2
3
4
6
UA-L20 in PDB LAHW 3
UA-L20 in PDB 1JMO_4
2.0
2.0
1.5
1.5
Density
1.0
1.0
0.5
0.5
0.0
0.0
2
5
2
3
Ix
X22
Figure 8: Estimated posterior distributions, confidence intervals and ground truth solutions for cases IAHW 3 and IJMO_4 4 in protein docking.
Rastrigin
Ackley
Griewank
2.4
2.2
3
2.0
1.8
1.8
12
15
10
21
24
27
30
12
15
18
21
24
27
30
12
15
18
21
24
27
30
1.0
2.5
1.6
2.0
1.4
1.0
0
Dimension
Dimension
Dimension
Figure 9: The corresponding confidence intervals to results in Figure 3 secondary metric and fie-breaker for uncertainty calibration accuracy and measure of the length of the confidence interval: only when the estimated confidence levels are similarly precise, tighter intervals indicates higher certainty about solutions
D COMPUTATION OF UA-L20 OBJECTIVE FUNCTION
We first rewrite the evidence
logp(zr (Z0,) = f log 21, )d / q( 0) log p(zr, do
p(Ø(zT)
= f <( 0) log p(z (1 (2 ((0: 0) log )db
<(<;: 0) log (Φ;00) 0) dΦ / <( 0) log
((: 0)
de
plz,, 2 Z
p(0lZT)
[T 0) |p(z), Φ 2 + KL[g( :: ())
where (b: [p([ ol.,.)]
ELBO and KL(g( (:: 0) ||p(pp/z))) zr) our objective. Therefore,
minimizing KL[g 0) ((()))) is equivalent to maximizing EL.BO (Yn & Zhou 2018).
16

---

Published as a conference paper at ICLR 2022
Rastrigin, dimension=11
Ackley, ey, dimension=155
Griewank, dimension=15
Estimated probability
Adam, 80
0.08
0.002
PSO
B0
fxx"
BAL
DM LSTL
UA-L20
0.00
2
4
10
15
20
25
Rastrigin, dimension=18
Acki ey, dimension-18
Griewank, dimension=118
Estimatedd probabiiity
0.10
0.08
0.002
0.06
0.2
0.01
0.04
of
0.02
0.00
0.00
0.0
10
2
10
||x
X
IK"
Xtru
Xerur
versus corresponding Xme// in three test functions. The peak closer to left side indicates the solution populations are in high confidence if they are closer to the optima.
Figure Scatter plot confidence solution regions segmented by ||x > optimization performance ||x
*||2.
0.20
Estimated probablitt [x
Adam_ _0
0.15
PSO
PSO
0.10
BAL
DM_L5TH LSTM
0.05
UA-L20
0.00
0.4
0.6
0.8
1.0
[x"
xxvel2
Figure 11: Seatter plot of confidence for solution regions segmented by ||x" x||2, versus the corresponding optimization performance ||x Xinel/a 2/2 in data privacy attack.
We rewrite our objective as:
(L(g((:0))pp(d) 2T) E )0)00) { log g( 0) 1 -lloog p( zT)
E log ( 0) logr(d) Beo Φ) 2
Epr((: { log (0;;0 0)
log logplzzl Zeo +) +
p())
1 f(x{) KL(g( +
E ABLATION STUDY
We perform ablation studies sample size in uncertainty quantification of UA-1.20 in Table 7 mean versus minimum of computation and wall-clock time comparison Table g on privacy attack experiment on gene BRCA1/2 in Table g Results show UA-L20 is not sensitive to sample size or mean/minimum calculation of &, and it is comparably efficient with competitor optimizers.
Moreover, we include study of the wall-time versus dimensionality ranging from t0 2,000, using the optimization of Rastrigin function in Figure 10] We report UA-L2O's 's wall-time for both meta- training and meta-testing, The results above show that the meta-tre meta-training time grows almost linearly with the dimensionality when dimensionality > 200 (which is partly attributed to the fact that the number of samples is now adopted as value correlated with the dimension). So meta-training
17

---

Published as a conference paper at ICLR 2022
in millions of dimensions for deep-learning model parameters is expensive yet still manageable, especially using multiple GPU cores. Most importantly, meta-testing was much faster and the time remained flat with regards to the dimensions, that once UA-L20 is trained, its deployment remains fast and scales well to the dimensionality. fact, 1.20 usually takes the setting of offline training (which could be time-consuming) and online deployment (which is fast enough and scales well) Chen et al. (2021).
We further compare with baselines learning rate ensemble and stochastic gradient MCMC of Adam, shown in Table IT We also make comparison with warm-start UA-L20 to show the im- provement of UA-L2O training.
Table 7: Ablation of sample size in posterior estimation BRCA 1/2 privacy attack.
Sample size
E x
Xine |
<0.9
0.9
cn.s 0.8]
To.3
FD.8
100
0.30
0.25
0.25
0.06
0.06
1,000
0.30
0.27
0.31
0.06
0.04
10,000
0.30
0.25
0.27
0.06
Table 8: Ablation of mean/minimum & computation in posterior estimation on BRCA1/2 privacy attack.
E||x* Xine 2
|c0.9
0.9
|co.s
0.8]
Vo.9
Vo.a
Mean
0.31
0.27
0.26
0.07
0.04
Min
0.30
0.25
0.27
0.06
0.04
Table 9: Comparison of wall-clock time on BRCA1/2 privacy attack.
Adam
Adam BOO
PSO
PSO BO
BAL
DM
UA-L20
Wall-Clook Time
30
33
22
Table Comparison wall-cloce clock time on Rastrigin function with different dimensionality.
Dimension
Mcta-training (s)
Meta-test (s)
10
607
36
100
200
662
37
500
1,059
38
1,000
1,998
38
2,000
3,904
38
Table 11: More baseline comparison of UA-L20 privacy attack of gene BRCA1/2.
B
Xine la
[c.99
0.9
<<o.8 0.8)
7o.9
To.8
UA-1.20 Warm-Start Only
0.76
0.55
0.57
01.0
0.30
UA-L20
0.30
0.25
0.27
0.06
0.04
F UTILIZATION OF OPTIMIZER UNCERTAINTY
We demonstrate the "usefulness" of the proposed optimizer uncertainty via performing the experi- ment of "failure detection" as a basic downstream task.
Specifically, an optimization solution is defined "failure" if its distance to the global optimum beyond threshold 0.5vd / (where d is the dimension and the threshold was chosen for approxi- mately balanced success/failure cases). Instead of fixing the confidence level or and estimating the confidence interval (as observed in Figure 4 that better optimization correlated better with tighter confidence intervals), here let each method estimate the confidence level at fixed 0.5vd ("success")) for the optimization solution in each trajectory and use a to classify the quality of such solutions over 1000 trajectories. The classification performance assessed using the under
18

---

Published as a conference paper at ICLR 2022
the receiver-operator characteristic curve (AUROC) and the area under the precision-recal curve (AUPRC). The results on privacy attack are shown Table 12 indicating the best posterior estima- tion of UA-L20.
Table 12: "Failure" case detection with optimizer uncertainties privacy attack.
Adam
Adam BO
PSO
BAL
DM LSTM
UA-L20
AUROC
0.53
0.59
0.74
0.78
0.00
0.37
0,99
AUPRC
0.97
0.98
0.73
0.90
0.00
0.95
0,99
19


Reviews:
[{"review_id": "zEfxowZ6Dad", "reviewer": "Reviewer_3nHB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is generally well-written and easy to follow. The idea of quantifying the uncertainty of the optimization algorithm is novel and interesting. However, there are some unclear points that make me not sure of the acceptance of the paper.\n\n1) Motivation is not clear - why do we need optimizer uncertainties?\n\nThe \"contributions\" section in the introduction states that the paper discusses the fundamental question of 1) why modeling optimizer uncertainty, and in my opinion, this would be the most important question to be discussed. However, it is not clear from the paper why the optimizer uncertainty really matters.  At least, judging from the experiments, uncertainty-aware optimizers do not always give superior optimization results in terms of error, as can be seen from the results for the synthetic functions. UA-L2O seems to improve calibration, so the optimizer uncertainty somehow increases for the difficult optimization problems, and the authors stated in section 4.4 that \"such calibration capability is critical to detect the failure cases in real-world application\".  However, I don't think better calibration can actually lead to better detection of failure cases. For instance, in the synthetic experiments, the uncertainty calibration metrics displayed in the second and third columns do not show significant improvement over baselines without uncertainty; I mean UA-L2O is indeed better for some cases, but the margin is not that significant, so it is not clear we can actually make use of such uncertainty to actually detect the failure cases. In order to argue this, 1) one should first clearly define what a \"failure\" case is, and 2) do some experiments to run multiple optimization runs, and 3) do the binary classification based on the estimated uncertainty and compute AUROC to quantitatively show that we can actually benefit from the estimated uncertainty in terms of detecting failure cases. For the purpose of detecting failure cases, I can also think of very naive baselines, such as just monitoring the validation loss or training loss values and treating the ones with loss values larger than a certain threshold as a failure.  \n\n2) Important details are missing\n\nAs written in the review of the paper, L2O algorithms usually require a meta-training stage where the learned optimizer is trained with multiple optimization runs for the tasks sampled from some specific task distribution. Such meta-training is indeed time-consuming, and requires care in design; for instance, we need a diverse task distribution to guarantee the generalizability of the learned optimizers. However, I fail to see the detailed description of the meta-training protocols, other than the description of the ELBO being used for the inner optimization objective.  Also, it is not clear how the Monte-Carlo estimation is done for the actual deployment of the learned optimizer; how many samples are required? Are the samples drawn at the initial stage of the optimization or drawn at each iteration of the optimization? How sensitive is the algorithm to the number of samples? How does the algorithm scale with the number of samples being used? I particularly think the last question is important because it is directly related to the practicability of the algorithm; at least, I'd like to see the wall-clock time of the UA-L2O compared to non-Bayesian baselines.\n\n3)  The \"space of algorithms\" is a slight over-statement\n\nAlthough I find the notion of the \"space of the optimization algorithms\" quite appealing, the actual implementation given in the paper seems rather disappointing; as the authors already stated in the paper, there are many elements to be considered for a single optimization run. Needless to say the parameters $\\phi$ for the learned optimizers, we also have the things like initialization scheme (or initial distribution for the parameters and their hyperparameters),  learning rate schedule, batch size, number of epochs to run, and so on. The uncertainty modeled for optimizer in this paper is restricted to the meta-parameter $\\phi$, so I think to say that the proposed algorithm considers a \"random sample of an optimizer $g$ from the space of optimizers $G$\" is a slight over-statement. \n\n4) Some important baselines are missing\n\nThe easiest baseline I can think of is ensembles; each run of an ensemble (with different random seeds) can be considered as a sample from an implicitly defined space of optimizers. Ensembles are known to excel in many applications both in terms of predictive accuracy and uncertainty calibration (although such uncertainty is for prediction, not like the optimizer uncertainty discussed in this paper). The downside of ensembles is that they require a longer training time than the usual optimization algorithms, but L2O also requires a considerably long meta-training time. So I think it is definitely worth comparing the proposed approach to (deep) ensembles with non-Bayesian optimizers. Another baseline I think worth comparing is stochastic gradient MCMC algorithms such as stochastic gradient Langevin dynamics or stochastic gradient Hamiltonian Monte-Carlo; such algorithms can be understood as noise-injected versions of gradient-based optimization algorithms, and samples collected from such samples can also be used to construct optimizer uncertainty (at least in terms of proximity to the true optima). \n\nConsidering the above-mentioned concerns, I recommend rejection, but I'm happy to discuss with the authors. Please let me know if I'm misunderstanding something.\n", "summary_of_the_paper": "This paper proposes a Bayesian Learning to Optimize technique called Uncertainty-Aware Learning to Optimize (UA-L2O). The main contribution of this paper is its introduction of the concept of the space of optimization algorithms and treating an optimizer as a random sample from it, and deriving a proper Bayesian inference algorithm based on that concept. The space of optimization algorithms, to simplify, is expressed as a space of parameters of learned optimizers $\\phi$, and a Gaussian prior is placed on those parameters. The posterior inference procedure is approximated with variational inference with reparameterization tricks, and the actual deployment of the learned optimizer is done with Monte-Carlo approximation with learned variational distribution for the optimizer parameters $\\phi$. The proposed algorithm is applied to various synthetic and real-world optimization problems and showed some useful calibration behaviors.", "summary_of_the_review": "The idea is interesting, but there are a few points to be addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "xFRsZBz39yj", "reviewer": "Reviewer_x9M2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In the very beginning, you say that no optimization algorithm can guarantee global optimum in general. It almost sounds like you say that your solution can. Do the authors agree with this?\nAlso, Bayesian optimization *in theory* can guarantee global optimization, but in practice probably not.\n\nI would like the authors to give concrete ideas of how the space G looks. Does it contain usual optimizers such as Adam? And different levels of momentum etc. \nIs it always a good idea to just make this space so high-dimensional?\n\nOn page 4 you say you extend the \"Bayesian treatment\". Can the authors expand on what they mean? Do you use more then Bayes theorem?\n\nWhy does MCMC lead to degenerate solutions? My belief is that MCMC in general is more accurate than variational inference, so is there something to worry about here?\n\nThe first equality in (4) is not the ELBO. The KL distance here is the difference between the ELBO and the marginal log likelihood.\nCan the authors do the computations in the second equality for me?\n\nCan I use the variation in g to conclude when convergence has happened? \n\nYou mention, but only very briefly, warm-starting theta. How is this done in practice? And how good is the solution immediately after warm-starting? In other words, how much improvement is done after this warm-start?\n\nI found the experiment section very difficult to read, and therefore I am not convinced of the method's superiority over the compared methods. \nAlso, in future versions, I think a comparison with some hyper-parameter optimization is necessary. \n\n", "summary_of_the_paper": "The paper introduces uncertainty-aware learning-2-optimize methodology. In particular, they parametrize a optimizer space as a neural network g and use variational inference to compute approximate posterior of g and the attained best solution x*.", "summary_of_the_review": "There are some clear errors in the paper, and I am not convinced that the empirical performance justifies this methodology to be accepted at ICLR. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "w7Ywbssayg", "reviewer": "Reviewer_Q3k1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Treating optimizers as random elements in an algorithmic space is a novel concept to the best of my knowledge. Endowing priors on updating rules and defining likelihoods on the iterates also seem to be new in the Bayesian nonparametrics literature. In addition to data and model uncertainties, this paper brought attention to the role and importance of modeling optimizer uncertainty.\n\nHere are some comments / questions:\n1. It is known that the variational distribution often underestimates uncertainty, e.g., see ``Covariances, Robustness, and Variational Bayes\" by Giordano et al. (2018). However from Figure 3, it seems that UA-L2O is still able to capture the estimated confidence quite well for low dimensions. So how did UA-L2O solve this underestimation problem?\n\n2. Is the neural network architecture of $g$ fixed? In the experiments, it seems to me that the authors modeled $g$ using LSTM. However this contradicts with Definition 1 because $g$ is allowed to change according to the prior?\n\n3. Why $\\hat{\\mathbf{x}}$ is not computed using mean of the posterior distribution of the solutions $\\mathbf{x}^*$? Is there a reason to use the minimum of the generated solutions and not the posterior mean?\n\n4. UA-L2O seems to have trouble optimizing Rastrigin, as evident by the plot of $E||x^*-x_{\\text{true}}||_2$ against dimension $D$ in Figure 3, which looks like it is increasing to infinity. However the structure of Rastrigin seems to be the simplest among all three test functions. Can the authors give some intuition behind this?\n\n5. For Ackley and Griewank in Figure 3, is there a reason why there is a dip in dimension $27$ in the precision of the estimated confidence (2nd column)?\n\n6. In (3), what is $\\mathbf{x}_t^{\\phi}$?\n\n7. There are quite a number of grammar mistakes and some parts are quite puzzling. For example, the line after (2), Prior arts on hyper-parameter?\n\nTYPOS:\n1. The first line in Section 3.1, am -> an\n2. In (3), $\\exp(-\\lambda\\|\\boldsymbol{\\phi}\\|_2^2)$ -> $\\exp(-\\frac{1}{2\\lambda}\\|\\boldsymbol{\\phi}\\|_2^2)$", "summary_of_the_paper": "The authors looked at uncertainty quantification (UQ) of optimization algorithms. Working under the L2O framework and adopting the Bayesian approach, they used a neural network to model the updating rule, and endowed the network weights with a prior. By treating the solution iterates and their gradients as data, the authors defined a conditional likelihood on them. The resulting posterior on the network weights is then approximated using a variational algorithm. To conduct inference, the authors computed the predictive density of the solution through Monte Carlo sampling. This end-to-end pipeline is called uncertainty aware L2O (UA-L2O) and it is applied to quantify uncertainty in various learning tasks.", "summary_of_the_review": "I think UA-L2O is a novel and promising method that might lead to further research in quantifying optimizer's uncertainty.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MCwHh_aYyos", "reviewer": "Reviewer_vJE3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the main idea of the paper novel and interesting. The experimental section is extensive and shows the benefit of using the method across different applications. However I believe Section 3 of the paper is hard to follow and understand. Particularly, I think the authors should clarify the following points:\n\n1) Provide an intuition for what the likelihood and the prior represent. What is $p(g|\\mathbf{z}_t)$ that is  $p(g|\\mathcal{D})$ representing? In standard Bayesian inference $p(\\theta|\\mathcal{D})$ gives how probable is $\\mathcal{D}$ to be generated from $\\theta$. Is it correct to say that in your setting $\\mathcal{D}$ are the data points $(x_0….x_t)$ evaluated via optimization (and potentially the gradients) and $p(g|\\mathcal{D})$ represent the probability of $g$ being the optimizer used to generate the data, ie used to optimize? \n2) It is not clear to me why you are starting from an optimizer $g$ and the you speak about $\\phi$. It seems to me that $g$ is a deterministic function parametrized by $\\phi$ so I don't see the point of having both eq 2 and eq 3.\n3) Definition 1. In point 2 there is no $\\mathbf{x}$ on the left but the distribution on the right factorizes across $\\mathbf{x}_i$. Are you making the assumption that $\\mathbf{x}_i = \\mathbf{z}_i$ or is $\\mathbf{z}_t$ a set of historical information (iterates + gradients) as specified above?\n4) How exactly are you sampling from (5)? \n5) Could you specify the form of the assumed variational distribution $q(\\cdot)$ in Section 3.2?\n6) It seems like the paper is highly based on L2O, it would be useful to be more explicit about what you are improving with respect to that paper. Even thought that method has no uncertainty estimation it would be nice to have it in the experimental session? also, most baselines under comparison are non-Bayesian methods and the primary goal of the paper is to quantify uncertainty. Why are you not comparing to BO? \n7) In the experimental session, in the paragraph \"Evaluation metrics\", $\\mathbf{x}^*$ is sampled from $p(\\mathbf{x}^*)$. Is this supposed to be $p(\\mathbf{x}^* |\\mathbf{z}_T)$ as given in Eq 5?\n8) It is not clear to me why a lower value of $r_{\\sigma}$ should be preferred? My understanding is that the bounds in Eq 6 are computed by sampling from Eq 5 and then getting the lowest value for $\\hat{\\mathbf{x}}$. The true value $\\mathbf{x}_{true}$ is not involved thus we could have a narrower posterior but be off the true optimum. \n\n\n**Minor comments**\n1) What do you mean by “**Optimizing** the posterior via Markov chain Monte Carlo (MCMC)”? Do you mean sample from the posterior distribution? Or computing the MAP? I don't see how that would lead to a degenerate posterior distribution.\n2) Figure 1 could be used to explain the methodology better. It is not mentioned in the text and the caption is to short to be informative.\n3) It would be nice to include a discussion of why using a NN to model the optimizer is a good choice\n", "summary_of_the_paper": "This paper considers the problem of quantifying the uncertainty of an optimizer used to do inference on a given model. The authors take a Bayesian approach and treat the optimizer as a random variable in the space of algorithms. They derive an approximate posterior via variational inference.\n", "summary_of_the_review": "I really like the idea of considering the additional level of uncertainty and I find it novel. However, I found the methodological session difficult to follow and I think that could be improved by providing an intuition on the prior on $g$ and the connected posterior. At the practical level it is not clear to me how the proposed approach could improve the performance of existing algorithms. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewers place this manuscript right at the threshold of acceptance, I find the revisions that they have made to address the majority of the reviewers concerns. That, combined with some of the reviewers' scores being slightly miscalibrated with their (largely positive) reaction to the author feedback, I am advocating for this paper to be accepted.
Decision: Accept (Poster)