A Study of Bayesian Neural Network Surrogates for Bayesian Optimization
Authors: Yucen Lily Li
Tim G. J. Rudner
Andrew Gordon Wilson
Venue: ICLR 2024 poster (2024)
Keywords: Bayesian Optimization
Gaussian Processes
Bayesian Neural Networks
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We conduct a study of Bayesian neural networks as surrogate models for Bayesian optimization.

Abstract: Bayesian optimization is a highly efficient approach to optimizing objective functions which are expensive to query. These objectives are typically represented by Gaussian process (GP) surrogate models which are easy to optimize and support exact inference. While standard GP surrogates have been well-established in Bayesian optimization, Bayesian neural networks (BNNs) have recently become practical function approximators, with many benefits over standard GPs such as the ability to naturally handle non-stationarity and learn representations for high-dimensional data. In this paper, we study BNNs as alternatives to standard GP surrogates for optimization. We consider a variety of approximate inference procedures for finite-width BNNs, including high-quality Hamiltonian Monte Carlo, low-cost stochastic MCMC, and heuristics such as deep ensembles. We also consider infinite-width BNNs, linearized Laplace approximations, and partially stochastic models such as deep kernel learning. We evaluate this collection of surrogate models on diverse problems with varying dimensionality, number of objectives, non-stationarity, and discrete and continuous inputs. We find: (i) the ranking of methods is highly problem dependent, suggesting the need for tailored inductive biases; (ii) HMC is the most successful approximate inference procedure for fully stochastic BNNs; (iii) full stochasticity may be unnecessary as deep kernel learning is relatively competitive; (iv) deep ensembles perform relatively poorly; (v) infinite-width BNNs are particularly promising, especially in high dimensions.

Published as a conference paper at ICLR 2024
A STUDY OF BAYESIAN NEURAL NETWORK SURROGATES FOR BAYESIAN OPTIMIZATION
Yucen Lily Li, Tim G. J. Rudner, Andrew Gordon Wilson New York University
ABSTRACT
Bayesian optimization is highly efficient approach to optimizing objective func- tions which are expensive query. These objectives are typically represented by Gaussian process (GP) surrogate models which are easy to optimize and support ex- act inference. While standard GP surrogates have been well-established in Bayesian optimization, Bayesian neural networks (BNNs) have recently become practical function approximators, with many benefits over standard GPs such as the ability naturally handle non-stationarity and learn representations for high- high-dimensional data. In this paper, we study BNNs as alternatives to standard GP surrogates for optimization. We consider a variety of approximate inference procedures for finite- width BNNs, including high-quality Hamiltonian Monte Carlo, low-cost stochastic MCMC, and heuristies such as deep ensembles. We also consider aifinit-wiithh BNNs, linearized Laplace approximations, and partially stochastic models such as deep kernel learning. We evaluate this collection of surrogate models on diverse problems with varying dimensionality, number of objectives, non-stationarity, and discrete and continuous inputs. We find: (i) the ranking of methods is highly problem dependent, suggesting the need for tailored inductive biases; (ii) HMC is the most successful approximate inference procedure for fully stochastic BNNs; (iii) full stochasticity may be unnecessary as deep kernel learning is relatively com- petitive; (iv) deep ensembles perform relatively poorly; (v) infinite-width BNNs are particularly promising, especially in high dimensions.
INTRODUCTION
Bayesian optimization (O'Hagan, 1978) is a distinctly compelling success story of Bayesian inference. In Bayesian optimization, we place a prior over the objective we wish to optimize, and use a surrogate model to infer posterior predictive distribution over the values of the objective at all feasible points in space. We then combine this predictive distribution with an acquisition fumction that trades-off exploration (moving to regions of high uncertainty) and exploitation (moving regions with a high expected value, for maximization). The resulting approach converges quickly to global optimum, with strong performance in many expensive black-box settings ranging from experimental design, to learning parameters for simulators, to hyperparameter tuning (Frazier, 2018; Garnett, 2023).
While many acquisition functions have been proposed for Bayesian optimization (e.g. Frazier et al., 2008; Wang and Jegelka, 2017), Gaussian processes (GPs) with standard Matérn or RBF kernels are almost exclusively used as the surrogate model for the objective, without checking whether other alternatives would be more appropriate, despite the fundamental role that the surrogate model plays in Bayesian optimization.
Thus, despite promising advances in Bayesian optimization research, there is an elephant in the room: should we be considering other surrogate models? It has become particularly timely to evaluate Bayesian neural network (BNN) surrogates as alternatives to Gaussian processes with standard kernels: In recent years, there has been extraordinary progress in making BNNS practical (e.g. Daxberger et al., 2021; Khan and Rue, 2021; Rudner et al., 2022; Tran et al., 2022; Wilson and Izmailov, 2020). Moreover, BNNS can flexibly represent the non-stationary behavior typical of optimization objectives, discover similarity measures as part of representation learning which is useful for higher dimensional inputs, and naturally handle multi-output objectives. In parallel, Monte-Care acquisition functions (Balandat et al., 2020) have been developed which only require posterior samples, significantly lowering the barrier to using non-GP surrogates that do not provide closed-form predictive distributions.

---

Published as a conference paper at ICLR 2024
In this paper, we exhaustively evaluate Bayesian neural networks surrogate models for Bayesian optimization. We consider conventional fully stochastic multilayer BNNs with a variety of approximate inference procedures, ranging from high-quality full-batch Hamiltonian Monte Carlo (Izmailov et al., 2021; Neal, 1996; 2010), to stochastic gradient Markov Chain Monte Carlo (Chen et al., 2014), to heuristics such as deep ensembles (Lakshminarayanan et al., 2017). We also consider infinite- width BNNS (Lee et al., 2017; Neal and Neal, 1996), corresponding to GPS with fixed non-stationary kernels derived from a neural network architecture, as well partially Bayesian last-layer deep kernel learning methods (Wilson et al., 2016). This particularly wide range of neural network- based surrogates allows us to evaluate the role of representation learning, -otsstaiionarity and stochasticity in modeling Bayesian optimization objectives. Moreover, given that so much is unknown about the role of the surrogate model, we believe is particularly valuable not to have a "horse in the race", such as special BNN model particularly designed for Bayesian optimization, in order to conduct an unbiased scientific study where any outcome is highly informative.
We also extensively study a variety of synthetic and real-world objectives-with wide range of input space dimensionalities, single- and multi-dimensional output spaces, and both discrete and continuous inputs, and non-stationarities.
Our study provides several key findings: (1) while stochasticity is often prized in Bayesian optimiza- tion (Garnett, 2023; Snoek et al., 2012), due to the small data sizes in Bayesian optimization, fully stochastic BNNS do not consistently dominate deep kernel learning, which is not stochastic about network parameters; (2) of the fully stochastic BNNs, HMC generally works the best for Bayesian optimization, and deep ensembles work surprisingly poorly, given their success in other settings; (3) on standard benchmarks, standard GPs are relatively competitive, due to their strong priors and simple exact inference procedures; (4) there no single method that dominates across most problems, demonstrating that there is significant variability across Bayesian optimization objectives, where tailoring the surrogate to the objective has particular value: (5) infinite-widhh BNNs are surprisingly effective high-dimensional optimization. These results suggest that the -u-Euulidean similarity metrics constructed from neural networks are valuable for -igh-dimensional Bayesian optimization, but representation learning (provided by DKL and finite-width BNNs) is not as valuable as a strong prior derived from a neural network architecture (provided by the infinite-width BNN).
This study also serves as an evaluation framework for considering alternative surrogate models for Bayesian optimization. Our code is available at https: / /////gtttttt www.ittuu.ooooooo
wttpuu/..ooo/cuenii..e hpc//luccennllll /bbnnbbb...
2 RELATED WORK
There is a large body of literature on improving the performance of Bayesian optimization. However, an overwhelming majority of this research only considers Gaussian process surrogate models, focusing on developing new acquisition functions (c.g. Frazier et al.. 2008; Wang and Jegelka, 2017), additive covariance functions (Gardner et al., 2017: Kandasamy al., 2015), using gradient information (Wu et al., 2017), multi-objectives (Swersky et al., 2013), trust region methods that use input partitioning for higher dimensional and statioatt data (Eriksson et al., 2019), and covariance functions for discrete inputs and strings (Moss et al., 2020). For a comprehensive review, see Garnett (2023).
There has been some prior work focusing on other types of surrogate models for Bayesian optimiza- tion, such as random forests (Hutter al., 2011) and structuuet Parzen estimators (Bergstra et al., 2013). Snoek et al. (2015) apply Bayesian linear regression model to the last layer of deterministic neural network, which can be helpful for the added number of objective queries associated with higher dimensional inputs. Deep kernel learning (Wilson et al., 2016), which transforms the inputs of a Gaussian process kernel with a deterministic neural network, may also be used with Bayesian optimization, especially in specialized applications like protein engincering (Stanton et al., 2022). The linearized- Laplace approximation to produce a linear model from a neural network has also recently been applied to Bayesian optimization (Kristiadi al., 2023). Neural networks have also been used for Bayesian optimization in the the contextual bandit setting, using the neural tangent kernel for exploration (Zhou et al., 2020; Dai al., 2022; Lisicki al., 2022).
Despite the recent practical advances in developing Bayesian neural networks for many tasks (e.g. Wilson and Izmailov, 2020), and recent Monte- Carloo acquisition functions which make it easier to surrogates like BNNs that do not provide closed-form predictive distributions (Balandat et al., 2020), there is a vanishingly small body of work that considers BNNS as surrogates for Bayesian optimization. This is surprising, since would indeed expect BNNS to have properties naturally
2

---

Published as a conference paper at ICLR 2024
aligned with Bayesian optimization, such the ability to learn non-stationary functions without explicit modeling interventions and gracefully handle nigh-dimensional input and output spaces.
The possible first attempt to use a Bayesian neural network surrogate for Bayesian optimization (Springenberg et al., 2016) came before most of these advances in BNN research, and used form of stochastic gradient Hamiltonian Monte Carlo (SGHMC) (Chen et al., 2014) for inference. Like Snoek et al. (2015), the focus was largely scalability advantages over Gaussian processes; however, the reported performance gains were marginal, and puzzling in that they were largest for a small number of objective function queries (where the neural net would not be able learn rich representation). Kim et al. (2021) used the same method for BNNS with Bayesian optimization, also with SGHMC, targeted at scientific problems with known structures and high dimensionality. In these applications, BNNs leverage auxiliary information, domain knowledge, and intermediate data, which would not typically be available in many Bayesian optimization problems. Foldager al. (2023) also studied BNN surrogates through mean-field BNNs and deep ensembles, and Müller et al. (2023) used neural network surrogates which approximate the posterior through in-context learning. However, the innate differences between approximate inference methods for BNNS have not been explored.
Our paper provides several key contributions in the context of this prior work, where standard GP surrogates are nearly always used with Bayesian optimization. While finite-width BNN surrogates have been attempted, they are often applied in specialized settings without an effort to understand their properties. Little is known about whether BNNs could generally be used as an alternative to GPs for Bayesian optimization, especially in light of more recent general advances in BNN research. This is the first paper to provide a comprehensive study of BNN surrogates, considering a range of model types, experimental settings, and types of approximate inference. We test the utility of BNNs in a variety of contexts, exploring their behavior as we change the dimensionality of the problem and the number of objectives, investigating their performance non-stationary functions, and also incorporating problems with a mix of discrete and continuous input parameters. Moreover, we are the first study infinite BNN models in Bayesian optimization, and to consider the role of stochasticity and representation learning in neural network based Bayesian optimization surrogates. Finally, rather than champion specific approach, we provide an objective assessment, also highlighting the benefits of GP surrogates for general Bayesian optimization problems.
3 SURROGATE MODELS
We consider a wide variety of surrogate models, separately understanding the role of stochasticity, representation learning, and strong priors in Bayesian optimization surrogates. We provide additional information about these surrogates and background about Bayesian optimization in Appendix A.
Gaussian Processes. Throughout our experiments, when we refer to Gaussian processes, we always mean standard Gaussian processes, with the Matérn-5/2 kernel that is typically used in Bayesian optimization (Snoek al., 2012). These Gaussian processes have the advantage of simple exact inference procedures, strong priors, and few hyperparameters, such as length-scale, which controls rate of variability. On the other hand, these models are stationary, meaning the covariance function is translation invariant and models the objective as having similar properties (such as rate of variation) at different points in input space. They also provide a similarity metric for data points based on simple Euclidean distance of inputs, which is often not suitable for higher dimensional input spaces.
Fully Stochastic Finite-Widtl Bayesian Neural Networks. These models treat all of the param- eters of the neural network as random variables, which necessitates approximate inference. We consider three different mechanisms of approximate inference: (1) Hamiltonian Monte Carlo (HMC), an MCMC procedure which is the computationally expensive gold standard (lzmailov al., 2021; Neal, 1996; 2010); (2) Stochastic Gradient Hamiltonian Monte Carlo (SGIMC),, scalable MCMC approach that works with mini-batches (Chen et al., 2014); (3) Deep Ensembles, considered a prac- tically effective heuristic that ensembles neural network retrained multiple times, and has been shown to approximate fully Bayesian inference (Izmailov et al., 2021; Lakshminarayanan et al., 2017). These approaches are fully stochastic, and can do representation learning, meaning that they can learn appropriate distance metrics for the data as well as particular types of non-stationarity.
Deep Kernel Learning. Deep kernel learning (DKL) (Wilson et al., 2016) is a hybrid Bayesian deep learning model, which layers GP on top of neural network feature extractor. This approach can do non- non-Euclided Euclidean representation learning. handle non-stationarity and also uses exact inference. However, it is only stochastic about the last layer.
3

---

Published as a conference paper at ICLR 2024
True Objective
Mean
2o
Function Draw
Likelihood Variance 0.1
Prior Variance 0.1
Hidden Layers
Width 64
-3
-6
0.2
0.5
0.8
0.2
0.5
0.8
0.2
0.5
0.8
0.2
0.5
0.8
Likelihood Variance
1.0
Prior Variance
1.0
Hidden Layers
Width 128
-3
Likelihood Variance 10.0
Prior Variance
Hidden Layers
Width
-3
-6
0.2
0.5
0.8
0.2
0.5
0.8
0.2
0.5
0.8
0.2
0.5
0.8
(a) Likelihood Variance
(b) Prior Variance
(c) Network Depth
(d) Network Width
Figure The design of the BNN has significant impact on the uncertainty estimates. We visualize the uncertainty estimates and function draws produced by full-batch HMC on simple toy objective function with four function queries (denoted black). For the visualizations above, we fix all other design choices with the following base parameters: likelihood variance 1, prior variance 1, number of hidden layers = 3, and width = 128. We see that varying the different aspects model leads to significantly different posterior predictive distributions.
Linearized Laplace Approximation. The linearized-Laplace approximation (LLA) is a determin- istic approximate inference method that uses the Laplace approximation (MacKay, 1992; Immer al., 2021) to produce a linear model from neural network, and has recently been considered for Bayesian optimization in concurrent work (Kristiadi et al., 2023).
Infinite-Widee Width Bayesian Neural Networks. infinite-width neural networks (I-BNNs) refer the behavior neural networks as the number of nodes per hidden layer increases to infinity. Neal (1996) famously showed with central limit theorem argument that BNN with single infinite-widh width hidden layer converges to a GP with a neural network covariance function, and this result has been extended to deep neural networks by Lee al. (2017). I-BNNS fully stochastic and very different from standard GPs, as they can handle non-stationarity and provide a non-Euclidean notion of distance inspired by neural network. However, cannot do representation learning and instead has fixed covariance function that provides relatively strong prior.
3.1
ROLE ARCHITECTURE
We conduct sensitivity study into the role of key design choices for BNN surrogates. We highlight results for HMC, as is the gold standard for approximate inference BNNs (Izmailov et al., 2021).
Gaussian processes involve relatively few design choices-essentialll essentially only the covariance function, which is often set to the RBF or Matérn kernel, and we are also able to have an intuitive understanding of what the induced distributions over functions look. In contrast, with BNNs, we must consider the architecture, the prior over parameters, and the approximate inference procedure. It is also less clear how different modeling choices BNNS affect the inferred posterior predictive distributions. To illustrate these differences for varying network, prior, and variance parameters, we plot the inferred posterior predictive distributions over functions for different network widths and depths, the activation functions, and likelihood and variance parameters in Figure 1, and evaluate the performance under different model choices for three synthetic data problems Figure 2. We focus on fully-connecto multi-layer perceptrons for this study: while certain architectures have powerful inductive biases
4

---

Published as a conference paper at ICLR 2024
Hartmann
(d-10, b-1)
DTLZI
Ai Aeyley (d-10, gm1)
DTL21 (d+5, om2)
159250
2.50
2.75
17
158750
159500
2.25
-18
158800
-19.
159250
-19
159000
2.00
U
158000
10.0 Lioethood Verience
0.1
1.5
0.3
1.0
13.0
8.1
1.0
10.0
Ukoihood Vartence
Litehood Vadiace
Layers
Hiddon Layers
Likelihood Variance
Network Depth
Hartmann idef,
Ackley (dH10,
DTLZ1 Ides, om2)
-175
(dm10, om))
DTLZ1 (de3, om2)
2.75
1595.00
-180
Reward
Deward
150000
18.5
2.25
158500
-19.0
159000
2.00
10
-19.
158000
1.75
-20
-2.0
0.1
1.5
10.0
1.0
10.0
0.1
3.5
15.0
64
120
255
64
125 256
Variance
Variance
Networrk Widch
Width
(c) Prior Variance
(d) Network Width
Figure 2: There is no single architecture for HMC that performs the best across all problems. We compare the impact of the design on the Bayesian optimization performance for different benchmark problems. For each set of experiments, we fix all other aspects of the design and plot the values of the maximum reward found using HMC after 100 function evaluations over 10 trials.
for vision and language tasks, generic regression tasks such as Bayesian optimization tend to be well-suited for fully-connected multi-layer perceptrons, which have relatively mild inductive biases and make loose assumptions about the structure of the function.
Model Hyperparameters. We consider isotropic priors over the neural network parameters with zero mean and variance parameters 0.1, and 10. Similarly, consider Gaussian likelihood functions with variance parameters 0.1, 1, and 10. The corresponding posterior predictive distributions for full-batch HMC shown in Figure la. As would be expected, increase in the likelihood variance results in a poor fit of the data and virtually no posterior collapse. In contrast, increasing the prior variance results in higher predictive variance between data points with good fit to the data points, whereas prior variance that is too small leads to over-regularization and uncertainty collapse. As shown in Figure 2a and Figure 2c, lower likelihood variance parameters and larger prior variance parameters tended to perform best across three synthetic data experiments.
Network Width and Depth. To better understand the effects of the network size on inference, we explore the performance when varying the number of hidden layers and the number of parameters per layer, each corresponding to an increase in model complexity. In Figure 1c, we see that there is significant increase in uncertainty we increase the number of hidden layers. Figure 1d also shows an increase in uncertainty as we increase the width of the network, where a smaller width leads to function draws that are much flatter than function draws from a larger width. However, the best size to choose seems to be problem-dependent, shown in Figure 2b and Figure 2d.
Activation Function. The choice of activation function in neural network determines important characteristics of the function class, such as smoothness periodicity. The impact of the activation function can be seen in Appendix D.11 with function draws from the ReLU BNN appearing more jagged and function draws from the tanh BNN more closely resembling the draws from a GP with a Squared Exponential Matérn 5/2 covariance function.
EMPIRICAL EVALUATION
We provide an extensive empirical evaluation of BNN surrogates for Bayesian optimization. We first assess how BNNs compare to GPs in relatively simple and well-understood settings through commonly used synthetic objective functions, and we perform an empirical comparison between GPs and different types of BNNs (HMC, SGHMC, LLA, ENSEMBLE, I-BNN, and DKL). To further ascertain whether BNNS may be a suitable alternative to GPS in real-world Bayesian optimization problems, study six real-world datasets used in prior work Bayesian optimization with GP
5

---

Published as a conference paper at ICLR 2024
GP
DKL
H8NN
HMC
SGHMC
LUA
Ensemble
Branin (d=2, o=1)
Hartmann (d=6, o=1)
Ackley (d=10, o=1)
-16
Reward
Reward
Reward
-20
60
90
110
170
110
170
Function Evaluations
Function Evaluations
Function Evaluations
BraninCurrin (d=2, o=2)
DTLZ1 (d=5, o=2)
DTLZ5 (d=20, o=3)
159000
970
Reward
80
158000
Reward
157000
920
10
30
156000
Function Evaluations
Function Evaluations
Function Evaluations
Figure 3: BNNS often comparable to GPs on standard synthetic benchmarks. However, the type of BNN used has a big impact: HMC typically outperforms other BNN approximation methods, while SGHMC and deep ensembles seem have less reliable performance and are often unable to effectively find the maximum. I.I.A also has poor performance across the single-objective problems. For each benchmark function, we include for the number of input dimensions, and o for the number of objectives. We plot the mean and one standard error of the mean over 10 trials.
surrogates (Dreifuerst et al., 2021; Eriksson et al., 2019; Maddox et al., 2021; Oh et al., 2019; Wang et al., 2020). We also provide evidence that the performance of BNNS could be further improved with careful selection of network hyperparameters. We conclude evaluation with case study of Bayesian optimization tasks where simple Gaussian process models may fail but BNN models would be expected to prevail. To this end, we design set of experiments assess the performance of GPs and BNNS as function of the input dimensionality and in settings where the objective function is non-stationary.
4.1 SYNTHETIC BENCHMARKS
We evaluate BNN and GP surrogates on a variety of synthetic benchmarks, and we choose problems with a wide span of input dimensions to understand how the performance differs as we increase the dimensionality of the data. We also select problems that vary in the number of objectives to compare the performance of the different surrogate models. Detailed problem descriptions can be found in Appendix I. and we include the experiment setup in Appendix C. We use Monte-Carlo based Expected Improvement (Balandat et al., 2020) as our acquisition function for all problems.
As shown in Figure 3, we find BNN surrogate models to show promising results; however, the specific performance of different BNNS varies considerably per problem. DKL matches GPs in Branin and BraninCurrin, but seems to perform poorly on highly non-stationary problems such as Ackley. I-BNNS also seem to slightly underperform compared to GPs on these synthetic problems, many of which have a small number of input dimensions. In contrast, we find finite-width BNNS using full HMC to be comparable GPs, performing similarly in many of the experiments, slightly underperforming in Hartmann and DTI.75, and outperforming GPs in the 10-dimensional Ackley experiment. However, this behavior is not generalizable all approximate inference methods: the performance of SGHMC and LLA vary significantly per problem, matching the performance of HMC and GPs in some experiments while failing approach the maximum value others. Deep ensembles also consistently underperform the other surrogate models, plateauing at noticeably lower objective values on problems like BraninCurrin and DTLZ1. This result is surprising, since ensembles are often seen as an effective way to measure uncertainty (Appendix D.3). We provide additional experiments studying how the performance of surrogates changes as we vary their hyperparameters Appendix D, and we find that these hyperparameters generally have minimal effects on the performance.
4.2 ROAL- BENCHMARKS
To provide an evaluation of BNN surrogates in more realistic optimization problems, we consider diverse selection of real-world applications which span variety of domains, such solving

---

Published as a conference paper at ICLR 2024
GP
DKL
H8NN
HMC
SGHMC
LUA
Ensemble
PDE Optimization (d=4, o=1)
Interferometer (d=4, o=1)
Lunar Lander (d=12, o=1)
200
-0.05
Reward
Reward
Reward
0.15
100
0.2
-0.25
10
20
30
150
300
450
Function Evaluations
Function Evaluations
Function Evaluations
Pest Control (d=25, o=1)
Cell Coverage (d=30, o=2)
Oil Spill Sorbent (d=7, o=3) 1e6
-12
0.045
1.2
Reward
Reward
0.03
0.035
Reward
1.0
-16
0.025
0.8
0.6
40
7D
100
120
150
180
Function Evaluations
Function Evaluations
Function Evaluations
Figure 4: Real world benchmarks show mixed results. BNNS outperform GPs some problems and underperform on others, and there does not seem to be a noticeable preference for any particular surrogate increase the number of input dimensions. Additionally, there does not appear to be clear separation between the top row of experiments, which optimize over continuous parameters, and the bottom row of experiments, which also include some discrete inputs. For each benchmark, we include d for the number of input dimensions, and for the number of objectives. We plot the mean and one standard error of the mean over 10 trials.
differential equations and monitoring cellular network coverage (Dreifuerst et al., 2021; Eriksson et al., 2019; Maddox al., 2021; Oh al., 2019; Wang et al., 2020). Many of these problems, such as the development of materials to clean oil spills, have consequential applications; however, these objectives often multi-moda and difficult to optimize globally. Additionally, unlike the synthetic benchmarks, many real-world applications consist of input data with ordinal or categorical values, which may be difficult for GPS to handle. Several of the problems also require multiple objectives to be optimized. Detailed problem descriptions are provided in Appendix B.2.
We share the results of our experiments in Figure 4, and details about the experiment setup can be found in Appendix C. The results are mixed: BNNS are able significantly outperform GPS in the Pest Control dataset, while GPs find the maximum reward in the Cell Coverage and Lunar Lander experiments. The Pest Control, Cell Coverage, and Oil Spill Sorbent experiments all include discrete input parameters, and there seems to be a slight trend of GP and I-BNNS performing well, and SGHMC and LLA performing more poorly. Similar to the findings from the synthetic benchmarks, we see that the different approximate inference methods for fnite-width BNNs lead to significantly different Bayesian optimization performance, with HMC generally finding higher rewards compared SGHMC, LLA, and deep ensembles. Additionally, it appears that GPs perform well in the two multi-objective problems, although that may not be generalizable to additional multi-objective problems and may be more related to the curvature of the specific problem space.
4.3 LIMITATIONS OF GP SURROGATE MODELS
Although popular, GPS suffer from well-known limitations that directly impact their usefulness as surrogate models. To contrast BNN and surrogates, we explore two failure modes of GPS and demonstrate that BNN surrogate models can overcome these issues.
To use GPs, we must specify a kernel function class that governs the covariance structure over data points. We typically constrain models to have kernels of the form k(x, x') [[[[x - x']l) because easier to describe the functional form and learn the hyperparameters of the kernel. However, because the covariance between two values only depends on their distance and not on the values themselves, this setup assumes the function stationary and has similar mean and smoothness throughout the input space. Unfortunately, this assumption does not hold true in many woaldworld settings. For example, in the common Bayesian optimization application of choosing hyperparameters of neural network, the true loss function landscape may have vastly different behavior in one part of the input space compared to another. BNN surrogates, in contrast to GP surrogates, able to model non-stationary functions without similar constraints.
Non-Stationary
Objective Functions.
7

---

Published as a conference paper at ICLR 2024
GP
DKL
H8NN
HMC
SGHMC
LUA
Ensemble
Polynomial (d=200)
NN Draw (d=400)
Distillation (d=6320)
60
Roward
Reward
Roward
50
-80
100
300
400 Function Evaluations
100
300
Function Evaluations
Function Evaluations
Figure 5: 1-BNNS outperform other surrogates in many high-dimensional settings. We show the results of maximizing a polynomial function (left), maximizing a fixed function draw from a neural network (center), and optimizing the parameters of neural network in the context of knowledge distillation (right). All of these objectives are high-dimensional and non-stationary, and we find that BNNS consistently find higher rewards than GPS across all problems. We plot the mean and one standard error of the mean over 10 trials, and d corresponds to the number of input dimensions.
In Appendix D.9, we show the performance of BNN and GP surrogate models for a non-stationary objective function. Because the GP assumes that the behavior of the function is the same throughout the input domain, it cannot accurately model the input-dependent variation and underfits around the true optimum. In contrast, BNN surrogates can learn the stationaaitty of the function.
High-Dimensional Input Spaces. Due to the curse of dimensionality, GPs do not scale well to high- dimensional input spaces without careful human intervention. Common covariance functions may fail to faithfully represent bigh-dimensional input data, making the design of custom-tailored kernel functions necessary. In contrast, neural networks well-suited for modeling high sigh-iimensional input data (Krizhevsky et al., 2012).
To measure the effect of dimensionality on the performance of GPs and BNNS, we use synthetic test functions provided by nigh-dimensional polynomial functions and function draws from neural networks. We also construct a realistic high-dimensional problem by using Bayesian optimization to set the parameters of neural network in the context of knowledge distillation. Knowledge distillation refers to the act of "distilling" information from a larger teacher model to a smaller student model by matching model outputs (Hinton et al., 2015), and it is known to be diffieult optimization problem (Stanton et al., 2021). For full descriptions, see Appendix B.
We share the results of our findings in Figure 5 and Appendix D.4. We see I-BNNS clearly stand out in these nigh-ddmeesioonll settings. The I-BNN has several appealing features in this setting: (1) it provides a non- -un-Euclidean and non-stationary similarity metric, which can be particularly valuable in high-dimensions; (2) it does not have any hyperparameters for learning, and thus is not "data hungry"- which is especially important in high dimensional problems with small data sizes, since these settings provide relatively little information for representation learning. Additionally, find that other BNN surrogate models also outperform GPs across the high-dimensional problems, providing compelling motivation for BNNS as surrogate models for Bayesian optimization.
4.4 MODEL RANKINGS
Problems
High Dimensional
To further interpret our findings, we visualize the model performance in Figure 6. We deter- mine the relative performance of each model by ry. For each trial, let rh denote the highest max- imum reward found by any model and r denote the lowest. The score of model with maximum reward is (r: Tr)/(rk - TI).
Belative Performance
11.00
1.00
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
I-NNN
HMC Ensemble
LLA
DKL
SGHMC
L-NN
HMC
Ensemble
LLA
SGMMC
DKL
GP
We plot the distribution of scores in Figure 6. Across all experiments, I-BNNS and GPs have competitive performance, while SGIMC and DKL perform more poorly. However, in high- dimensional settings, 1-BNNS outperform all other surrogate models. GPS, in contrast, per- form poorly in this setting, consistently finding lower rewards than BNN surrogates.
Figure 6: We rank models by their relative per- formance. Across all problems, GPs and I-BNNS have strong relative performance. However, in high-dimensional problems, I-BNNS consistently outperform other surrogate models while GPS tend to find much lower rewards.
8

---

Published as a conference paper at ICLR 2024
4.5 ADDITIONAL PRACTICAL CONSIDERATIONS
Network Architecture. To further understand the impact of network architecture on the performance of BNNS, we conduct an extensive neural architecture search over selection of the benchmark problems. While thorough search is often impractical for Bayesian optimization problems, we use this experiment to investigate the flexibility of BNNS. We find that the performance of BNNS can significantly increase for problems such as Pest Control, demonstrating the usefulness of BNN for Bayesian optimization when the architecture is well-suited for given problem. We provide additional details and showcase the performance of BNNS with different architectures in Appendix D.I.
Quality of Mean and Uncertainty Estimates. To better understand the quality of the mean and uncertainty estimates of different surrogate models, we conducted ablation studies for which we created hybrid models that combine the mean estimate of one surrogate with the uncertainty estimate of another. By comparing the performance of a given surrogate model to hybrid models that have the same mean as the non-hybrid model but the uncertainty estimates of different surrogate model (or the other way around), we find that IMC and I-BNN surrogates often have better mean estimates compared to GPs while GPs have better uncertainty estimates, which we detail in Appendix D.2.
Large Number of Function Queries. We investigate the effect of performing a larger number of function queries on performance when using BNN and GP surrogate models. We find in this setting that (1) I-BNNs remain competitive; (2) BNNs perform well, leveraging the data for representation learning: (3) the performance of deep ensembles is greatly improved, consistent with the explanation that their poor performance many tasks is due to limited data. We share details in Appendix D.10.
Runtime. In real-world Bayesian optimization problems, querying the objective function is typically significantly more cineuconsuiing than re-)ttrinnng surrogate model after new data has been obtained, making the quality of the surrogate model paramount and the time needed for (re- (trannnag the surrogate model of secondary interest. Nevertheless, given the varying training requirements of BNNS and GPs, we provide wall-clock times of all surrogate models across our experiments in Appendix D.11. Notably, I-BNNs are particularly competitive both in performance and runtime.
4.6 REVISITING STANDARD ASSUMPTIONS
While Gaussian processes are typically used as the default surrogate model, there are many design choices, such the choice of kernel and the method of hyperparameter selection, which play crucial role. It is commonly prescribed to use the Matérn kernel and perform hyperparameter marginalization rather than optimization (e.g. Eriksson et al., 2019; Snoek et al., 2012). In Figure A.13 and Figure 121 we compare the performance of different specifications of GPS across our diverse benchmarks. In contrast to conventional wisdom, we do not find that using the Matérn kernel and hyperparameter marginalization significantly improves the performance of GPS in general; in fact, there are many problems where the RBF kernel or hyperparameter optimization are preferable.
DISCUSSION
While Bayesian optimization research has made significant progress over the last few decades (Garnett, 2023), the surrogate model remains a crucial yet underexplored design choice, with standard GPS being viewed as default surrogates. Given recent advances BNNs and related approaches, it is, therefore, particularly timely to consider neural network surrogates for Bayesian optimization.
Although we found that BNNs are competitive with standard GPs for Bayesian optimization, our findings that DKL is competitive with BNNS, and that infinite-width BNNS show promising performance in general-Bee but especially for higher dimensional settings raise the question of whether fully stochastic treatment of finite BNN surrogates is typically necessary for Bayesian optimization. Since infinite-width BNNs do not involve learning many hyperparameters and do not require approximate inference, they are well- positions to become a de facto standard surrogate for Bayesian optimization.
Finally, we found that, on the one hand, no single surrogate model consistently outperforms all other surrogate models across all Bayesian optimization problems considered in our evaluation. This finding supports the use of simple models with strong but generic assumptions -sciiioonsssshhhhh as standard GP models. On the other hand, the standard benchmarking problems were designed with only GP surrogates in mind, and we found that standard GPs underperform BNNS and other neural network- bewsork-based surrogates on challenging high-dimensional problems.

---

Published as a conference paper at ICLR 2024
Acknovledgements. We thank Greg Benton for helpful guidance in the beginning stages of this research, and Sanyam Kapoor for discussions. This work is supported by NSF CAREER IIS 2145492, NSF I-DISRE 193471, NIH R01DA048764-01AL, NSF IIS- 1SS-199 NSF 1922658 NRT-HDR, Meta Core Data Science, Google Al Research, BigHat Biosciences, Capital One, and an Amazon Research Award.
REPRODUCIBILITY
We provide the code needed to reproduce all experiments the supplementary material attached, and we also provide experiment details for all of our results in Appendix C. We also release our code on GitHub: https::////.mm / htttpss://gtttubb.cc///eeeee wwwpi//.cco/ccceniccc http//ycceennllnniemm coannni//nnbbaaa bo.
REFERENCES
Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wil- son, and Eytan Bakshy. Botorch: A framework for efficient monte-carlo bayesian optimization. Advances in neural information processing systems, 33:21524-21533. 2020.
James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning, pages 115-123. PMLR, 2013.
Tianqi Chen, Emily B Fox, Carlos Guestrin, and Michael Jordan. Stochastic gradient hamiltonian monte carlo. In International Conference on Machine Learning, pages 1683-1691, 2014.
Zhongxiang Dai, Yao Shu, Bryan Kian Hsiang Low, and Patrick Jaillet. Sample-then-optimin batch neural thompson sampling, 2022.
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems, 2021.
Ryan M Dreifuerst, Samuel Daulton, Yuchen Qian, Paul Varkey, Maximilian Balandat, Sanjay Kasturia, Anoop Tomar, Ali Yazdan, Vish Ponnampalam, and Robert W Heath. Optimizing coverage and capacity in cellular networks using machine learning. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8138-8142. IEEE, 2021.
David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local bayesian optimization. Advances in neural information processing systems, 32, 2019.
Jonathan Foldager, Mikkel Jordahn, Lars Kai Hansen, and Michael Riis Andersen. On the role of model uncertainties in bayesian optimization, 2023.
Peter Frazier. tutorial on bayesian optimization. Foundations and Trends@ Machine Learning. 11(1):1-9, 2018.
Peter Frazier, Warren B Powell, and Savas Dayanik. knowledge-gradient policy for sequential information collection. SIAM Journal on Control and Optimization, 2008
Jacob Gardner, Chuan Guo, Kilian Weinberger, Roman Garnett, and Roger Grosse Discovering and exploiting additive structure for bayesian optimization. In Artificial Intelligence and Statistics, 2017.
Roman Garnett. Bayesian optimization. Cambridge University Press, 2023.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0253/ 2015.
Matthew D. Hoffman, David M. Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research, 14(1):1303-147, May 2013. ISSN 1532-4435.
10

---

Published as a conference paper at ICLR 2024
Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In International conference on learning and intelligent optimization, pages 507-523. Springer, 2011.
Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of bayesian neural nets via local linearization, 2021.
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are bayesian neural network posteriors really like? In International conference on machine learning. pages 4629 4640. PMLR, 2021.
Kirthevasan Kandasamy, Jeff Schneider, and Barnabás Póczos. High dimensional bayesian op- timisation and bandits via additive models. In International conference on machine learning. 2015.
Mohammad Emtiyaz Khan and Håvard Rue. The bayesian learning rule.
arXiv preprint
arXiv:2107.04562, 2021.
Samuel Kim, Peter Y Lu, Charlotte Loh, Jamie Smith, Jasper Snoek, and Marin Soljacic. Deep learn- ing for bayesian optimization of scientific problems with nigh-dimensiooal structure. Transactions of Machine Learning Research, 2021.
Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, and Vincent Fortuin. Promises and pitfalls of the linearized laplace in bayesian optimization, 2023.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep con- volutional neural networks. In Advances in Neural Information Processing Systems 25:, pages 1106-1114, 2012.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.0016 2017.
Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter. Smac3: A versatile bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research, 23(54):1-9, 2022. URL http: / http//mmlroorroor http:///.....rppapp///////////// http:/ppppsss//// /v33 21 2118288.108..mmm1
Michal Lisicki, Arash Afkanpour, and Graham W. Taylor. Empirical analysis of representation learning and exploration in neural kernel bandits, 2022.
Sanae Lotfi, Pavel Izmailov, Gregory Benton, Micah Goldblum, and Andrew Gordon Wilson.
Bayesian model selection, the marginal likelihood, and generalization, 2023.
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computa- tion, 4(3):488472, 1992.
Wesley J Maddox, Maximilian Balandat, Andrew G Wilson, and Eytan Bakshy. Bayesian optimization with high- nigh-dimensiioal outputs. Advances in Neural Information Processing Systems, 34:19274- 19287, 2021.
Henry Moss, David Leslie, Daniel Beck, Javier Gonzalez, and Paul Rayson. Boss: Bayesian optimization over string spaces. Advances in Neural Information Processing Systems, 2020.
Samuel Müller, Matthias Feurer, Noah Hollmann, and Frank Hutter. Pfns4bo: In-context learning for bayesian optimization, 2023.
Radford M Neal. Bayesian Learning for Neural Networks. 1996.
Radford M. Neal. MCMC using Hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, 54:113-163 13-162, 2010.
11

---

Published as a conference paper at ICLR 2024
Radford M Neal and Radford M Neal. Priors for infinite networks. Bayesian learning for neural networks, pages 29-53, 1996.
Changyong Oh, Jakub Tomezak, Efstratios Gavves, and Max Welling, Combinatorial bayesian optimization using the graph cartesian product. Advances Neural Information Processing Systems, 32, 2019.
A O'Hagan. On curve fitting and optimal design for regression. J. Royal Stat. Soc. B, 40:1-32, 1978.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sehastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems 32. 2019.
Carl Edward Rasmussen and Christopher KI Williams. Gaussian processes for machine learning. volume 2. MIT Press Cambridge, MA, 2006.
Tim G. J. Rudner, Freddie Bickford Smith, Qixuan Feng, Yee Whye Teh, and Yarin Gal. Continual Learning via Sequential Function-Space Variational Inference. In Proceedings of the 38th Inter- national Conference Machine Learning, Proceedings of Machine Learning Research. PMLR, 2022.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In International conference on machine learning, pages 2171- PMLR, 2015.
Dmitry Sorokin, Alexander Ulanov, Ekaterina Sazhina, and Alexander Lvovsky. Interferobot: aligning an optical interferometer by reinforcement learning agent. Advances in Neural Information Processing Systems, 33:13238-13244 2020.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. Advances in neural information processing systems, 29, 2016.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 34: 6906-6919, 2021.
Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating Bayesian optimization for biological sequence design with denoising autoencoders. In International Conference on Machine Learning. 2022.
Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimization. Advances in neural information processing systems, 26, 2013.
Dustin Tran, Jeremiah Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet. Huiyi Hu, Neil Band, Tim G. J. Rudner, Karan Singhal, Zachary Nado, Joost van Amersfoort andAndreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D. Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snoek, and Balaji Lakshminarayanan. Plex: Towards Reliability Using Pretrained Large Model Extensions. 2022.
Boqian Wang, Jiacheng Cai, Chuangui Liu, Jian Yang, and Xianting Ding. Harnessing novel machine- Saahiinn-leaning-asseee assisted evolutionary algorithm to co-optimize three characteristics an electro- spun oil sorbent. ACS Applied Materials & Interfaces, 12(38);42442-4284) 2020.
Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient bayesian optimization. In International Conference on Machine Learning, pages 3627-3635. PMLR, 2017.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-1I), pages 681-688, 2011.
12

---

Published as a conference paper at ICLR 2024
Christopher Williams. Computing with infinite networks. Advances in neural information processing systems, 9, 1996.
Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697-4708,, 2020.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning.
In Artificial intelligence and statistics, 2016.
Jian Wu, Matthias Poloczek, Andrew G Wilson, and Peter Frazier. Bayesian optimization with gradients. Advances in neural information processing systems, 30, 2017.
Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural contextual bandits with ucb-based exploration. In International Conference Machine Learning, pages 1 11492-11502. PMLR, 2020.
13

---

Published as a conference paper at ICLR 2024
Appendix
A Study of Bayesian Neural Network Surrogates for Bayesian Optimization
TABLE OF CONTENTS
Background
15
A.I Bayesian Optimization
15
A.2 Gaussian Processes
15
A.3
Bayesian Neural Networks
15
Problem Descriptions
18
B.1 Synthetic Datasets..
18
B.2 Real-World Datasets
18
B.3 High-Dimensional Problems
18
C Experiment Details
20
C.1 General Setup
20
C.2 Synthetic Benchmarks
20
C.3 Real-World Benchmarks
20
C,4 High-Dimensional Experiments
21
C.5 Neural Architecture Search
21
D Further Empirical Results
22
D.1 BNN Architecture
22
D.2 Ablation Studies
24
D.3 Deep Ensembles..
26
D.4 Infinite-Width BNNs
29
D.5 Hamiltonian Monte Carlo
32
D.6 Gaussian Processes
33
D.7 Deep Kernel Learning...
34
D.8 Acquisition Batch Size
35
D.9 Limitations of Gaussian Process Surrogate Models
36
D.10 Large Number of Function Queries
38
D.11 Runtime
39
14

---

Published as a conference paper at ICLR 2024
APPENDIX A BACKGROUND
A.1 BAYESIAN OPTIMIZATION
Bayesian optimization is global optimization method commonly used for black-box functions which are costly to evaluate. Specifically, we formulate the optimization as a maximization problem where we want to solve x arg max, fxx. where x represents all possible inputs to objective function f. using a limited number of function queries. In Bayesian optimization, we iteratively select new function evaluations using the following procedure: (1) use surrogate model to find P(f(D), where D [x. Vs Vi= and Yi is a noisy observation of (x:); (2) select the next X to query by maximizing an acquisition function, which is an inexpensive computation that typically uses the mean and variance of the posterior p p(J(D) to compute how useful a function evaluation would be; and (3) query the function at value X;+1 to observe yt+1 ~ (x d.n) and add the result to the dataset D := D U (xL+1, \t+1).
Historically, Bayesian optimization has seen wide success in range of fields including drug design, engineering challenges, and hyperparameter optimization. A more recent and growing area of interest is in higher dimensional settings, where the objective, J (xx is multi-dimensional consisting of several tasks we wish to optimize jointly. In multi-objective Bayesian optimization, we want to find the input x that maximizes k related objective functions F = (x). fx ). There is typically no single solution that maximizes all K objectives simultaneously. Therefore, performance is typically compared using Pareto dominance, where one solution equally well or better on all objectives. Pareto dominance can be measured using hypervolume, which indicates how much of a bounded region is dominated by solution.
another if it performs
Core to the Bayesian optimization procedure is the of acquisition function to select the next candidate points given of observations Fi; f (xx,) ) and surrogate model trained to fit these observations. Acquisition functions are functions of the predictive distribution of the surrogate model, and seek candidate points according to criteria such as the probability of improvement or the expected improvement over the currently found minimizer of the objective.
The chosen surrogate model will also have a signficant impact on the performance of Bayesian optimization. Since the true function f may take variety of forms, the surrogate model should be flexible enough to accurately represent it. Additionally, the uncertainty estimates that the surrogate model provides should be well-calibrated to ensure that the input that maximizes f can be found in a limited number of iterations.
A.2 GAUSSIAN PROCESSES
Gaussian processes (GPS) (Rasmussen and Williams, 2006) are distributions over functions entirely specified by mean function µ(x) and a covariance function k(x,x') For regression tasks with GPS, we assume y(x) = fx) + e with fx) ~ (x(((x), k(x, x')) and e ~ (O0,, o²), where o² is the likelihood variance. The functions u(x) and k(x, x') are the mean and kernel functions of the GP that govern the functional properties and generalization performance of the model. In practice, we tune the hyperparameters of u(x) and k(x, x') on the training data to optimize the marginal likelihood of the GP, which maximizes the probability that the GP model will have generated the data.
By the definition of a GP, for any finite collection of inputs x = | Xles]], / fxx,, and thus v(x) are jointly normal. Therefore, can apply the rules of conditioning partitioned multivariate normals to form a posterior distribution (f (Xwst) y ((Xiran)), which will also be Gaussian.
In the context of Bayesian optimization, this simple conditioning procedure means that given some set of points at which we have already queried the objective function, we can quickly generate posterior over potential candidate points and, with the help of an acquisition function, select the next points to query the objective.
A.3 BAYESIAN NEURAL NETWORKS
Bayesian neural networks (BNNS) are neural networks with stochastic parameters for which a posterior distribution is inferred using Bayesian inference.
More specifically, for regression tasks, we specify a Gaussian observation model, p(y(x) x, 0) (h((x )), where yx) represents a noisy observed value and h(x ()) represents the output of neural network with parameter realization 0 for input x. For BNNS, prior distribution is placed over the stochastic parameters O of the neural network, and the corresponding posterior distribution is given by p(0) D)- = p(D (0)(())/pl)) ())) This posterior distribution can then be used in combination
15

---

Published as a conference paper at ICLR 2024
with the acquisition function for Bayesian optimization to select the next set of candidate points to query.
There are many different choices to consider when using Bayesian neural networks, such the inference method used to compute the posterior distribution, the architecture of the neural network, and the selection of which parameters are stochastic. this work, we study the performance of five different types of BNNs with varying inference methods and stochasticity.
A.3.1
FINITE- WIDTH NEURAL NETWORKS
For fully- cullysttochastic Bayesian neural networks, every parameter of the neural network is stochastic and has a prior placed over it. Exact inference over these stochastic network parameters for fully- stola-stochastic finite-width BNNS is analytically intractable, since neural networks are non-linear their parameters. To enable Bayesian inference in this setting, approximate inference methods can be used to find approximations to the exact posterior p(0 (D). In this work, we focus on four types of approximate inference: Hamiltonian Monte Carlo (HMC; Neal (2010)), stochastic- stochastic-graient HMC (Chen al., 2014), linearized Laplace approximations (Immer et al., 2021), and deep ensembles of deterministic neural networks (Lakshminarayanan al., 2017).
Hamiltonian Monte Carlo. Hamiltonian Monte Carlo is a Markov Chain Monte Carlo method that produces asymptotically exact samples from the posterior distribution (Neal, 2010) and is commonly referred to as the "gold standard" for inference in BNNs. At test time, HMC approximates the integral p(y((ees) DD) ~ M Eemp(p((x (xy(e)t)) 0.) using samples O, drawn from the posterior over parameters.
Full-batch HMC provides the most accurate approximation of the posterior distribution but is com- putationally expensive and in practice limited to models with only few hundred thousand parame- ters (Izmailov et al., 2021). Full-batch IMC allows to study the performance of BNNs in Bayesian optimization without many of the confounding factors of inaccurate approximations of the predictive distribution.
Stochastic Gradient Hamiltonian Monte Carlo. Stochastic gradient methods (Welling and Teh, 2011; Hoffman et al., 2013) are commonly used inexpensive approach sampling from the posterior distribution. Unlike full-batch IMC, which computes the gradients over the full dataset, stochastic gradient HMC instead samples mini-batch compute noisy estimate of the gradient (Chen et al., 2014). While these methods may seem appealing when working with large models or datasets, they can result in inaccurate approximations and posterior predictive distribution with unfaithful uncertainty representations.
Deep Ensembles. Deep ensembles are ensembles of several deterministic neural networks, each trained using maximum posteriori estimation using different random seed and initialization (and sometimes using different subsets of the training data). The ensemble components can viewed as forming an efficient approximation to the posterior predictive distribution, by choosing parameters that represent typical points in the posterior and have high functional variability (Wilson and Izmailov, 2020). Deep ensembles are easy to implement in practice and have been shown to provide highly accurate predictions and good predictive uncertainty estimate (Lakshminarayanan et al., 2017; Ovadia al., 2019).
A.3.2 FUOLY-STCCHASTIC INFINITE- WIDTH NEURAL NETWORKS
Infinitely-wide neural networks (Neal and Neal, 1996) refer to the behavior of neural networks when the number of nodes in each internal layer increases to infinity. Each one of these nodes continues is stochastic and has a specific prior placed over its parameters.
Infinite-width Neural Network. It is possible to specify GP that has an exact equivalence to an infinitely-wide fully-connected Bayesian neural network (I-BNN) (Lee et al., 2017). For a single-layer network, the Central Limit Theorem can be used to show that in the limit of infinite width, each output of the network will be Gaussian distributed, and the exact distribution can be computed (Neal and Neal, 1996; Williams, 1996). This process layers (Lee et al., 2017). I-BNNS can be used in the same way as GPs to calculate the posterior distribution for infinitelywwide neural networks.
then be applied recursively for additional hidden
16

---

Published as a conference paper at ICLR 2024
A.3.3 LINEARIZED FINITE- WIDTH NEURAL NETWORKS
Linearized Laplace Approximation. The linearized Laplace approximation (Immer al., 2021) (LLA) is deterministic approximate inference method which uses the Laplace approximation (MacKay, 1992; Immer et al., 2021) to produce linear model from neural network, LLAS can be represented as Gaussian processes with mean functions provided by the MAP predictive function, and covariance functions provided by the finite, empirical neural tangent kernel at the MAP estimate.
A.3.4 PARTIALIY-STOCHAKE
FINITE-W NEURAL. NETWORKS
For partially-stochatiic neural networks, we learn point estimates for a subset we learn the full posterior distribution for the remaining parameters in the neural network.
parameters, and
Deep Kernel Learning. Deep Kernel Learning (DKL.) is hybrid method that combines the flexibility of GPS with the expressivity of neural networks (Wilson et al., 2016). In DKL, neural network (-) parameterized by weights w is used to transform inputs into intermediate values, kernels can then be applied. Specifically, given inputs x and base kernel
where additional (BASE(X, x'), (x, x') ( (x), Iw (x)).. Unlike GPs with standard kernels which depend only on the distance between inputs and therefore assume that the mean and smoothness of the function are consistent throughout, DKL. does not assume stationarity and is able to represent how the properties of the function change due its neural network feature extractor.
17

---

Published as a conference paper at ICLR 2024
APPENDIX B PROBLEM DESCRIPTIONS
B.I SYNTHETIC DATASETS
In the single-objective case, Branin is function with two-dimensional inputs with three global minima, Hartmann is a six-dimensional function with six local minima, and Ackley is a multi- dimensional function with many local minima which is commonly used to test optimization algorithms. We convert all problems to maximization problems, and the goal of Bayesian optimization in single- objective settings to find the maximum value of the objective function.
For multi-objective Bayesian optimization benchmarks, BraninCurrin is two-dimensional input and two-objective problem composed of the Branin and Currin functions, and DTL.Z1 and DTL.7 are multi-dimensiona and multi-objective test functions which are used to measure an optimization algorithm's 's ability to converge to the Pareto-frontier. Here, the goal is to find an input corresponding to the dinlii-dmeesional objective value with the maximum hypervolume from a problem-specific reference point.
B.2 WORLD DATASETS
Interferometer In this problem, the goal to tune an optical interferometer through the alignment of two mirrors. We use the simulator in Sorokin et al. (2020) to replicate the Bayesian optimization problem in Maddox et al. (2021). Each mirror has a continuous x and y coordinate, which should be optimized so that the two mirrors reflect light with minimal amounts of interference.
Lunar Lander Following Eriksson et al. (2019), we aim to learn the parameters of a controller for lunar lander as implemented in OpenAl gym. The controller has 12 continuous input dimensions, corresponding to events such as "change the angle of the rover if it is tilted more than degrees." The objective is to maximize the average final reward over 50 randomly generated environments.
Cellular Coverage We want to optimize the cellular network coverage and capacity from 15 cell towers (Dreifuerst et al., 2021). Each tower has a continuous parameter corresponding to transmit power parameter and an ordinal parameter with values corresponding to tilt, for total of 15 continuous and 15 ordinal input parameters. There are two different objectives: maximize the cellular coverage, and minimize the total interference between cell towers.
Oil Spill Sorbent In this problem, we optimize the properties of material to maximize its per- formance as a sorbent for marine oil spills (Wang et al., 2020). We tune 5 ordinal parameters and continuous parameters which control the manufacturing process of electrospun materials, and optimize over three objectives: water contact angle, oil absorption capacity, and mechanical strength.
Pest Control Minimizing the spread of pests while minimizing the prevention costs of treatment an important problem with many parallels (Oh al., 2019). In this experiment, we define the setting as a categorical optimization problem with 25 categorical variables corresponding to stages of intervention, with 5 different values at each stage. We optimize over two objectives: minimizing the spread of pests and minimizing the cost of prevention.
PDE Optimization Here, following Maddox et al. (2021), we optimize 4 continuous parameters corresponding to the diffusivity and reaction rates of a Brusselator with spatial coupling. The objective of the problem is to minimize the variance of the PDE output.
B.3 HIGH- DIMRNDIIENSIIAALL. PROBLEMS
Polynomial this optimization problem, the goal is to find the maximum value of a polynomial function. For input x E Rdd the objective value is calculated using dd/4 II]_ (Xi_; G++)). ) where G (O, 1). We set the boundaries of the input space to be [0, 1] and this problem setup can be used for any number of dimensions d.
Neural Network Draw We want to find the maximum value of a function specified by a draw from a neural network. For the neural network, we use MLP with d inputs connected to hidden layers of 256 nodes each with tanh activation, connected to a final layer of size I (unless otherwise specified). We set each parameter w in the network value drawn from (0, 1). The final objective function is equivalent to the output of the neural network with the specified weights. With this setup, we can vary d alter the input dimensionality of the problem.
Knowledge Distillation The goal of knowledge distillation is to use larger teacher model train smaller student model, typically done by matching the teacher and student predictive distributions. In this problem, we Bayesian optimization determine the optimal parameters of the student neural
18

---

Published as a conference paper at ICLR 2024
network by setting objective function the KL divergence between the teacher and student predictive distributions. Knowledge distillation is known to be a difficult optimization problem, and this is problem also has many more dimensions than typical Bayesian optimization benchmarks. For our experiment, we use the MNIST dataset, and we train a LeNet-5 for our teacher model. For our student model,
small CNN with the following architecture:
1. convolutional layer with 16 convolution kernels of 3x3 (ReLU activation)
2. max pool layer 2x2
3. convolutional layer with 12 convolution kernels of 3x3 (ReLU activation)
4. max pool layer 2x2
5. fully connected layer with 10 outputs
19

---

Published as a conference paper at ICLR 2024
APPENDIX C EXPERIMENT DETAILS
C.I GENERAL SETUP
For all datasets, we normalize input values to be between [0, 1] and standardize output values to have mean 0 and variance 1. We also use Monte-Carlo based Expected Improvement our acquisition function.
GP: For single-objective problems, we use GPS with a Matérn 5/2 kernel, adaptive scale, a length- scale prior of Gamma(3, 6), and output-scale prior of Camma(2.0, 0.15). which combine with the marginal likelihood to form a posterior which we optimize for hyperparameter learning. For multi-objective problems, we use the same GP to independently model each objective.
I-BNN: We use I-BNNs with hidden layers and the ReLU activation function. We set the variance of the weights to 10.0, and the variance of the bias to 1.6. For multi-objective problems, we independently model each objective.
For each of the following surrogate models which include neural networks, we use MLPs with 3 hidden layers and 128 nodes each. During each iteration Bayesian optimization, we use a grid search over prior variance (0.1, 1.0, 10.0) and likelihood variance (0.1, 0.32, 1.0) to find the combination which maximizes C, where L represents the likelihood of the surrogate model on a random 20% of the existing function evaluations when the model is trained the other 80%.
DKL: We set up the base kernel using the same Matérn 5/2 kernel that we use for GPs. For the feature extractor, we use the model parameters as explained above. For multi-objective problems, we independently model each objective.
HMC: We use HMC with an adaptive step size, and we choose the architecture as explained above. We model multi-objective problems by setting the number of nodes in the final layer equal to the number of objectives.
SGHMC: We use SGHMC with minibatch size of 5 and neural network architecture as indicated above. We follow the implementation in Springenberg et al. (2016) and use scale-adaptive SGHMC with heteroskedastic likelihood variance as determined by the output of the neural network. We model multi-objective problems by setting the number of nodes in the final layer equal to the number of objectives.
LLA: We use the model architecture as explained. We model multi-objective problems by setting the number of nodes in the final layer equal to the number of objectives.
ENSEMBLE: We use ensemble of 5 models, each with the architecture explained above. Each model is trained on a random 80% of the function evaluations. We model mujti-objective problems by setting the number of nodes in the final layer equal to the number of objectives.
We run multiple trials for all experiments, where each trial starts with a different set of initial function evaluations drawn using a Sobol sampler.
C.2 SYNTHETIC BENCIMARKS
Branin We ran 10 trials using batch size 5 with 10 initial points.
Hartmann We ran trials using batch size 10 with 10 initial points.
Ackley We ran 10 trials using batch size 10 with initial points.
BraninCurrin We ran 10 trials using batch size 10 with 10 initial points.
DTLZ1 We ran 10 trials using batch size 5 with 10 initial points.
DTLZ5 We ran 10 trials using batch size 1 with 10 initial points.
C.3 REAL-WORLD BENCHMARKS
PDE Optimization We ran 10 trials using batch size I with 5 initial points.
Interferometer We ran 10 trials using batch size 10 with 10 initial points.
Lunar Lander We ran 10 trials using batch size 50 with 50 initial points.
Pest Control We ran 10 trials using batch size 4 with 20 initial points.
Cell Coverage We ran 10 trials using batch size 5 with 10 initial points.
Oil Spill Sorbent We 10 trials using batch size 10 with 10 initial points.
20

---

Published as a conference paper at ICLR 2024
C.4 HIGH- DIMENSINNNSO EXPERIMENTS
Polynomial We ran 10 trials using batch size 10 with 100 initial points.
Neural Network Draw We ran 10 trials using batch size 10 with 100 initial points.
Knowledge Distillation We ran 10 trials using batch size 10 with 100 initial points.
C.5 NEURAL ARCHITECTURE SEARCH
We used SMAC (Lindauer al., 2022) find the optimal hyperparameters of HMC for Cell Coverage, Pest Control, and DTLZ5 benchmark problems using Bayesian optimization. For each benchmark, our new objective function was the average maximum value found for three runs of Bayesian optimization using the same problem setup as detailed above. We used the hyperparameter optimization facade in SMAC, and for each problem, we used Bayesian optimization to select 200 different HMC architectures to find the optimal combination from the following set of possible values:
Network width: [32, 512] (continuous)
Network depth: { 2, 3, 4, 5, 6) (discrete)
Network activation: ("ReI.U", "tanh") (discrete)
1og10 of likelihood variance: [-3.0, 2.0] (continuous)
1og10 of prior variance: [-3.0, 2.0] (continuous)
We then use the optimal architecture and run the benchmark for 10 trials with the same setup as described Appendix C.2 and Appendix C.3 to compare the results HMC with and without neural architecture search.
For cell coverage, the final HMC model was an MLP with 5 hidden layers, 184 parameters per layer, tanh activation, likelihood variance of 26.3, and prior variance of 0.54.
MLP with hidden layer of size 321, tanh activation, likelihood variance of 0.26, and prior variance of 0.31.
For pest control, the final HMC model
For DTLZ5, the final HMC model was an MLP with 3 hidden layers, 297 parameters per layer, relu activation, likelihood variance of 0.01, and prior variance of 0.30.
21

---

Published as a conference paper at ICLR 2024
APPENDIX D
FURTHER EMPIRICAL RESULTS
D.1 BNN ARCHITECTURE
D.1.1 NEURAL ARCHITECTURE SEARCH
To further investigate the impact of architecture on the performance of BNNs, we conduct an extensive neural architecture search over selection of the benchmark problems, varying the width, depth, prior variance, likelihood variance, and activation function. For our experiments, we use SMAC3 (Lindauer al., 2022), a framework which uses Bayesian optimization to select the best hyperparameters, and we detail the experiment setup in Appendix C. While thorough search over architectures is often impractical for realistic settings of Bayesian optimization since requires a very large number of function evaluations, we use this experiment to demonstrate the flexibility of BNNS and to showcase its potential when the design is well-suited for the problem.
We show the effect neural architecture search on HMC surrogate models in Figure A.I. On the Cell Coverage problem, the architecture search did not drastically change the performance of HMC. In contrast, extensively optimizing the hyperparameters made significant difference on the Pest Control problem, leading to IICC finding higher rewards than GPs while using fewer function evaluations; however, on this problem, I-BNN, which does not require specifying an architecture, still performs best. Neural architecture search was also able to improve the results DTLZ5, leading HMC to be competitive with other surrogate models such as I-BNNS and DKL. The difference in the benefits of the search may be attributed to some problems having less inherent structure than others, where extensive hyperparameter optimization may not be as necessary. Additionally, our original HMC surrogate model choice may already have been suitable choice for some problems, extensive search over architectures may not significantly improve the performance.
Cell Coverage (d=30, (m2)
DTLZ5 (d=20, 0=3)
Pest Control (d=25,00 0-1))
12
0.04
Reward
Reward
950
Reward
0.03
-16
900
0.02
25
50
75
100
25
50
75
100
25
50
100
125
Function Evaluations
Function Evaluations
Function Evaluations
Figure A.1: The impact of neural architecture search on HMC is problem-dependent. The dashed green line indicates the performance of HMC after an extensive neural architecture search, compared to the solid green line representing the HMC model selected from a much smaller pool of hyperparameters. We see that it has minimal impact Cell Coverage (left), moderate impact DTLZ5 (center), and extensive impact on Pest Control (right), even outperforming GPs. For each benchmark, we include for the number of input dimensions, and for the number of objectives. We plot the mean and one standard error of the mean over 10 trials.
22

---

Published as a conference paper at ICLR 2024
D.1.2 SMALLER ARCHITECTURE
Here, benchmark the performance of BNNS using the relatively small architecture specified by Kristiadi et al. (2023): an MLP with 2 hidden layers of size 50 with ReLU activation. In the main text, our experiments used MLPs with 3 hidden layers of size 128 with tanh activation. Our experiments show that the larger BNN from our main text typically leads to better performance across datasets. However, even with the smaller architecture, we find that the relative performance of the surrogate models mostly remains consistent. We still find that HMC often outperforms other approximate inference methods such as deep ensembles, and we see that SGHMC often finds suboptimal rewards. We find that LLA seems to have slightly improved performance using this smaller architecture, especially over many of the real datasets.
1-NN
HMC
SGHMC
Ensemblee
Branin (d=2, o-1)
Hartmann (d-6, o-1)
Ackley (d-10, 0m1)
16
Rewarrd
Reward
Reward
-18
60
110
170
110 Function Evaluations
Function Evaluations
Function Evaluations
50
BraninCurrin (d-2, 0-2)
159000
DTLZ1 (d-5, 0m2)
DTLZ5 (d-20, 0m3)
970
Reward
80
158000
Reward
945
157000
920
10
30
156000
Function Evaluations
Function Evaluations
Evaluations
(a) Synthetic Benchmarks
PDE Optimization (d-4, 0-1)
Interferometer (d-4, o-1)
Lunar Lander (d-12, o-1)
0.6
-0.05
Reward
Re-ward
0.4
Reward
100
0.2
-0.25
10
20
30
100
150
300
450
Function Evaluations
Function Evaluations
Function Evaluations
Pest Control (d=25, o-1)
Coverage (d-30, a-2)
Oil Spill Sorbent (d=7, o-3)
0.045
1.2 1e6
-14
Reward
0.035
Roward
1.0
-16
0.8
0.025
40
70
100
25
100
120
150
180
Function Evailuaions
Function Evaluations
Function Evaluations
(b) Real-World Benchmarks
Figure A.2: BNN results with different architecture. We show the performance of BNN surrogate models with the relatively small architecture specified by Kristiadi et al. (2023): an MLP with 2 hidden layers of size 50 with ReLU activation. For each benchmark, we include d for the number of input dimensions, and o for the number of objectives. We plot the mean and one standard error of the mean over 10 trials.
23

---

Published as a conference paper at ICLR 2024
D.2 ABLATION STUDIES
To better understand the quality of the mean estimates and uncertainty estimates of different surrogate models, we conducted ablation studies by creating hybrid models which combine the mean estimate of one surrogate with the uncertainty estimate of another. When we compare this hybrid model with each individual model, we are able get insight into the relative performance of the mean and uncertainty estimates. We provide the results for GP vs HMC, the gold standard inference for BNNs, and GP vs I-BNN, the model with the most success in high-dimensions.
HMC
Uncertainty
HMC Mean, Uncertainty
Dranin (d=2, o=1)
Hartmann (d=8, o=1)
Ackley (d=10, o=1)
Roward
Reward
Reward
-1755
20.00
75
100
100
Function Evalustions
Function Evaluations
Function Evalustions
BraninCumin (d-2, om2)
DTL21 (d=5, o-2)
DTL25 (d-20, o-3)
980
Reward
Reward
150000
0
140000
920
100
Function Evaluations
Evaluations
Function Evalustions
PDE Optimization (d=4, o=1)
Interferometer (d=4, o=1]
Lunar Lander (d=12, o=1)
0.6
-0.05
Rewcadd
0.4
Roward
150
-0.00
0.2
100
-0.15
50
0.0
200
400
Function Evaluations
Function Evalustions
Function Evalustions
Pest Control (d=25, e=1)
Cell Coverage (d=30, o=2)
Oil Spill Sorbent (d=7, o=3)
0.035
1e6
Reward
Reward
0,030
Reward
2.0
-16
0.025
0.8
-17
50
100
Function Evalustions
Function Evalustions
Function Evalustions
Polynomial (d=200)
NN Draw (d=400)
Distillation (d=6320)
60
Reward
16
Reward
50
-60
-70
30
40
100
200
200
400
600
200 300
Function Evaluations
Function Evaluetionn
Function Evalustions
Figure A.3: When we compare GP vs HMC in lower dimensions, we see that mean and uncertainty estimates of the GP need to work together to achieve optimal results, and HMC does not significantly improve when we instead use the GP mean or uncertainty estimates. In contrast, in higher dimensions, the best performing model is the HMC-Mean Gncunncryy hybrid, suggesting HMC has better mean estimates while GPs have better uncertainty estimates in this setting. We plot the mean and one standard error of the mean over 10 trials, d refers the number of input dimensions, and refers to the number of output dimensions.
24

---

Published as a conference paper at ICLR 2024
HBNN
Mean, HBNN Uncertainty
I-BNN Mean, GP Uncertainty
Branin (d=2, o=1)
Hartmann (d=6, o=1)
Ackley (d=10, o=1)
Reward
Reward
Reward
-16
-5
-18
-20
-10
25
50
75
100
100
200
100
200
Function Evaluations
Function Evaluations
Function Evaluations
BraninCurrin (d-2, om2)
DTLZ1 (d=5, om2)
DTLZ5 (d=20, om3)
160000
980
Reward
40
Reward
Reward
960
150000
940
20
920
140000
50
100
25
50
75
100
50
100
Function Evaluations
Function Evaluations
Function Evaluations
PDE Optimization (d=4, o-1)
Interferometer (dm4, o-1)
Lunar Lander (d-12, o-1)
0.6
150
Reward
-0.1
Reward
Reward
100
0.2
0.2
50
0.0
10
20
30
50
100
400 Function Evaluations
200
Function Evaluations
Function Evaluations
Pest Control (d=25, o=1)
Cell Coverage (d=30, o=2)
Oil Spill Sorbent (d=7, o=
12
1.21e6
0.035
Reward
Reward
Reward
1.0
0.030
0.8
0.025
50
100
50
100
0.6
100
150
200
Function Evaluations
Function Evaluations
Function Evaluations
Polynomial (d=200)
NN Draw (d=400)
Distillation (d=6320)
60
Reward
60
Reward
Reward
-40
50
40
-60
40
-80
100
200
300
200
400
600
100
200
300
Function Evaluations
Function Evaluations
Function Evaluations
Figure A.4: The I-BNN-Mean UP-erreryyn hybrid model outperforms GPs across a diverse set of problems, suggesting I-BNNs often provide better mean estimates. In the highest-dimensionnll prob- lem of Knowledge Distillation (6,320 dimensions). I-BNNs outperform GPs in both the uncertainty and the mean estimates, suggesting that their non-Euclidean and non-stationary similarity metric is advantagcous. We plot the mean and one standard error of the mean over 10 trials, d refers to the number of input dimensions, and o refers to the number of output dimensions.
25

---

Published as a conference paper at ICLR 2024
D.3 DEEP ENSEMBLES
While deep ensembles often provide good accuracy and well-calibrated uncertainty estimates in other settings (Lakshminarayanan al., 2017), show they perform relatively poorly for Bayesian optimization. For instance, when compared to other BNNs on benchmark problems such as BraninCurrin and DTLZI, the maximum reward found by deep ensembles plateaus at a lower value than other surrogate models. These findings are also supported by results shown in Dai et al. (2022), where deep ensembles do not perform well Bayesian optimization because they are unable to to explore the space effectively.
True Objective
Mean
2a
Function Draw
Depth:
Depth:
Depth:
-1
-2
-2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Widthh:
Width: 128
Width: 256
1
-1
1
-2
-2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Models:
Models: 5
Models: 10
1
-1
-1
-1
-2
-2
Train Points:
Train Points:
Train Points: 12
1
-1
-1
-2
-2
-2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Figure A.5: We visualize the uncertainty of deep ensembles in various scenarios. For each set of experiments, fix all other design choices with the following base parameters: depth 3, width 128, models = 5, train points = 9.
26

---

Published as a conference paper at ICLR 2024
100 Data Points
1e11
1000 Data Points
10,000 Data Points
1.0
1.5
0.8
1.0
1.0
0.6
0.4
0.5
0.5
0.2
0.0
-1
Weights
Weights
Weights
(a) Loss landscape
1000 Points
10000 Points
Model
Indegendent Model
0,02
0.02
Independent
0.00
0.00
0.00
-0.02
0.02
0.02
Independent Model
Independent Model
Independent Model
(b) Cosine similarity between models
Figure A.6: With minimal training data, the loss landscape is relatively smooth, and separately- trained models are less diverse. In the top figure, we train two models, corresponding to æ 0 and x 1. We then linearly interpolate the weights between the two models to measure how the loss changes. As we increase the number of datapoints, the loss landscape becomes less smooth and models are able to find diverse basins of attraction. For the bottom figure, we train 10 models and plot the cosine similarity between weights. With less training data, the weights of the models are more related and the models are less diverse. We plot the results for DTTLZI with 5 input dimensions and 2 output dimensions.
To further investigate the behavior of deep ensembles, conduct a sensitivity study, varying the architecture, the amount of training data, and the number of models in the ensemble.
In Figure A.5, we see that smaller training sizes can paradoxically lead to less uncertainty with deep ensembles. critical component in the success of deep ensembles is the diversity of its models. After training, each model falls within a distinct basin of attraction, where solutions across different basins correspond diverse functions. Intuitively, however, the low data regime there are fewer settings of parameters that give rise to casily discoverable basins of attraction, making it harder to find diverse solutions simply by re-initializing optimization runs. We consider, for example, the straight path between the weights of Model 1 and Model 2, and we follow how the loss changes we linearly interpolate between the weights. Specifically, given neural network weights w1 from Model and W2 from Model 2, and ((w) representing the loss of the neural network with weights on the training data, we plot the loss L(w2 + (Wi W2) * x) for varying values of z.
We share results Figure A.6 for DTLZI, a problem where deep ensembles performed poorly. We can see that in low-data regimes, although the loss between the two models does increase, the loss is significantly bigher in other regions of the loss landscape. This behavior suggests that the basins are not particularly distinct, as the loss stays relatively flat between them, and thus less likely to provide diverse solutions. However, as we increase the amount of training data, the models are able to find more pronounced basins.
We also verify that the flatter regions correspond to decrease in model diversity by measuring the cosine similarity between the model weights, and we see that in the low-data regime, models have more similar weights and therefore are less diverse.
In general. Bayesian optimization problems contain significantly fewer datapoints than where deep ensembles are normally applied. Standard Bayesian optimization benchmarks rarely exceed about 600 data points (objective queries), while in contrast deep ensembles are often trained on problems like CIFAR-I 10 which have 50,000 training points.
27

---

Published as a conference paper at ICLR 2024
2 Models
Models
10 Models
Branin (d-2, o-1)
Hartmann (d-6, o-1)
Ackley (d-10, o-1)
-19.0
19.5
Reward
Reward
Reward
20.0
-0.55
25
50
75
100
100
200
100
200
Function Evaluations
Function Evaluaatoon
Function Evaluations
BraninCurrin (d=2, o=
DTLZ1 (d=5, o=2)
DTLZ5 (d=20, o=3)
30
Reward
Reward
150000
Reward
20
940
140000
920
130000
0
50
100
25
50
100
50
100
Function Evaluations
Function Evaluations
Function Evaluations
PDE Optimization (d-4,
Interferometer (d-4, o-1)
Lunar Lander (d-12, o-1)
0.6
150
Reward
Reward
0.4
Reward
-0.0
50
-0.15
0.2
0.0
10
20
50
400 Function Evaluations
200
Function Evaluations
Function Evaluations
(d=25, o=1)
Coverage (d=30, o=2)
Spill (d=7, o=3)
1e6
-16.5
0.04
750000
Reward
Reward
Reward
700000
-17.0
650000
150 Function Evaluations
Function Evaluations
Function Evaluations
Polynomial (d= 200)
(d=400)
Distillation (d=6320)
.50
Reward
Reward
#0
-70
30
200
Function Evaluations
Function Evaluations
Function Evaluations
Figure A.7: compare behavior of ensembles with different numbers of models, we find
that the different ensembles perform similarly across many experiments, showing the robustness of our results to this hyperparameter. We plot the mean and one standard error of the mean over 10 trials, d refers to the number of input dimensions, and refers to the number of output dimensions.
28

---

Published as a conference paper at ICLR 2024
D.4 INFINITE- WIDTH BNNS
I-BNNS outperformed GPS in Bayesian optimization on a variety of high-dimensional problems, and we show results in Figure A.8. The first row of results corresponds finding the maximum value of random polynomial function, and the second and third rows show the results of maximizing function draw from a neural network. For the neural network function draw benchmark, we experimented with many different architectures for the neural network to ensure that we had a diverse set of objective functions to maximize, as denoted in the title of each plot. In all cases, I-BNNS were able to find significantly larger values than GPS.
Polynomial (d=100)
Polynomial (d=200)
Polynomial (d=400)
3.5
3.0
Reward
2.5
I-BNN
Reward
Reward
6
GP
2.0
1.5
2
400
600
400
Function Evaluations
Function Evaluations
Function Evailuations (d (d-200), tanh 2x512
(d 200), tanh 2x64
NN Draw (d- tanh 2x256
25.0
80
120
22.5
70
Reward
Reward
100
20.0
60
17.5
80
15.0
60
200
400
600
200
400
600
200
400
600
Function Evaluations
Function Evaluations
Function Evaluations
NN Draw (d=200), relu 2x64
NN Draw (d=200), relu 2x256
NN Draw (d=200), relu 2x512
3000
6000
20000
Reward
2500
Reward
4000
Reward
2000
15000
1500
2000
10000
1000
200
400
600
200
400
200
400
600
Function Evaluations
Function Evaluations
Function Evaluations
Figure A.8: I-BNNS outperform GPS in high dimensions. We plot the mean and one standard error of the mean over 3 trials, and d corresponds to the number of input dimensions. For the neural network draw benchmark, the architecture of the neural network which the function is drawn from is also included, where 64 means the network has 2 hidden layers of 64 nodes each.
29

---

Published as a conference paper at ICLR 2024
True Objective
Mean
2o
Function Draw
Depth:
Depth:
Depth:
2
0
0
0
-1
-1
-1
2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.30
0.75
1.00
Weight Variance:
Weight Variance:
Weight Variance:
-1
-1
-1
-2
-2
-2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Bias Variance:
Bias Variance:
Bias Variance:
2
2
-2
-2
-2
0.25
0.50
0.75
1.00
0.50
0.75
1.00
0.50
0.75
1.00
Activation: erf
Activation: relu
2
2
-2
Figure A.9: We visualize how posterior predictive distribution of I-BNNS changes with different design choices. For each set of experiments, we all other design choices with the following base parameters: depth = 3, weight variance 10, bias variance 5, activation relu.
In Figure A.9, conduct sensitivity analysis the design of I-BNNS, showing how the posterior changes as we vary certain hyperparameters. Unlike standard GPS with RBF or Matérn kernels which are based in Euclidean similarity, I-BNNS instead have non-stationary and non-Euclidean similarity metric which is more suitable for figh-dimensiioal problems. Additionally, I-BNNS consist of relatively strong prior, which particularly useful the data-scarce settings common to high-dimensional-pe probllms.
30

---

Published as a conference paper at ICLR 2024
Problem
Dim
GP MLL
I-BNN MLL
Branin
-3775.3
-1354.1
Hartmann
-1320.4
-1630.1
Ackley
10
-1463.6
-1824.3
Ackley
20
-1657.3
-2070.2
Ackley
50
-2788.9
-2316.4
Ackley
100
4131.6
-2456.9
NN Draw
10
-3226.9
-1829.6
NN Draw
50
-15544.3
-2344.7
NN Draw
100
339197.5
-2453.8
NN Draw
200
-35775.2
-2584.0
NN Draw
400
.48710.9
-2737.2
NN Draw
800
-63123.4
-2868.9
Distillation
6230
-2126658.7
-3059.8
Table A.I: I-BNNS have larger marginal likelihoods than GPs on high-dimensional problems. Here, we show the marginal log likelihood (MLL) of GPS and I-BNNS various benchmark problems, and we estimate the MLL by sampling 1000 points randomly from the input domain.
To explore the suitability of the neural network kernel, we compare the marginal likelihood of GPs and I-BNNS on problems with various dimensions, and we report results Table A.1. We see that I-BNNS and GPs have similar marginal likelihoods problems with fewer dimensions, such as Branin and Hartmann, but as we increase the number of dimensions, the marginal likelihood of GPS becomes lower than that of I-BNNS. Interestingly, as quickly for Ackley as it does for the neural network draw test problem. This may be due to the neural network draw objective function having higher non-stationarity, so the standard GP kernel is less suitable for this problem compared to Ackley. For the knowledge distillation experiment, we that the marginal likelihood of GPS plummets, and it was also able to find high rewards for the problem as shown in Figure 5. Overall, I-BNNS have higher marginal likelihood than GPs on nigh-dimensiooal problems, indicating that the I-BNN prior may be more reasonable in these settings.
that the marginal likelihood of GPs does not decrease
31

---

Published as a conference paper at ICLR 2024
D.5 HAMILTONIAN MONTE CARLO
We also explore the effect of the activation function on IIMC. In Figure A. we see that the function draws from BNNS with tanh activations appear quite different from the function draws from BNNS with ReLU activations. The tanh draws are smoother and have more variation, while the ReLU draws seem to be more jagged. We also include results in Figure A.11 which compare the performance of ReLU and tanh activations in Bayesian optimization problems. We see that there does not appear to be an obvious trend, the optimal choice of activation function problem-specific.
7.5
7.5
5.0
5.0
2.5
2.5
0.0
0.0
2.5
5.0
5.0
-7.5
-7.5
Figure A010: The function draws of BNNS with tanh activations appear to be similar to the function draws from a GP. contrast, the function draws from BNNS with ReLU are often jagged, and the network seems to deviate less from the mean.
Ackley (d=10
DTLZ1 (d=5, a=2)
19.0
159200
Reward
Reward
-19.4
159000
-9966
158800
19.88
158600
relu Activation
tanh
relu Activation
tanh
Figure A.11: In practice, ReLU and tanh activation functions can have comparable performance synthetic functions.
32

---

Published as a conference paper at ICLR 2024
D.6 GAUSSIAN PROCESSES
Matern Opt
Matern Marg
RBF Opt
RBF Marg
Branin (d=2, om1)
Hartmann (d=6, om1)
Ackley (d=10, om1)
16
Reward
Reward
Reward
-18
-20
25
50
100
100
200
100 Function Evaluations
200
Function Evaluations
Function Evaluations
BraninCurrin (d-2, 0m2)
DTLZ1 (d-5.0
DTLZ5 (d-20, 0m3)
980
Reward
Reward
150000
Reward
960
20
140000
920
100
25
50
100
50
100
Function Evaluations
Function Evaluations
Function Evaluations
The point estimate for hyperparameter selection tends to improve performance on synthetic datasets. The solid lines refer to hyperparameter optimization using a point estimate, while dashed lines correspond to fully Bayesian marginalization. We see that while the methods perform similarly on Branin, Hartmann, and DTLLZI, the point estimates perform slightly better for other datasets. The optimal choice of GP kernel is also problem-dependen and RBF perform similarly, although the point estimates for Matern outperform RBF on Ackley. We plot the mean and one standard error of the mean over 10 trials, d refers to the number of input dimensions, and o refers to the number of output dimensions.
Figure 12
In many instances, Matérn
Matern Opt
Matern Marg
RBF Opt
RBF Marg
PDE 0 Optimization (dm4, om1)
Interferometer (dm4, om1)
Lunar Lander (d=12, om1)
150
0.6
-0.05
Reward
Reward
Reward
0.4
100
0.2
50
10
20
30
0.0
25
75
100
200
400
Function Evaluations
Function Evaluations
Function Evaluations
Pest Control (d-25, o-1)
Cell Coverage (d-30, o-2)
Oil Spill Sorbent (d-7, o-3)
1e6
0.04
Roward
Reward
Roward
1.0
0.03
-16
0.8
25
50
75
100
125
25
50
100
100
125
150
175
200
Function Evaluations
Function Evaluations
Function Evaluations
Figure A.13: The method of hyperparameter selection greatly impacts the performance of GPS on real-world datasets. The solid lines refer to hyperparameter optimization using point estimate, while dashed lines correspond to fully Bayesian marginalization. We see real-world datasets often find marginalization of the hyperparameters also problem-dependen In many instances, Matérn and RBF perform similarly, although the point estimates for Matérn outperform RBF on discrete problems such as Pest Control and Cell Coverage. We plot the mean and one standard error of the mean over 10 trials, d refers to the number of input dimensions, and o refers to the number of output dimensions.
be more effective. The optimal choice of GP kernel is
33

---

Published as a conference paper at ICLR 2024
D.7 DEEP KERNEL LEARNING
Rather than using the marginal likelihood to optimize parameters for DKL, we can also use the conditional marginal likelihood (Lotfi et al., 2023). We see in Figure A.14 that there is no clear preference for using the marginal likelihood (ML), which we use through all other experiments in the paper, or conditional marginal likelihood (CML) for our problems.
DKL-ML
DKL-CLML
Branin (d=2, o-1)
Hartmann (d- 6, a=1)
Ackley (d= 10, o-1)
2.5
-19
Reward
Reward
2.0
Reward
1.5
-20
1.0
25
75
100
100
200
100
200
Furrction Evaluations
Function Evaluations
Function Evaluations
BraninCurrin (d=2, a-2)
DTLZ1 (d=5, a=2)
DTLZ5 (d=20, ow3)
975
Reward
40
Reward
150000
Reward
950
200
925
140000
900
50
25
100
100
Furctiio Evaluations
Function Evaluations
Function Evaluations
PDE Optimization (d=4, e-1)
Interferometer (d=4, o=1)
Lunar Lander (d=12, o-1)
200
0.4
Reward
0.1
Reward
Reward
0.2
100
-0.2
0.0
10
20
50
100
400 Function Evaluations
200
Function Evaluations
Function Evaluations
Pest Control (d=25, o=1)
Cell Coverage (d-30, o=2)
Oil Spill Sorbent (d=7, o=3)
1e6
Reward
-16
Reward
0.0275
Reward
0.0250
-17
0.0225
50
100
20
40
60
100
150
200
Function Evaluations
Function Evaluations
Function Evaluations
Figure A.14: There is no clear preference for using the marginal likelihood (ML) or conditional marginal likelihood (CML) to optimize the parameters of DKL
34

---

Published as a conference paper at ICLR 2024
D.8
ACQUISITION BATCH SIZE
DKL
I-BNN
HMC
SGHMC
LLA
Ensemble
Batch Size 2
Batch Size 10
Batch Size 20
-17
-16
-16
Reward
Reward
Reward
-19
-20
-20
-21
75
Function Evaluations
Function Evaluations
Function Evaluations
Ackley (d=10, 0=1)
Batch Size 2
Batch Size 5
Batch Size 10
980
980
960
Reward
Reward
Reward
940
940
920
920
920
25
50
75
100
900
25
50
7.
100
900
25
50
75
100
Function Evaluations
Function Evaluations
Function Evaluations
(b) DTLZ5 (d=20, o=3)
Size
Size
-14
-15
Reward
Rewward
Rewarr nd
-16
-16
-16
-17
-17
25
50
100
125
50
75
100
125
50
75
100
125
Function Evaluations
Function Evaluations
Function Evaluations
Pest Control (d=25, 0=1)
Batch Size 2
Batch Size
Batch Size 10
0.040
0.05
0.040
Reward
Reward
0.04
Reward
0.035
0.030
0.03
0.030
0.025
0.025
0.02
25
50
75
100
25
50
75
100
25
50
75
100
Function Evaluations
Function Evaluations
Function Evaluations
(d) Cell Coverage (d=30, o=2)
Figure A.15: The batch size of the acquisition function impacts the performance of the models, but the general trends remain similar. The relative performance of the models are similar across varying batch sizes, with few notable exceptions. The I-BNN seems plateau on DTLZ5, and is unable to find high rewards. GPs also seem to perform significantly better on Cell Coverage with a batch size of compared to other sizes. For each benchmark. we include for the number of input dimensions, and o for the number of objectives. We plot the mean and one standard error of the mean over 10 trials.
35

---

Published as a conference paper at ICLR 2024
D.9 LIMITATIONS OF GAUSSIAN PROCESS SURROGATE MODELS
Due to their assumptions of stationarity as noted in Section 4.3, GPS struggle in non-stationary settings. In Figure 161 compare the posterior distribution from with the posterior from different BNN surrogate models. We show results after 20 iterations of Bayesian optimization, starting with an initial point of greater variance between -2 2 and 2, and there is also slight downward trend. We see that due to their stronger assumptions, GPS are not able to find the true global maximum 0.8, instead getting suck in local optima. In contrast, HMC and I-BNNs are able to find the global maximum within 20 iterations.
The function we wish maximize is staatioatoo::: the function has
Objective
Draw
0.5
0.5
0.0
0.0
0.5
0.5
-0.5
1.0
-1.0
-1.0
(a) GP
(b) HMC
(c) I-BNN
Figure A.16: GPS struggle to find the global maximum for non-stationary functions. After 20 function evaluations, the GP does not accurately model the function around x -1 because is unable to account for the sudden increase in scale due to its assumption of stationarity. BNNs do not suffer from the same pathologies and are able to find the true maximum.
An additional limitation of GP surrogate models, which we demonstrate in Figure A.17, is its perfor- mance on multi-objective problems in Bayesian optimization. Although GPs have been successfully extended to wide range of multi-objective problems, in the interest of making the approaches scalable, there are many assumptions placed onto the kernel. In the most naive setting, we can model each objective independently. While this approach is convenient, it completely ignores the relationship between objective values and has no notion of shared structure, so it is unable to take advantage of all of the information in the problem. Multi-objective covariance functions can also be decomposed as Kronecker product kernels. While this approach can have significant computational advantages compared modeling the full covariance function, it requires each objective to itself be modeled with the same underlying kernel. Thus, this method of modeling multiple objectives will fail capture the nuances of each particular objective when the functions have differing properties.
In Figure A.17, show the result of twenty iterations of Bayesian optimization a synthetic multitask example, where we care about optimizing over the fourth function but provide additional information through the other three datasets. We use the GP with Kronecker product kernels model the multiple objectives. In our experiment, although the GP able to learn the proper length scale and variance over the three additional functions with similar length scales, struggles accurately the fourth objective. Because the GP is unable to account for the differences between the four functions, does find the global optimum. Unlike GPs, BNNS are restricted to striet covariance structures and are able to produce well-calibrated uncertainty estimates in multi-objective settings. The BNNs are able accurately fit four functions, including the fourth objective function which has much smaller length scale compared to the others.
36

---

Published as a conference paper at ICLR 2024
Objective 1
Objective 2
0.50
0.20
Values
Values
0.00
0.00
-0.20
-5.0
-2.5
0.0
2.5
5.0
-5.0
-2.5
0.0
2.5
5.0
Objective 3
Objective 4
1.00
Values
0.25
Values
0.00
0.00
-0.25
-5.0
-2.5
0.0
2.5
5.0
-5.0
-2.5
0.0
2.5
5.0
Objective 1
Objective 2
1.00
1.00
Values
Values
0.00
0.00
-1.00
-1.00
-5.0
-2.5
0.0
2.5
5.0
-5.0
-2.5
0.0
2.5
5.0
Objective
Objective
1.00
Values
Values
0.00
0.00
-2.00
-1.00
5.0
-2.5
0.0
2.5
5.0
-5.0
-2.5
2.5
5.0
(b) HMC
Figure A.17: have hard time finding the global maximum in multi-objective settings. Multi-task GPs learn one length scale across all objectives, which may not be suitable for many datasets. In this example, it does not find the global minimum in the 4th objective because treats the shorter length scale as noise. In contrast, BNNs are able to appropriately model the uncertainty for the 4th objective and find its true minimum.
37

---

Published as a conference paper at ICLR 2024
D.10 LARGE NUMBER OF FUNCTION QUERIES
To further accentuate the distinctions between BNNS and GPS, we experiment with larger number of function queries. Specifically, we look to maximize function with 200 input dimensions drawn from fully connected neural network with 5 layers and 256 nodes per layer. We expect this function to have high degree of statsoationarity, making it difficult for standard GPs. We start with 2000 initial function queries and end the experiment at 3000 function queries. We share results in Figure A.18.
GP
DKL
I-BNN
HMC
SGHMC
LLA
Ensemble
Large NN Draw (d=200, o=1)
70
Max Reward
60
50
2000 2200 2400 2600 2800 3000
Function Evaluations
Figure A.18: BNNs outperform GPs when there are a large number of function queries.
This experiment reveals several valuable findings: (1) I-BNNs remain competitive relative to the alternatives; (2) BNN surrogates start to perform more effectively, able to leverage the additional data for representation learning; (3) deep ensembles, in particular, are greatly improved with more data, in line with our explanation that the relative poor performance was due to an inability to find diverse models corresponding posterior modes when there is limited data. At the same time, we note most Bayesian optimization problems do not have many objective queries, since Bayesian optimization is often found be most valuable when the objective expensive query. In this light, perhaps (1) is the most valuable of the findings, since it shows consistency of the relatively strong I-BNN surrogate, In the future, we might expect I-BNNS to become mainstream surrogate model for Bayesian optimization.
38

---

Published as a conference paper at ICLR 2024
D.11 RUNTIME
While inference time is relevant for the comparison of surrogate models, in many real-world Bayesian optimization scenarios, the most expensive computation often lies in the querying of the objective function, which may include actions such as synthesizing new material, training a large neural network convergence, etc. For these scenarios, the quality the surrogate model uncertainties may be much more important than the cost of inference. For completeness, in rare instances where inference-time relevant consideration, we provide wall-clock times of all surrogate models across experiments in Table A.2, and we compare the performance of the surrogate models within fixed time budget in Figure A.19
Problem
GP
DKL
I-BNN
HMC
SGHMC
LLA
Ensemble
Branin
2.62
124.55
3.97
214.40
142.21
19.29
190.24
Hartmann
8.20
123.00
7.79
663.22
381.57
49.25
413.50
Ackley
4.28
263.78
4.69
141.66
201.48
28.29
225.41
BraninCurrin
14.17
198.80
19.76
237.71
365.86
61.09
432.94
DTLZI
8.25
117.02
10.82
183.74
127.54
21.97
194.68
DTLZ5
21.50
308.12
22.18
194.81
147.07
38.99
231.96
PDE
240.48
512.98
239.23
456.48
628.14
312.40
981.50
Interferometer
16.09
939.49
17.97
690.07
540.59
83.61
944.87
Lunar Lander
458.96
3140.30
621.20
3118.65
684.61
802.13
829.44
Pest Control
3.84
268.06
5,43
66.18
219.08
27.87
253.37
Cell Coverage
9.50
593.09
13.26
101.23
181.11
35.00
243.17
Oil Spill
37.51
300.20
44.81
291.53
350.30
124.71
438.89
Polynomial
16.28
673.40
20.59
211.94
486.75
106.77
540.41
NN Draw
232.48
1458.11
391.95
896.34
451.34
799.88
1124.63
Distillation
4719.40
3982.80
4937.27
4282.89
299.40
3223.16
3007.93
Table A.2: Wall-clock time in seconds of one trial of each experiment, with experiment details
specified in Appendix C. We record the mean time over 10 trials.
Ackley (d=10)
GP
-16
DKL
Reward
I-BNN
HMC
-18
SGHMC
LLA
Ensemble
400
600
Time (s)
Figure A.19: For objective functions which very inexpensive to query, like Ackley, we find GPS and I-BNNS to outperform other surrogate models given a fixed amount of time. However, for the many Bayesian optimization problems instead which expensive query, and the total time would be dominated by the function query instead of inference time. We record the mean and standard error over 10 trials.
39


Reviews:
[{"review_id": "H0teV8Q600", "reviewer": "Reviewer_6qj4", "summary": "This paper provides a comprehensive empirical study of using Bayesian neural networks as the surrogate in Bayesian optimization. The paper considers a number of different BNNs, and performed experimental comparisons in a variety of experiments. Some interesting insights are shown from the experiments, including when standard GP surrogate is better and when BNN is better, HMC is often the best method for inference for BNN, deep kernel learning is usually competitive and deep ensemble is usually not, etc.", "questions": "- I find the contrast between deep kernel learning and deep ensemble particularly interesting, because both methods are able to use the strong representations learned by neural networks. I suppose the reason why deep kernel learning works better is because it can be readily plugged into BO which will take care of the exploration, while deep ensemble doesn't do well in exploration. Please see if this makes sense.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The methods under comparison are carefully selected to span a wide range of possible BNNs, and experiments are nicely designed to unveil specific insights about the relative strengths/weaknesses of different families of methods.\r\n- I think some of the conclusions/insights from the empirical comparisons can indeed be useful for future applications of Bayesian optimization, such as the competitiveness of deep kernel learning, the promising results of infinite-width BNNs in high-dimensional problems (which is a new observation to the best of my knowledge), etc.\n- The synthetic experiments in Figures 1 and 2 are nicely designed to illustrate the influence of different factors, and also find which are the parameter combinations likely to work better. The experiments \"Quality of Mean and Uncertainty Estimates\" on the potential of mixing different mean and uncertainty estimates are also particularly interesting.\n- The paper is well written, the contributions are nicely organized and discussed.", "weaknesses": "**(1)** I think it would make the study more complete if another relevant line of works is discussed: using (non-Bayesian) neural networks as the surrogate in BO and using neural tangent kernel for exploration. The recent line of work on neural bandits has made it possible to use (non-Bayesian) neural networks as the surrogate in BO while still preserving the regret guarantee of BO by using the theory of the NTK, The relevance of neural bandits in BO has been shown by [1] below, and you can also refer to [1] to find the related works on neural bandits. In fact, I think the findings in [1] can be used to corroborate some of the findings in this work. For example, [1] also found that deep ensemble doesn't work well and explains it by arguing that deep ensemble cannot do principled exploration, I think this is in fact consistent with what's observed in this work, because the performance of deep ensemble plateaus at low objective values because it's subpar exploration ability makes it unable to find the region containing the global optimum. The paper [2] below also did an empirical study of neural bandit methods, so the findings in [2] may also be compared/combined with those in this paper to potentially get more insights. For example, [2] also found that neural bandits tend to work better when the objective function is complicated.\nIn fact, the connection between BNN-surrogate BO and neural bandits has also been discussed by the concurrent work of Kristiadi et al. (2023) in the context of linearized-Laplace approximation. The recent work of [3] has also shown the potential of using NTK in kernel regression, which may also be an empirical justification for the potential of BO with NTK based surrogate.\n\n[1] Sample-Then-Optimize Batch Neural Thompson Sampling, 2022.      \n[2] Empirical Analysis of Representation Learning and Exploration in Neural Kernel Bandits, 2021.      \n[3] Kernel Regression with Infinite-Width Neural Networks on Millions of Examples, 2022.\n\n**(2)** Another minor point which can make the paper easier to read is that when referring to the appendix, it'll make it easier for the reader if the specific subsection is referred to instead of just \"Appendix D\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Z30aw9Qpx", "reviewer": "Reviewer_XqQk", "summary": "The paper is an empirical investigation into the use of Bayesian neural networks (BNNs) as surrogate models for BO instead of the traditional GPs. The BNN inference procedures investigated are HMC, SGHMC, deep ensembles, infinite-width BNNs, linearlized Laplace approximations, and deep kernel learning. The authors compare the performance of GPs and the various BNN inference procedures via the maximum reward attained over several standard synthetic benchmarks, real-world benchmarks, and high-dimensional settings. The paper also investigates several secondary aspects of BNNs for BO, including the role of the NN hyperparameters, the performance of hybrid models, the effect of the number of function evaluations available, and the computational runtime.", "questions": "1. This question concerns the experimental details outlined in Appendix C.1 and what I've gleaned from the code. When a GP model is used, it undergoes hyperparameter optimization via maximizing the marginal likelihood w.r.t. to the hyperparameters at every BO iteration. When a HMC model is used, it is also described to undergo a hyperparameter optimization procedure which is an iterated grid search that chooses the set of hyperparameters that (to my understanding) maximizes the maximum reward attained after all BO iterations in a trial. This optimization procedure is different from the GP one in that it requires an entire BO trial to compute the score of a single set of hyperparameters, and hence requires several BO trials as opposed to the GP one that is optimized per iteration and does not require several BO trials. Is this an accurate understanding? If so, could you comment on the validity of comparing the results of the GP model and the HMC model (along with the other BNN models since their hyperparameters are arrived at via the HMC search as well)? The concern is that the GP model did not have the same opportunity to do a 'meta-optimization' over several BO trials which might have been used to optimize other (hyper-)hyperparameters such as the choice of lengthscale and outputscale priors. Or was it the explicit intention to compare a standard GP setup against an (a priori unknown) 'optimal' BNN ?\n\n2. From Figure 6, GP and I-BNN are the top 2 performing models on all problems, and I-BNN dominates by far on high dimensional problems. However, I-BNNs are equivalent to (and implemented as) GPs with a specific neural-network based kernel. One may reach the conclusion that GPs still reign supreme in BO: use a standard Matern kernel for low dimensional problems, and the I-BNN kernel for high dimensional problems, and ignore BNNs that are not also GPs. Would you say this is a fair alternate conclusion?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The results are of interest to BO researchers and BO practitioners looking for potential methods of improving the effectiveness of BO in real-world applications.\n2. The empirical investigation is extensive and carefully planned, covering several synthetic and real-world benchmarks, and provides support for many interesting hypotheses as well, such as the relative performance on high dimensional problems and the role of hyperparameters including network architecture.\n3. This paper is a gold standard for clarity and writing.", "weaknesses": "1. For an empirical paper whose conclusions rest solely on the experimental results, 5 trials for each experimental setup is too little, as evidenced by multiple plots having heavily overlapping confidence intervals, Figure 6 in particular.\n\n2. A few clarifying questions, please see the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aXRIZK8grf", "reviewer": "Reviewer_v5S3", "summary": "This paper creates a study about the performance of Bayesian surrogates inside the Bayesian optimization framework. They performed experiments on real and synthetic datasets in order to understand the performance of surrogates. They report some findings such as the ranking of methods is highly problem dependent, HMC is the most successful approximate inference procedure for fully stochastic BNNs, and that infinite-width BNNs are promising.", "questions": "- How would more recent methods rank in the comparison, such as PFN4BO [1]?\n- How is the performance of DKL affected by regularization approaches  such as [2], [3]?\n\n\n[1] Müller, Samuel, et al. *PFNs Are Flexible Models for Real-World Bayesian Optimization.*\n\n[2] Lotfi, S., Izmailov, P., Benton, G., Goldblum, M., & Wilson, A. G. *Bayesian model selection, the marginal likelihood, and generalization.*\n\n[3] Patacchiola, M., Turner, J., Crowley, E. J., O'Boyle, M., & Storkey, A. J. *Bayesian meta-learning for the few-shot setting via deep kernels.*", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "- The topic is interesting and somewhat important for the community.\n- The presentation of theory and the experiments is well presented and sound.", "weaknesses": "- There is no discussion on the interaction of the surrogate and the acquisition function.\n- I agree with the authors that the time might not be relevant when the function evaluations are expensive, still it is important to create an experiment assuming fast function evaluations and see whether the ranking holds\n- Although the dataset collection is diverse, the study is performed on a very small amount of datasets. There is no guarantee that these findings extrapolate easily to new datasets.\n- Although the insights are interesting (some of them are not surprising), the effective impact is questionable. How could these insights lead to SotA in drug discovery algorithms, active learning, material science or hyperparameter optimization?\n- No discussion on the regularization effect on the surrogates. What happens if I regularize the DKL or the Bayesian neural networks via some useful prior?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)