Supplementary material for the paper : Bayesian Optimization with Robust Bayesian Neural Networks

In this supplementary material, we provide additional details for the experimental setup of the paper: Bayesian Optimization with Robust Bayesian Neural Networks. We also present a set of additional plots for each experiment from the main paper. A Extension to parallel Bayesian optimization In this section, we define a variant of our algorithm for settings in which we can perform multiple evaluations of ft in parallel. Utilizing such parallel (and asynchronous) function evaluations in a principled manner for BO is non-trivial as we ideally would like to marginalize over the outcomes of currently running evaluations when suggesting new parameters x for which we want to query ft. To achieve this, Snoek et al.  proposed an acquisition function which we refer to as Monte Carlo EI αMCEI that we adopt here for our model. Formally, we estimate αMCEI as αMCEI(x;D, R) = ∫ θ,y αEI ( x;D ∪ {(xi, y i)}xi∈R ) p(y i | xi, θ)p(θ | D)dθdy ≈ 1 M M ∑ k=1 αEI ( x;D ∪ {(xi, y i)}xi∈R ) with y i ∼ p(y i | xi, θ)p(θ | D), (1) where {(xi, y i)}xi∈R is the set of currently running function evaluations (and their predicted targets) and where, in practice, we simply take M = 1 sample. Note that the computation of αEI within the sum again requires multiple samples from the HMC procedure. As for EI we can differentiate through the computation of Equation (1) and maximize it using gradient ascent. B Computational Requirements For any BO method it is important to keep the computational requirements for training and evaluating the model in mind. To this end we want to draw a comparison between our method and DNGO with respect to computational costs. First, SGHMC sampling is similarly cheap as standard SGD training of neural networks, i.e. training a DNGO model from scratch and sampling via SGHMC has similar computational costs. If we were to start sampling from scratch with every incoming data-point and would fix the number of MCMC steps to be equivalent to K runs through the whole dataset then the computational complexity for sampling would grow linearly with the number of data-points. In practice, we warm-start the optimizer during BO with the last sample from the previous evaluation and perform 2000 SGHMC steps (corresponding to 2000 batches) as burn-in, followed by 50 · 50 29th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain. 3 2 1 0 1 2 2.0 1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0 A fit of the sinc function using Variational Inference sinc(x) BBB −3 −2 −1 0 1 2 −2.0 −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 2.0 A fit of the sinc function using Dropout sinc(x) Dropout MC 3 2 1 0 1 2 2.0 1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0 A fit of the sinc function using Probabilistic Backpropagation (PBP) sinc(x) PBP −3 −2 −1 0 1 2 −2.0 −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 2.0 A fit of the sinc function using SGHMC sinc(x) Adaptive SGHMC  Four fits of the sinc function from 20 data-points. On the top-left the regression task was solved using our re-implementation of the Bayes by Backprop (BBB) approach from Blundell et al. . On the top-right we used our re-implementation of the Dropout MC approach from Gal and Ghahramani . In the bottom-left probabilistic Backpropagation  was used. On the bottom-right is a fit using SGHMC. As it can be observed most methods are overly confident, or have constant uncertainty bands, in large regions of the input space. Note that this function has no observation noise. sampling steps (retaining every 50th sample). This budget was fixed for all tasks. SGHMC sampling is thus slightly faster than DNGO and orders of magnitude faster than GPs (see also the comparison between DNGO and GPs from Snoek et al. ); including acquisition function optimization it takes < 30 seconds. With more function evaluations we would increase the budget but expect a runtime of < 2min to select the next point for 50k function evaluations. We note that, if one wants to perform BO in large input spaces (e.g. for ML models with a very large number of parameters) it could be necessary to also increase the size of the used neural network model. C Additional Experiments C.1 Obtaining well calibrated uncertainty estimates with Bayesian neural networks As mentioned in the main paper, there exists a large body of work on Bayesian methods for neural networks. In preliminary experiments, we tried several of these methods to determine which algorithm was capable of providing well calibrated uncertainty estimates. All approximate inference methods we looked at (except for the MCMC variants) exhibited one of two problems (including the variational inference method from Blundell et al. , the method from Gal and Ghahramani  as well as the expectation propagation based approach from Hernández-Lobato and Adams ): either they did severely underfit the data, or they poorly predicted the uncertainty in regions far from observed data points. The latter behaviour is exemplified in Figure 1 (left) where we regressed the sinc function from 20 observations with a two layer neural network (50 tanh units each) using our implementation of the Bayes by Backprop (BBB) aproach from Blundell et al. . In contrast, a fit of the same data with our method more faithfully represents model uncertainty as depicted in Figure 1 (right).