Antifragile and Robust Heteroscedastic Bayesian Optimisation
Authors: Ryan Rhys-Griffiths
Miguel Garcia-Ortegon
Alexander A. Aldrick
Alpha A. Lee
Venue:  (2020)
Keywords: Bayesian Optimisation
Gaussian Processeses
Heteroscedasticity

Abstract:     Bayesian Optimisation is an important decision-making tool for high-stakes applications in drug discovery and materials design. An oft-overlooked modelling consideration however is the representation of input-dependent or heteroscedastic aleatoric uncertainty. The cost of misrepresenting this uncertainty as being homoscedastic could be high in drug discovery applications where neglecting heteroscedasticity in high throughput virtual screening could lead to a failed drug discovery program. In this paper, we propose a heteroscedastic Bayesian Optimisation scheme which both represents and optimises aleatoric noise in the suggestions. We consider cases such as drug discovery where we would like to minimise or be robust to aleatoric uncertainty but also applications such as materials discovery where it may be beneficial to maximise or be antifragile to aleatoric uncertainty. Our scheme features a heteroscedastic Gaussian Process (GP) as the surrogate model in conjunction with two acquisition heuristics. First, we extend the augmented expected improvement (AEI) heuristic to the heteroscedastic setting and second, we introduce a new acquisition function, aleatoric-penalised expected improvement (ANPEI) based on a simple scalarisation of the performance and noise objective. Both methods are capable of penalising or promoting aleatoric noise in the suggestions and yield improved performance relative to a naive implementation of homoscedastic Bayesian Optimisation on toy problems as well as a real-world optimisation problem.

Under review as a conference paper at ICLR 2019
ANTIFRAGILE AND ROBUST BAYESIAN OPTIMISATION
HETEROSCEDASTIC
Anonymous authors Paper under double-blind review
ABSTRACT
Bayesian Optimisation is important decision-making tool for high-stakes ap- plications in drug discovery and materials design. An oft-overlooked modelling consideration however is the representation of input-dependent or heteroscedas- tic aleatoric uncertainty. The cost of misrepresenting this uncertainty as being homoscedastic could be high drug discovery applications where neglecting heteroscedasticity in high throughput virtual screening could lead to failed drug discovery program. In this paper, we propose a heteroscedastic Bayesian Optimisa- tion scheme which both represents and optimises aleatoric noise in the suggestions. We consider cases such as drug discovery where we would like to minimise or be robust to aleatoric uncertainty but also applications such as materials discovery where it may be beneficial to maximise or be antifragile to aleatoric uncertainty. Our scheme features heteroscedastic Gaussian Process (GP) as the surrogate model in conjunction with two acquisition heuristics. First, we extend the aug- mented expected improvement (AEI) heuristic to the heteroscedastic setting and second, we introduce a new acquisition function, aleatoric- aleatoric-penalised expected im- provement (ANPEI) based simple scalarisation of the performance and noise objective. Both methods are capable of penalising promoting aleatoric noise in the suggestions and yield improved performance relative a naive implementation of homoscedastic Bayesian Optimisation on toy problems as well as real-world optimisation problem.
INTRODUCTION
Bayesian Optimisation is already being utilised to make decisions in high-stakes applications such as drug discovery [1, 2, 3, 4, 5], materials discovery [6, 7, 8]. roboties [9], sensor placement [10] and tissue engineering [11]. In these problems heteroscedastic input-dependent noise is rarely accounted for and the assumption of homoscedastic noise often inappropriate. As a case study, heteroscedastic noise is the rule rather than the exception in the majority of scientific datasets. This the case, not only in experimental datasets, but also in datasets where properties are predicted computationally. We illustrate this for molecular hydration free energies in Figure 1 using the dataset of [12] where there is a distribution noise values and in general the noise function might be expected grow in proportion chemical complexity [13]. The consequences of neglecting heteroscedastic noise are illustrated using a second example in Figure 2. The homoscedastic model will underestimate noise in certain regions which could induce Bayesian Optimisation scheme to suggest values possessing large aleatoric noise. In an application such as high-throwd high-throughput virtual screening the cost of misrepresenting the noise during the screening process could amount year wasted in the physical synthesis of drug [14).
In materials discovery, we may derive benefit from or be antifragile [15] towards high aleatoric uncertainty. In an application such as the search for high-performin perovskite solar cells, we are faced with an extremely large compositional space, with millions of potential candidates possessing high aleatoric noise for identical reproductions([6] In this instance we may want to guide search towards a candidate possessing a high photoluminescence quantum efficiency with high aleatoric noise. If the cost of repeating material syntheses is small relative to the cost of the search, the large aleatoric noise will present opportunities to synthesise materials possessing efficiencies far above their mean values.

---

Under review as a conference paper at ICLR 2019
In this paper present heteroscedastic Bayesian Optimisation scheme capable of both representing and optimising aleatoric noise in the suggestions. Our contributions are:
1. The introduction of novel combination of surrogate model and acquisition function designed optimise heteroscedastic aleatoric uncertainty to be used situations where one wants to minimise (be robust to) aleatoric uncertainty as well as situations where one wants to maximise (be antifragile towards) aleatoric uncertainty.
2. A demonstration of our scheme's ability outperform naive schemes based on homoscedas- tic Bayesian Optimisation on toy problems in addition to real-world optimisation problem.
3. The provision of an open-source implementation of the model.
350
300
Frequency
250
Frequency
300
200
150
200
100
0.00 0.01 0.02 0.03 6.64 6.05 0.06 0.07 0.88 0.09 0.10
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Error magnitude (kcal/mol)
Error magnitude (kcal/mol)
(a) Histogram of calculated uncertaintics
(b) Histogram experimental uncertainties
Figure Fig. (a) shows the frequency histogram of the calculated uncertainties against the error magnitude (kcal/mol) for the FreeSolv hydration energy dataset ([12]). Fig. (b) shows a similar plot for the experimental uncertainties.
(a)
Dataset
(b) Homoscedastic GP Fit
(c) Heteroscedastic GP Fit
Figure 2: Comparison of Homoscedastic and Heteroscedastic GP Fits to the Soil Phosphorus Fraction Dataset.
2 RELATED WORK
The most similar work to our own is that of [17] where experiments are reported on heteroscedastic
Branin-Hoo function using the variational heteroscedastic GP approach of [18] although to the best of our knowledge this work does not consider sequential evaluations. A modification to El, expected risk improvement introduced [19] and is applied to problems in robotics where robustness to aleatoric noise is desirable. [20, 21] implement heteroscedastic Bayesian Optimisation but don't tt introduce an acquisition function that penalises aleatoric uncertainty. [22, 23] consider the related problem of safe Bayesian Optimisation through implementing constraints in parameter space. In this instance, the goal of the algorithm is to enforce performance threshold for each evaluation of the black-box function and so is unrelated problem definition. In terms of acquisition functions,
2

---

Under review as a conference paper at ICLR 2019
[24, 25] propose principled approaches to handling aleatoric noise in the homoscedastic setting that could be extended to the heteroscedastic setting. Our primary focus in this work however, is to highlight that beteroscedasticity in the surrogate model is beneficial and so examination of a subset of acquisition functions is sufficient for this purpose.
BACKGROUND
Bayesian Optimisation features surrogate model for the black-box objective. The surrogate model should maintain calibrated uncertainty estimates in order to guide the acquisition of new data points. GPs are often popular choice of surrogate model in BO applications because of their ability to represent uncertainty.
3.1 GAUSSIAN PROCESSES
In the terminology of stochastic processes we may formally define GP as follows: Definition 1. A Gaussian Process is a collection of random variables, any finite number of which have a joint Gaussian distribution.
The random variables consist of function values fxx at different locations x within the design space. The GP is characterised by mean function
m(x) = [[](()]
(I)
and covariance function
k(x, xx) - m(x))).
(2)
The process is written as follows
f(x)
k(x,x)))
(3)
In our experiments, the prior mean function will be set to the data mean after normalisation of the inputs and standardisation of the outputs. As such, m(x) 0 will be assumed henceforth. The covariance function computes the pairwise covariance between two random variables (function values). The covariance between pair of output values f(x and f(x') is function of an input pair x and x'. As such, the kernel encodes smoothness assumptions about the latent function being modelled; similarity in input space yields outputs that are close [26]. The most videly-utilised kernel the squared exponential (SE) kernel
| x'
AssOe (ar, x²) oî exp
(4)
2
where of is the signal amplitude hyperparameter (vertical lengthscale) and & is the (horizontal) lengthscale hyperparameter. We use the squared exponential kernel all experiments. For further information on Gaussian Processes the reader is referred to [27].
3.2 BAYESIAN OPTIMISATION
Problem Statement
The global optimisation problem is defined
arg min fxx
(S)
where x" is the global optimiser of black-box function f X V. X' is the design space and typically a compact subset Rdd What makes this optimisation problem practically relevant in applications the following properties:
3

---

Under review as a conference paper at ICLR 2019
1. Black-box Objective: We do not have the analytic form of f. We can however evaluate pointwise anywhere in the design space X.
2. Expensive Evaluations: Choosing an input location x and evaluating f'x) takes a very long time.
3. Noise: The evaluation of given is a noisy process. In addition, this noise may vary across X. making the underlying process heteroscedastic.
We have a dataset of past observations D = ( 44) |== consisting of observations of the black-box function f and fit a surrogate model to these datapoints. We then leverage the predictive mean as well as the uncertainty estimates of the surrogate model to guide the acquisition of the next data point xn+1 according a heuristic known as an acquisition function.
HETEROSCEDASTIC BAYESIAN OPTIMISATION
We wish to perform Bayesian Optimisation whilst optimising input-dependent aleatoric noise. In order to represent input-dependent aleatoric noise, heteroscedastic surrogate model is required. We take the most likely heteroscedastic GP approach [28], adopting the notation presented there for consistency. We have dataset D {(x,, 4))_ in which the target values t; have been generated according to f = f(x,) + Gj- We assume independent Gaussian noise terms 6 ~ (0, o;) with variances given by a; r(x;). In the heteroscedastic setting typically non-constant function over the input domain x. In order to perform Bayesian Optimisation, we wish to model the predictive distribution P(t* xi..... x²) at the query points xi, X Placing a GP prior f and taking rxx) as the assumed noise rate function, the predictive distribution is multivariate Gaussian N N(p", Σ*)
with mean
1² = E K (K+
(6)
and covariance matrix
nd =ar[t"] = + R² K* (K + R)====*
where K E Raxn Kij = x(x;, K E Raxa Kij = k(x,, xz), K
Raxa
Kij k(x,, xj), t = (1,,12,..., 1,)²²² R diag(r) with (r(x)), rxx2))..., and R = diag(r") with r" =====(((x))) ((x)),....
The most likely heteroscedastic GP algorithm [28] executes the following steps:
1. Estimate a homoscedastic GP, G1 on the dataset D ((xx, [4)
2. Given G1, we estimate the empirical noise levels for the training data using varit;, G1 (x,, D) 0.5 (t, E[x])² forming new dataset D' [(x, 2)) A note the form of this variance estimator is give in Appendix B.
3. Estimate a second GP, G2 on D'.
4. Estimate a combined GP, G3 on D using G2 to predict the logarithmic noise levels r;.
5. If not converged, set G3 to G and repeat.
The Bayesian Optimisation problem may be framed as
x²
= arg min f(æ),
(8)
mEx
where the black-box objective f. to be minimised has the form
f(x) g(x)) s(a))
4

---

Under review as a conference paper at ICLR 2019
s(x) is, in this instance. the true noise rate function. We investigate extensions of the expected improvement [29] acquisition criterion, the principal form of which may be written in terms of the targets and the incumbent best objective function value, 77, found so far
(I( E (n 1)+ = (1
1)+ p(t m)) di
(10)
where p(t x)) the posterior predictive marginal density of the objective function evaluated at æ.
(1) - +)+ = max (0, -- t) is the improvement over the incumbent best objective function value Evaluations of objective are noisy all of the problems we consider and so we use expected improvement with plug-in in [30], the plug-in in value being the GP predictive mean [31].
We propose two extensions to the expected improvement criterion. The first is an extension of the augmented expected improvement criterion
(EI E ((1 +)+1
On
(1I)
//vartt + o
of [32] where is the fixed alcatoric noise level. We extend AEI to the heteroscedastic setting by
exchanging the fixed aleaotric noise level with the input-dependent one:
(1
[1-
/r(x)
het-AEl = =E t)+]
(12)
var(t" t r(())
where r(æ) is the predicted aleatoric uncertainty input under the most likely heteroscedastic GP and var[t*] the predictive variance the heteroscedastic GP incorporating both aleatoric and epistemic components of the uncertainty. We also propose simple modification to the pected improvement acquisition function that explicitly penalises regions the input space with large aleatoric uncertainty. We call this acquisition function aleatoric noise-penalised expected improvement (ANPEI) and denote
ANPEI aEI((x) vr(z))
(13)
where is scalarisation constant which
0.5 for the experiments in this paper. In the multiobjective optimisation setting particular value of a will correspond to a point on the Pareto frontier. We use both the modification AEI (het-AEl) and ANPEI acquisition function in conjunction with the most likely heteroscedastic GP surrogate model in the experiments that follow.
For the experiments on antifragility to alcatoric uncertainty we propose the antifragile augmented expected improvement (AAEI) acquisition function
A - [[([-))+] (1-
var(t*) + r r(x)
(14)
Vr(z)
which has the effect of promoting solutions with high aleatoric noise.
EXPERIMENTS ON ROBUSTNESS TO ALEATORIC UNCERTAINTY
5.1
IMPLEMENTATION
Experiments were run using a custom implementation of Gaussian Process regression and most likely heteroscedastic Gaussian Process regression. The code is available at ht // /anonymous.

---

Under review as a conference paper at ICLR 2019
tay turat
8
(a) ID Toy Objective
(b) Noise Rate Function
(c) Black-box Objective
Figure 3: Toy ID Problem. The toy objective in a) corrupted with heteroscedastic noise according to the function in b). The combined objective, which when optimised maximises the sin wave subject to the minimisation of aleatoric noise, is given in c) and is obtained by subtracting the noise function from the ID sinusoid.
4gpen.scceerre//33332200--- 0ttpae////233111111166---------- 9---991155--999772000001/1 The squared expo- nential kernel chosen the covariance function for both the homoscedastic GP well as G and G2 of the most likely heteroscedastic GP. The lengthscales, E, the homoscedastic GP were initialised 1.0 for each input dimension across all toy problems after standardisation of the output values following the recommendation
[33]. The signal amplitude af was initialised to value of 2.5. The lengthscale, f, of G2 of the most likely heteroscedastic GP [28] was initialised to 1.0, the initial noise level of G2 likely heteroscedastic GP was run for 10 iterations and the sample size required to construct the variance estimator producing the auxiliary dataset was 100. Hyperparameter values were obtained by optimising the marginal likelihood using the scipy implementation of the L-BFGS optimiser. The objective function in all cases the principal objective g(æ) minus one standard deviation of the ground truth noise function s(r).
set to 1.0. The EM-like procedure required to train the most
5.2 1D ToY OBJECTIVE WITH LINEAR NOISE RATE FUNCTION
Referring to Equation 9 from section 4, in the first experiment we take one-dimensional sin wave
g(x) = sin(x) 0x)())
(15)
with noise rate function s(z) 0.25x. These functions as well as the black-box objective f(x) are shown in Figure The Bayesian Optimisation problem designed such that the first maximum in 3(a) is to be preferred as samples from this region of the input space will have smaller noise rate. The black-box objective in 3(c) illustrates this trade-off. In 6(a) we compare the performance of Bayesian Optimisation scheme involving vanilla GP in conjunction with the El acquisition function with the most likely heteroscedastic GP in conjunction with the ANPEI and het-AEl acquisition functions. The experiment is designed to contrast the performance of a standard Bayesian Optimisation scheme against our approach in a situation where minimising aleatoric noise is desirable.
5.3 BRANIN- HHoO WITH NON-LINEAR NOISE RATE FUNCTION
In the second experiment consider the Branin-Hoo function g(æ) with a non-linear noise rate function given by s(x) = 1.4z, + 0.3z2. Given that this example is minimisation problem, the black-bor objective consists of the sum of the Branin-Hoe Hoo function and the noise rate function. Contour plots of the functions are shown in Figure 4. A comparison, in terms of the best objective function values found, between the vanilla GP and El acquisition function with the most likely heteroscedastic GP and ANPEI and AEI acquisition functions is given in 6(b).
5.4 OPTIMISING THE PHOSPHORUS FRACTION OF SOIT.
In this real-world problem we apply heteroscedastic Bayesian optimisation to the problem of optimis- ing the phosphorus fraction of soil. Soil phosphorus is essential nutrient for plant growth and is

---

Under review as a conference paper at ICLR 2019
(a) 2D Branin-Hoo Function
(b) Non-linear Noise Function
(c) Black-box Objective
Figure 4: Toy 2D Problem. The Branin-Hoo objective function a) corrupted by the heteroscedastic noise function in b) s(21, 222 1.4x, + 0.322 The black-box objective function is obtained by summing the functions in a) and b). The sum required to penalise regions of large alcatoric noise because the objective is being minimised.
widely used as a fertilizer in agriculture. While the amount of arable land worldwide is declining, global population is expanding and so is food demand. As such, understanding the availability of plant nutrients that increase crop yield is essential. To this end, [34] have curated dataset on soil phosphorus, relating phosphorus content to variables including soil particle size, total nitrogen, organic carbon and bulk density. this experiment, we study the relationship between bulk soil density and the phosphorus fraction, the goal being minimise the phosphorus content of soil subject to heteroscedastic noise. We provide evidence that there is heteroscedasticity in the problem by comparing the fits of homoscedastic GP and the most likely heteroscedastic GP in Figure 2 and provide a predictive performance comparison based on negative log predictive density values in the appendix. In this problem, do have access to a continuous-valued black-box function ground truth noise function. As such, the surrogate models were initialised with subset of the data and the query locations selected Bayesian Optimisation were mapped to the closest data points in the heldout data. The following kernel smoothing procedure was used to generate pseudo ground truth noise values:
1. Fit a homoscedastic GP the full dataset.
2. At each point xz, compute the corresponding square error 8 = (y: p(x;))2
3. Estimate variances by computing moving average of the squared errors, where the relative weight of each sf was assigned with Gaussian kernel.
The performances of homoscedastic Bayesian Optimisation using El and AEI and heteroscedastic Bayesian Optimisation using ANPEI and AEI are compared in 5(c).
5.5 DISCUSSION
In all robustness experiments, the most likely heteroscedastic GP and ANPEI combina- tion/heteroscedastic GP and het-AEl combination outperform the homoscedastic GP and EI. The fact that the homoscedastic GP has no knowledge of the heteroscedasticity of the noise rate function puts serious disadvantage. In the first sin wave problem, designed to highlight this point, the heteroscedastic Bayesian Optimisation scheme consistently and preferentially finds the first maximum as that which minimises aleatoric noise. In contrast the homoscedastic GP, finds it impossible to differentiate between the two maxima. The experiments provide strong evidence that modelling heteroscedasticity Bayesian Optimisation is more flexible approach assuming homoscedastic noise.
EXPERIMENTS ON ANTIFRAGILITY TO ALEATORIC UNCERTAINTY
6.1 IMPLEMENTATION
The implementational details remain the same as for the robustness experiments save for the fact that the acquisition functions have changed such to promote antifragility to aleatoric uncertainty.
7

---

Under review as a conference paper at ICLR 2019
Dipestvv Fandles
Eer: Oijectidd andten fund
0.7
(a) ID Toy Objective
(b) 2D Branin-Hoo
(c) Soil Phosphorus Fraction
Figure 5: Results of heteroscedastic and homoscedastic Bayesian Optimisation on the 3 robustness problems considered. Error bars are computed using 10 random initialisations. The first problem is maximisation problem whereas the second and third are minimisation problems.
1.00
Cheetive Va
0.25
levosendeass
resersoole alc AEFE
-025
levooradadt
cootscollats als ANF
-10
Nunber of pac
(a) ID Toy Objective
(b) 2D Branin-Hoo
Figure 6: Results of heteroscedastic and bomoscedastic Bayesian Optimisation on the sin and Branin toy problems where antifragility to aleatoric uncertainty is desirable. Error bars are computed using 10 random initialisations. The first problem is a maximisation problem whereas the second is minimisation problem. Heteroscedastic AEI in this instance is the AAEI acquisition function.
This was achieved by changing the sign in the expression for ANPEI and using the AAEI acquisition function in place of AeI- The baseline homoscedastic GP and EI combination remains the same. The homoscedastic GP is incapable of representing input-dependent aleatoric uncertainty and so unable to incorporate information about this quantity in any acquisition function that is used in conjunction with it.
6.2 DISCUSSION
The results of Figure demonstrate that heteroscedastic Bayesian Optimisation is capable of seeking solutions possessing high aleatoric noise, a desirable property in materials discovery applications. The scalarisation heuristic used in the modified ANPEI acquisition function outperforms the AAEI acquisition function on the toy sin function example suggesting that this approach may be both valid and beneficial in its simplicity in certain scenarios.
7 CONCLUSION AND FUTURE WORK
We have presented an approach for performing Bayesian Optimisation with the explicit goal of optimising aleatoric uncertainty in the suggestions. We posit that such an approach can prove useful for the natural sciences in the search for molecules and materials that are robust and antifragile to experimental noise and future work we plan to apply our approach to molecular property optimisation [4]. We demonstrate concrete improvements on one and two-dimensional toy problems as well as a real-world optimisation problem and contribute an open-source implementation of the most likely heteroscedastic GP surrogate model for Bayesian Optimisation.
8

---

Under review as a conference paper at ICLR 2019
REFERENCES
[1] Rafael
Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lohaar Benjamín Sánchez-Lengelingg Dennis Sheberla, Jorge gguilera-parrrguinne Timothy D Hirzel, Ryan Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules, ACS Central Science, 4(2):688-77,, 2018.
[2] Ryan-Rhye Griffiths and José Miguel Hemánnezz-Lobato. Constrained Bayesian optimization for automatic chemical design. arXiv preprint arXiv:1709.05500 2017.
[3] Omar Mahmood and José Miguel Hernández-Loband cold approach to generating optimal samples. arXiv preprint arXiv: 199990009885.. 2019.
[4] Ksemia Korovina, Sailun Xu, Kirthevasan Kandasamy, Willie Neiswanger, Barnabas Poczos, Jeff Schneider, and Eric P Xing. Chembo: Bayesian optimization of small organic molecules with synthesizable recommendations. arXiv preprint arXiv: 0rXiv:/908.00425, 2019.
[5] Tristan Aumentado--Armstroong. Latent molecular optimization for targeted therapeutic design. arXiv preprint arXiv: arXiv:
2018
[6] Jialin Song, Yury S. Tokpanov, Yuxin Chen, Dagny Fleischman, Kate T. Fountaine, Harry A. Atwater, and Yisong Yue. Optimizing Photonic Nanostructures via Multi-fidelity Gaussian Processes. 18110077 [cs, stat), November 2018.
[7] Jialin Song, Yuxin Chen, and Yisong Yue. A general framework for multi-fidelity bayesian optimization with gaussian processes. The 22nd International Conference on Artificial Intelligence and Statistics, pages 3158-3167, 2019.
[8] Henry C Herbol, Weici Hu, Peter Frazier, Paulette Clancy, and Matthias Poloczek. Efficient search of compositional space for hybrid aggamic-inorganie perovskites via bayesian optimiza- tion. npj Computational Materials, 4(1):51, 2018,018, 8.
[9] Roberto Calandra, André Seyfarth, Jan Peters, and Marc Peter Deisenroth. Bayesian optimiza- tion for learning gaits under uncertainty. Annals of Mathematics and Artificial Intelligence, 76 (1-2):5-23, 2016.
[10] James Grant, Alexis Boukouvalas, Ryan-Rhy Griffiths, David Leslie, Sattar Vakili, and En- rique Munoz De Cote. Adaptive sensor placement for continuous spaces. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Ma- chine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2385-2393, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
[11] Simon Olofsson, Mohammad Mehrian, Roberto Calandra, Liesbet Geris, Marc Peter Deisenroth, and Ruth Misener. Bayesian multiobjective optimisation with mixed analytical and black-box functions: Application to tissue engineering. IEEE Transactions on Biomedical Engineering, 66(3):727-739, 2018.
[12] Guilherme Duarte Ramos Matos, Daisy Y Kyu, Hannes H Loeffler, John D Chodera, Michael R Shirts, and David Mobley. Approaches for calculating solvation free energies and enthalpies demonstrated with update of the freesolv database. Journal of Chemical & Engineering Data,
2017.
[13] Ryan-Rhys Griffiths, Philippe Schwaller, and Alpha A. Lee. Dataset bias in the natural sciences: A case study in chemical reaction prediction and synthesis design. ChemRxiv, 2018.
[14] José Miguel Hernandez-Lobato, James Requeima, Edward O Pyzer-Knapp, and Alán Aspuru- Guzik. Parallel and distributed thompson sampling for large-scale accelerated exploration of chemical space. In International Conference on Machine Learning, pages 1470-1479, 2017.
[15] Nassim Nicholas Taleb, Antifragile: Things That Gain Disorder. Random House, New York, 1st ed edition, 2012. ISBN 978-1-4000-66882-4
[16] Yuanyuan Zhou and Yixin Zhao. Chemical stability and instability of inorganic halide per- ovskites. Energy & Environmental Science, 12(5):1495-1511 May 2019. ISSN 1754-5706. doi: 10.1039/C3FF003559H.

---

Under review as a conference paper at ICLR 2019
[17] Roberto Calandra. Bayesian Modeling for Optimization and Control in Robotics. PhD thesis, Technische Universität, 2017.
[18] Miguel Lázaro-Gredilla and Michalis K Titsias. Variational heteroscedastic Gaussian process regression. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pages 841-848. Omnipress, 2011.
[19] Scott R Kuindersma, Roderic A Grupen, and Andrew G Barto. Variable risk control via stochastic optimization. The International Journal of Robotics Research, 32(7):806 825, 2013.
[20] John-Alexander M Assael, Ziyu Wang, Bobak Shahriari, and Nando de Freitas. Heteroscedastic treed Bayesian optimisation. arXiv preprint arXiv: 1400.7172, 2014.
[21] Ryo Ariizumi, Matthew Tesch, Howie Choset, and Fumitoshi Matsuno. Expensive multiobjec- tive optimization for robotics with consideration of heteroscedastic noise. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2230-2235. IEEE, 2014.
[22] Yanan Sui, Alkis Gotovos, Joel Burdick, and Andreas Krause. Safe exploration for optimization with gaussian processes. In International Conference on Machine Learning, pages 997-1005, 2015.
[23] Felix Berkenkamp, Andreas Krause, and Angela P. Schoellig. Bayesian optimization with safety constraints: Safe and automatic parameter tuning in robotics. ArXiv, abs/1 2016.
[24] Peter Frazier, Warren Powell, and Savas Dayanik. The knowledge-gradient policy for correlated normal beliefs. INFORMS journal on Computing, 21(4):599-613, 2009.
[25] Benjamin Letham, Brian Karrer, Guilherme Ottoni, Eytan Bakshy, et al. Constrained bayesian optimization with noisy experiments. Bayesian Analysis, 14(2):495-519, 2019.
[26] Pedro M Domingos. A few useful things to know about machine learning. Commun. acm, 55 (10):78-87, 2012.
[27] C. E. Rasmussen and C. K. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
[28] Kristian Kersting, Christian Plagemann, Patrick Pfaff, and Wolfram Burgard. Most likely het- eroscedastic Gaussian process regression. In Proceedings the the 24th International Conference on Machine Leurning- ICML '07, pages 393-400) Corvalis, Oregon, 2007. ACM Press. ISBN 978-1-59593-793-3. doi: 10. 1ell115//27349.... 1273546.
[29] Donald R Jones, Matthias Schonlau, and William Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455-4g) 492, 1998.
[30] Victor Picheny, Tobias Wagner, and David Ginsbourger. A benchmark of kriging-based infill criteria for noisy optimization. Structural and Multidisciplinary Optimization, 48(3)x607-626, 2013.
[31] Emmanuel Vazquez. Julien Villemonteix, Maryan Sidorkiewicz, and Eric Walter. Global optimization based noisy evaluations: an empirical study of two statistical approaches. In Journal of Physics: Conference Series, volume 135, page 012100. IOP Publishing. 2008.
[32] Deng Huang, Theodore T Allen, William Notz, and Ning Zeng. Global optimization of stochastic black-box systems via sequential kriging meta-models. Journal of global optimization, 34(3):441-466 2006.
[33] Iain Murray. Introduction to gaussian processes.
[34] Enqing Hou, Xiang Tan, Marijke Heenan, and Dazhi Wen. A global dataset of plant available and unavailable phosphorus in natural soils derived by Hedley method. Scientific Data, 5(1): 180166, December 2018. ISSN 2052-4463. doi: 10.1038/sdata.201 2018. 166.
10

---

Under review as a conference paper at ICLR 2019
A HETEROSCEDASTICCTY OF THE SOIL PHOSPHORUS FRACTION DATASET
Table 1 used to demonstrate the efficacy of modelling the soil phosphorus fraction dataset using heteroscedastic GP. The heteroscedastic GP outperforms the homoscedastic GP on prediction based on the metric of negative log predictive density (NLPD)
NLPD Σ
log p(4;
(16)
which penalises both over and under-confident predictions.
Table I: Comparison of NLPD values on the soil phosphorus fraction dataset. Standard errors are reported for 10 independent train/test splits.
Soil Phosphorus Fraction Dataset
GP
Het GP
NLPD
1.35 + 1.33
1.00 + 0.95


Reviews:
[{"review_id": "rJghy_26Kr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY OF REVIEW\n\nThis paper discusses an interesting problem of BO in the cases of robustness and antifragility to aleatoric noise/uncertainty. To tackle this problem, the authors replace the conventional homoscedastic GP model with a heteroscedastic GP model. In the case of robustness to aleatoric noise/uncertainty, the authors have modified EI by simply either scaling down [32] or subtracting from its value more when the aleatoric noise increases. In the case of antifragility to aleatoric noise/uncertainty, they do the opposite.\n\nThe modifications of EI to handle robustness and antifragility to aleatoric noise/uncertainty are simple and straightforward, one of which is similar to the augmented EI of [32].\n\nThere are some technical ambiguities, as detailed below. In particular, the choice of objective function (equation 9) for this problem needs to be justified and motivated by the practical applications described in Section 1.\n\nSince no convergence guarantee is given, a more extensive empirical analysis with real datasets needs to be provided to better understand the performance and behavior of the proposed BO algorithms. In particular, though the authors have motivated their problem using the compelling applications of materials and drug discovery, no experimental result for these applications has been provided to support the motivation of this work.\n\n\n\nDETAILED COMMENTS\n\nThe authors seem to motivate the significance of their problem of interest through the key applications of materials and drug discovery which I can appreciate. Unfortunately, experimental results in such applications were not available in this paper to \"close the loop\" in supporting the motivation of this work, begging the question whether their proposed BO algorithm indeed works for these key applications. For example, why is the FreeSolv hydration energy dataset not used for your experiments?\n\n\nFor Fig. 1, how exactly do you extract the error magnitudes from the FreeSolv hydration energy dataset? How do you exactly define the notion of calculated vs. experimental uncertainties? Fig. 1 shows that the noise peaks with a relatively high frequency at a single error magnitude value. Would the assumption of homoscedastic noise at this peaked value be detrimental to BO? A sensitivity analysis would be useful here.\n\n\nFor the soil phosphorus fraction dataset (Fig. 2), the skewed distribution of the measurements (few extremely large measurements and many small-valued measurements) may not be due to heteroscedastic aleatoric uncertainty. In fact, in the literature of earth/environmental science, such a dataset is often modeled using a log-Gaussian process (or log-normal kriging), that is, the log-measurements follow a GP:\n\nWebster, R., and Oliver, M. 2007. Geostatistics for Environmental Scientists. John Wiley & Sons, 2nd edition.\n\nCan the authors provide supporting evidence (in the form of references) that such a dataset is due to heteroscedastic aleatoric uncertainty?\n\n\n\nOn page 4, step 2 of the most likely heteroscedastic GP algorithm [28] cannot be understood: What is E[x]? Isn't G_1 a GP? Why is it able to accept x_i and D as inputs? How is z_i defined?\n\nThe authors say that \"A note on the form of this variance estimator is give in Appendix B.\" There is no Appendix B.\n\n\nCan the authors give a detailed discussion why is the expression of f(x) = g(x) + s(x) in equation 9 the right one to be minimized in practice (e.g., in the context of materials and drug discovery)? For example, do material scientists use such an objective function? Provide references. Furthermore, why is the same equation 9 being minimized for both the cases of robustness and antifragility to aleatoric uncertainty?\n\nCaption of Fig. 3: I can't understand the sentence: \"The combined objective, which when optimised maximises the sin wave subject to the minimisation of aleatoric noise\". This was repeated in Section 5.5: \"finds the first maximum as that which minimises aleatoric noise\". Isn't the minimum of the aleatoric noise at the origin (Fig. 3b)?\n\nCan the authors provide information on how much initial data was provided prior to running BO? How much data is used for learning the GP hyperparameters?\n\nThe performance advantage of heteroscedastic ANPEI over homoscedastic does not appear to be significant for Branin-Hoo function (Figs. 5b and 6b). Can the authors explain this?\n\n\n\nMinor issues\n\nThere are two different font types of x in bold.\n\nPage 5: fixed aleaotric\n\nWhich phrase is correct? \"is obtained by subtracting the noise function from the 1D sinusoid\" in the caption of Fig. 3 or \"The objective function in all cases is the principal objective g(x) minus one standard deviation of the ground truth noise function s(x)\"?\n\nFig. 5b: Shouldn't the vertical axis be labeled as ...+ Noise?\n\nFig. 6a: Shouldn't the vertical axis be labeled as ...- Noise?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SylAl7jVKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers the heterogeneous noise in Bayesian optimisation. The paper utilised the existing heterogeneous Gaussian process to model the surrogate function and proposes two acquisition functions from the Expected improvement to deal with such heterogeneous noise.\n\nThe proposed acquisition function is heuristic and straightforward from the existing ones, ie. augmented EI and EI. I would not count this as much in terms of novelty.\n\nThe idea of dealing with heterogeneous noise in BO is interesting.\n\nThe related background in Gaussian process is standard and could be omitted. \n\nThe experimental section is weak. The experiments are demonstrated using low dimensional functions (1-2 dim).\n\nThe experiment needs to compare with Random baseline. Under the noise setting, the Random approach performs very competitive to BO. \nThe y-axis in the experiment, the paper has considered the objective function value + noise. The reviewer suspects that the high/low performance is due to the high level of noise (?) rather than the better objective function value.\nThe released source code includes the Lidar, Scallop, Silverman datasets, but not the PHOSPHORUS soil. I would encourage the authors to check the source code before releasing.\nThe paper focuses on demonstrating the heteroscedasticity in the surrogate model using [1]. The reviewer is wondering what is the performance for other heteroscedastic GP approaches, such as [2,3,4] ?\n\nMinor points:\nPage 4, step number 2, what is $z_i$ and how can we get it in a new dataset D’.\n\n\n[1] Kersting, Kristian, et al. \"Most likely heteroscedastic Gaussian process regression.\" Proceedings of the 24th international conference on Machine learning. ACM, 2007.\n[2] Le, Quoc V., Alex J. Smola, and Stéphane Canu. \"Heteroscedastic Gaussian process regression.\" Proceedings of the 22nd international conference on Machine learning. ACM, 2005.\n[3] Binois, M., Gramacy, R. B., & Ludkovski, M. (2018). Practical heteroscedastic gaussian process modeling for large simulation experiments. Journal of Computational and Graphical Statistics, 27(4), 808-821.\n[4] Lázaro-Gredilla, M., & Titsias, M. K. (2011, June). Variational Heteroscedastic Gaussian Process Regression. In ICML (pp. 841-848).\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJeQfDipYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main contribution of this paper are:\n\n1. The use of a heteroscedastic GP when performing Bayesian Optimization, this is in contrast to the more common practice of assuming homoscedastic noise, even when this does not quite fit the data. They use the existing algorithm called most likely heteroscedastic GP, and quote previous work that performed BO using different heteroscedastic GP implementations.\n2. They introduce two new acquisition functions that incorporate the predicted observation noise, either making candidates more likely or less likely to be chosen when predicted noise is higher, depending on the requirements. This are fairly minor extensions/heuristics if taken on their own, as they do not provide a very strong motivation indicating why these acquisition functions are useful or better than existing ones, other than that they take heteroscedasticity into account.\n3. They run a set of experiments on the above settings. Unfortunately the experiments are very limited, and their method does not improve on the baselines in a statistically significant way. Two of the experiments are on simple synthetic settings, the third approximates a real world setting, although the approximation is quite rough and they don't convincingly argue for it being realistic, neither do they give convincing motivation of their objective which uses g +- standard deviation.\n4. They provide source code of their implementation.\n\nThe paper is easy to understand, and covers an interesting topic, so while I don't think it meets the bar of ICLR (due to lack of convincing and non-trivial contributions) I think it could perhaps be made into a workshop submission with some of the following changes:\n* A wider set of experimental settings, and more replication such that any differences become statistically significant. It would also be worthwhile comparing to random search.\n* A better justification of the objective used in the experiments, using g +- the standard deviation appears fairly arbitrary, and there is no strong enough reason to believe this is a good approximation of what the cost is in the case of real world problems.\n* Better theoretical justification of the acquisition function; one option is to introduce more principled acquisition function like, say, expected upper/lower bound.\n\n\nOther notes/comments:\nabstract: as well as a real-world -> as well as *on* a real-world...\nsection 1 \"As a case study\" -> not very clear what this means\nincumbent best is not well defined, is it the empirical value of f used or the mean predicted f on the evaluated candidates?\nsection 6.2 \"outperforms\" -> this is not clear if one looks at the confidence intervals in the results\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers initially gave scores of 1,1,3 citing primarily weak empirical results and a lack of theoretical justification.  The experiments are presented on synthetic examples, which is a great start but the reviewers found that this doesn't give strong enough evidence that the methods developed in the paper would work well in practice.  The authors did not submit an author response to the reviewers and as such the scores did not change during discussion.  This paper would be significantly strengthened with the addition of experiments on actual problems e.g. related to drug discovery which is the motivation in the paper.
Decision: Reject