Respecting the limit: Bayesian optimization with a bound on the optimal value
Authors: Hanyang Wang
Juergen Branke
Matthias Poloczek
Venue: Accepted by TMLR ()

Abstract: In many real-world optimization problems, we have prior information about what objective function values are achievable. In this paper, we study the scenario that we have either exact knowledge of the minimum value or a, possibly inexact, lower bound on its value. We propose bound-aware Bayesian optimization (BABO), a Bayesian optimization method that uses a new surrogate model and acquisition function to utilize such prior information. We present SlogGP, a new surrogate model that incorporates bound information and adapts the Expected Improvement (EI) acquisition function accordingly. Empirical results on a variety of benchmarks demonstrate the benefit of taking prior information about the optimal value into account, and that the proposed approach significantly outperforms existing techniques. Furthermore, we notice that even in the absence of prior information on the bound, the proposed SlogGP surrogate model still performs better than the standard GP model in most cases, which we explain by its larger expressiveness.

Published in Transactions on Machine Learning Research (03/2025)
Respecting the limit:
Bayesian optimization with a bound on the optimal value
Hanyang Wang
Hanyang. Wong@warrwick.cc.kk uk
Werwick Mathematics Institute
University of Warwick Coventry, CV4 7AL, United Kingdom
Juergen Branke
Juergen.
Auergen.Brankee Branke Bwbs. uk
Werwick Business School
University of Warwick
Coventry, CV4 7AL, United Kingdom
Matthias Poloczek
matpol&aamazo..co
Amazon
San Francisco, CA 94105, USA
Reviewed on OpenReview: https: // openrevvee.e....... net/ forum? id=y5Hf0ot.LLA
Abstract
In many real-world optimization problems, we have prior information about what objective function values are achievable. In this paper, we study the scenario that we have either exact. knowledge of the minimum value or a, possibly inexact, lower bound on value. We propose bound-aware Bayesian optimization (BABO), Bayesian optimization method that surrogate model and acquisition function to utilize such prior information. We present SlogGP, new surrogate model that incorporates bound information and adapts the Expected Improvement (EI) acquisition function accordingly. Empirical results on a variety of benchmarks demonstrate the benefit. of taking prior information about the optimal value into account, and that the proposed approach significantly outperforms existing techniques. Furthermore, we notice that even the absence of prior information on the bound, the proposed SlogGP surrogate model still performs better than the standard GP model in most cases, which explain by its larger expressiveness.
1 Introduction
For many real-world black-box optimization problems, evaluating solution can be computationally expen- sive, and optimization algorithms thus need to be sample efficient. Bayesian optimization (BO) is global
optimization method suitable for such problems (Brochu et al.) 2010 Shahriari et al. 2015 Garnett 2028
Due to its data efficiency, is widely used in many areas, such as protein design (Stanton et al. 2022), chem-
istry Folch et al. 2022). robotics design (Calandra al. 2016) and hyperparameter tuning (Cho et al. 2020
While BO typically assumes the objective function f(x) to be black-box, in some real-world applications, additional information about the achievable optimal value is available. For example, in hyperparameter tun- ing, the error rate of a model is always larger than or equal to 0%. In physical experiments, the temperature must be greater than or equal to 1 and the electric resistance must be greater than or equal to Q. Any heat engine, the efficiency cannot excced the Carnot efficiency, which depends on the temperatures of the heat source and sink. The upper bound of data rate in communication channel is given by the Hartley theorem, which defines the maximum capacity based bandwidth and signal-to-noise

---

Published in Transactions on Machine Learning Research (03/2025)
ratio. Latency cannot be reduced below the physical propagation time of signal. And in acrodynamics the drag cannot be reduced below zero because drag represents resistance.
Intuitively, exploiting such information should be helpful. The first BO paper that takes such information into account is Hutter et al. (2009), who point out that a log transformation of positive functions is usually beneficial. Other recent. works that may take output bound information into consideration are Nguyen & Osborne (2020]: Wang et al. (2018); Nguyen et al. (2021); ; Jeong & Kim (2021).
In this paper, we focus on BO for minimization with known lower bound f on the unknown optimal function value f", i.e., f* 2 f We propose new algorithm, bound-aware Bayesian optimization (BABO), which makes use of this lower bound information to improve efficiency in BO. BABO is based on & novel surrogate model, Shifted Logarithmic Gaussian Process, or SlogGP, which can take into account prior information on a lower bound of the objective function. SlogGP is defined as f(x) = eω(x) -S, where g(x) is a GP and Ç is a learnable parameter. As the resulting predictive distribution is no longer normal, we adapt the well-known expected improvement (EI) acquisition function to SlogGP, resulting in SlogEI. We then show how SlogEI can be further modified to also take into account the information the lower bound of the objective function. We call this new acquisition function Shifted Logarithmic Truncated Expected Improvement (SlogTEI). Combining SlogGP and SlogTEL, we obtain BABO.
We find that SlogGP with SlogEI can outperform a standard GP with EI in BO even if there is no information on the bounds of the objective function, primarily due to its larger expressiveness, where expressiveness refers to the ability of a model to represent a wide range of functions or patterns in the data. Incorporating lower bound knowledge into SlogGP and using Slog TEI (BABO) can further enhance its performance. We evaluate the proposed framework on several synthetic functions well as three real-world applications. Empirical results demonstrate that our new method outperforms conventional BO and other algorithms designed for the case of a known lower bound.
The structure of this paper is as follows. Section 2 gives an introduction to Bayesian optimization and surveys the literature on BO with lower bound information. In Section 3, we introduce the SlogGP model as well as the unknown-bound case. SlogGP
and the corresponding acquisition function in the and SlogTEI together form BABO, our bound aware Bayesian optimization. We also prove that SlogGP is more flexible than GP, which can explain the superior performance of SlogGP over GP even in the absence of lower bound information. Section 4 reports on the experimental results, demonstrating the empirical advantage of using BABO when a lower bound is known. The paper concludes with summary and some avenucs for future work.
2 Preliminaries
2.1
Bayesian Optimization
Bayesian optimization (BO) is a sequent strategy for global optimization of black-box functions. Given an objective function f(x) and the feasible set X, the goal of BO is to find an optimal solution x* e argminnexx/(x (x).
BO consists of two main steps. The first step is to build a surrogate model based on historical observations ((x1, .... (KN,yN)) A common choice for the surrogate model is a GP, though other models have been proposed, including random forest (Hutter et al. 2011), deep neural network (Snoek et al., 2015) and Mondrian trees (Wang et al.) 2018). For more information Gaussian processes, we refer to Rasmussen & Williams 2006) and Schulz et al. (2018).
The second step involves using the surrogate model for selecting the solution to be evaluated next, balancing the mean and variance predicted by the surrogate model, which is known as the apploration-exploitatioo tradc-off. In BO, this balance is achieved by optimizing a so-called acquisition function a. A common choice is Expected Improvement (EI) (Mockus 1998). The information from the newly evaluated solution then added to the set of observations and the surrogate model is updated before the next iteration.
2

---

Published in Transactions on Machine Learning Research (03/2025)
2.2 Acquisition functions for f f
If we have prior information that the global minimum f == min, f(x) is at least fh, then we can try to adjust the acquisition function to make use of the information. For Max-value Entropy Search (MES: Wang & Jegelka (2017) this is straightforward and has already been used as baseline in Nguyen & Osborne
(2020)
and Wang et al. 2018). MES selects the next solution to evaluate where expects the largest reduction in entropy of the predicted distribution of the optimal objective value. It does not assume prior knowledge about the optimal value, but uses Gumbel Sampling to sample a realization of f from the posterior. However, according to Wann et al. (2018). given lower bound fo on the minimum value f", we may calculate MES
as MES" (x (9) = (x. [( log & (7 (x, [")), where 7 (x) = -(x)=/" o(.) and ((-) are the 20(7(x, ()))
o(x)
PDF and CDF of the standard normal distribution. Note that this analytical expression for MES does not require Monte Carlo sampling. In addition, we show in Appendix A.1 that MES' shares the same maximizer
with the acquisition function [(f(x) < f²), i.e. argmaxxxx MES (x fa) = argmaxxex* PPf(( < ft).
2.3 Models for f" 2 l
Besides acquisition functions, we can attempt to incorporate the information into surrogate models.
Hutter al. 22009 2011 find that employing log-transformmtion as a preprocessing step for positive-valued functions, combined with correspondingly adapted EI acquisition function, can enhance the performance of BO. When shifting the function to be positive given the information about the lower bound, this is similar to our SlogGP model with a fixed rather than learnable Ç, and we will later show that it performs significantly worse than our model. [Joon & Kim (021) propose the objective bound conditional Gaussian process (OBCGP). OBCGP introduces a parameter XM: and conducts Gaussian process regression conditioned on (xM: f(xx)) through variational inference. In cases where the lower bound f is known, we can enforce the distribution of f(xxr to adhere this known bound. Nguyen & Osborne (2200) focus the case when the exact value of the optimum f* is known, which can be viewed as special case of known bound. A parabolic Gaussian process model [x) f (Gunter et al. 2014)) is used. To guarantee an analytical form of the posterior distribution, a linearization is applied, so the final model is - -)))(x)) + (x(x)g(x) - f It should be noted that while the range of the parabolic Gaussian process is [f)),o), after linearization, the model becomes a GP and samples no longer adhere to the bound. Second, the transformation of the GP inflates the predictive uncertainty at points with low predicted mean, which causes sampling of EI and UCB to be too greody. They therefore propose two new acquisition functions: Confidence Bound Minimization (CBM) and Expected Regret Mininization (ERM). Finally, the acquisition functions can only be used with light bound but not with a more general bound, though the latter is more common. To solve this last issue, Nguyen et al., (2021) extend the model of Gunter et al. (2014) to a case where only a probability distribution for the lower bound of the objective function is known, and propose new acquisition function called Bounded Entropy Search (BES). However, the other issues of linearly approximating the parabolic Gaussian processes remain.
Another candidate surrogate for the case of a known lower bound fo would be a non-negativity GP (Pen- soncault et al. 2020). This method uses a GP as its model but introduces constraints to the hyperparameter tuning process, en enforcing the probability of each f(x) crossing the known bound to be below 5%. However, this method faces couple of challenges. First, it primarily suitable for low-dimensional problems due to its computational cost. Second, the range of the non-negativity GP remains (-oo, oo), which means that there is still mismatch with the desired range [f', o0) of feasible objective values.
The primary issuc of the above-mentioncl models is their inability to properly model that the function f is known to lake values in the range [[², oo). Jensen et al. (2013) proposed regression models using truncated distribution (GP-TG) or Beta distribution (GP-BE) that have the desired range but lack an analytical form for posterior inference. They use Laplace approximation for the surrogate model. Additionally, Monte Carlo sampling would be needed for the acquisition function if this model were used in a BO framework. The hod presented in this paper has closed form and thus does not require costly approximations.
3

---

Published in Transactions on Machine Learning Research (03/2025)
3 Bound-aware Bayesian optimization
In this section, we present the Bound-aware Bayesian optimizer (BABO) and its two key components: the surrogate model, Shijted Logarillemic Gaussion Process (SlogGP) that can leverage lower bound s about the global minimum f if available, and the corresponding acquisition function Shifted Logarithmic Truncated Expected Improvement TEI)
3.1
The SlogGP Surrogate Model
The SlogGP model for an objective function f(.) is:
f(x) co(x) -s:
where g(x) is GP and is the shift, parameter that is learned from data during model fitting. The mean of f() is +o(x)+ - and its variance is (xx 1)e2m(x)++*(x)) where p(x) and o²(x) are the mean and variance of g(x).
This model can also be expressed In(f(x) + +) 9(x). When training the model, set the mcan of
(() to be the mean of the warped observed function values, i.. the mean of g(x) is E lo(w+<) and = yN]" is the observations.
The model can be viewed as a type of warped Gaussian process (Snelson et al. 2003 so we can learn the hyperparameters and parameters the model, in particular the hyperparameters of the covariance function and shift Ç by minimizing the negative log likelihood with respect to y Wi....,, UNl":
N
In(det K) +)+ 1 ( W( Σ
)
NCC = 2
In
2 In(2x),
N
Vi +
2=1
where is the covariance matrix and Ww)) ln Σ In(w++()
The SlogGP model f(x) = eo(x) can viewed as combining aspects of a parabolic GP S(x) = 9 - Ru et al. 2018 Nguyen & Osborne 2020) and a log- trg-fransformed GP f(x) = eg(x) Hutter et al. 2009 2011h. By integrating these two methods, the SlogGP model offers several advantages.
Gunter et al. 2014
Specifically, in comparison to log-transformed GPs, SlogGPs can be utilized without requiring any lower bound information and exhibit greater expressiveness due the learnable shift parameter Ç. When compared to parabolic GPs, SlogGPs do not require any approximation and can guarantee an analytical form of the acquisition function (see Section 3.3). More importantly, demonstrated in Section 3.2, SlogGPs possess greater expressiveness than standard GPs, given limited observation, a property that the other two models lack.
To the best of our knowledge, it is the first time that this model proposed for handling known lower bound conditions. Additionally, we find that SlogGP-bassed BО outperforms Ga-baaed BO even without any bound information, which we attribute to the enhanced expressiveness of SlogGPs.
3.2 Properties of the SlogGP Model
We begin with the case that we do not have a lower bound fo 2 fo and show that SlogGP is more general than a standard GP. In fact, as Theorem 3.1 shows, a SlogGP is reduced to a standard GP under certain conditions.
Note that a covariance function K can be written as K(x1, oz k(xx, x216g), where og is scaling hyperparameter called signal variance. Og are other hyperparameterss that independent of ag For noiscless GP, the covariance between x2 and x2 is Cou xx) K (x1,
Theorem 3.1. Define SlogGP f(x) {x(x)) - with g(x) a Gaussion process with zero mean, zero noise and covariance function K with signal variance of and other hyperparameters 0g-
4

---

Published in Transactions on Machine Learning Research (03/2025)
Then for any GP h(x) with mean Pg. signal variance gg, and other hyperparameters gg. have the SlogGP f(x) converge to h(x) in distrduutinn for any X by selling
Ng In (5 + igg
0 = 0g
0g 5+ + Pig
and letting - 00, i.e.,
lim ()(x)()) = (=)x)(=) Vz (-),00), Vx E X,
5000
where P.) denotes the probability density function of a random variable.
The proof for Theorem 3.1 can be found in Appendix A.4
Hence, in the limit, GP is a special case of a SlogGP. Under the conditions specified in Theorem 3.1, - 00, the lower bound - tends to negative infinity and the skewness converges to zero, causing the distribution to converge to Gaussian distribution. This relationship unidirectional: while SlogGP can approximate GP by setting its skewness parameter near zero, symmetric GP inherently lacks the flexibility to approximate skewed SlogGP distribution.
Figure iilustrates Theorem 3.1 by showing 200 samples from SlogGPs and GPs with three different covari- ance functions: RBF, Matern32 and Brownian. The left column contains samples from GPs. The middle column contains samples from SlogGPs whose parameters satisfy Theorem 3.1 (considering < oo cannot set ( to be large number Ç := 100) to approximate the GPs the left side. The right column contains SlogGPs with Ç 0.5. We can that the posterior distributions of the left and the middle are very close. If the shift Ç is not that large, we find that the distribution is more skewed, as is shown in the right side.
be achieved numerically,
GP
SlogGP (? 100)
SlogGP (? = 0.5)
RBF
1-
Matern32
-1
5.0
0.
5.4
0.
08
1.5
0.0
6.2
3.4
666
5.4
1.0
0.0
0.
0.4
5.6
0.8
3.6
Brownian
-1
-1
5.0
0.4
0.5
0.5
1.0
0.
0.2
0.4
0.5
0.5
1.0
0.0
0.2
0.4
2.5
0.8
2.0
Figure 1: Example of surrogate models (1D) for (left) a normal GP (center) SlogGP with the shift ç: 100 and parameters set according to Theorem 3.1 to match the normal GP, and (right) a SlogGP with the shift Ç : 0.5.
5

---

Published in Transactions on Machine Learning Research (03/2025)
3.3 An adaptation of El to the SlogGP Model
Expected improvement (EI) is popular acquisition function. Recall that for GP model, the posterior predictive distribution of function values at any particular location is Gaussian and thus El has a simple analytical form. However, for SlogGP, the posterior predictive distribution is no longer Gaussian and hence we have to adapt EI.
We derive alosed-form expression for Expected Improvement under the proposed surrogate model, which we term Shifted Logarithmic Expected Improvement (SlogEI):
In In(mmin () mx(x)
a SlogEl x; fmin) = (fmin + ()
o(x:)
- ex(()++ P00
+ In(fmin + + x(x)) - o2
(xx)
Derivation 1. For solution x and the current minimal observation Smin: we denote the predicted mean of g(x) as μ(x) and variance of g(x) as (x). Due to the normal distribution of g(x), ca(x) (denoted by Z) follows a log-normal distribution.
To calculate SlogEl, we firstly calculate ( =]+ using the definition of the log- normal distribution, where rj a constant and denotss the positive part function:
(n(a) #b0))
E[(n- =)+1
(1 2)
(x)2
dz
zx(x) 2n
(mite)
1
(ufe)
(n 2)
2x(3))
dz +
0.
dz
20 (x) v 2m
20 xx)x) 2m
11
(nit) 200))
(1 2)
e
(x)
dz
20 (x) (x) 2m 2m
1
(ule) 2(0))
77
(uts) m(x)) 2x(K)
1]
dz
dz
20 (x) 2t
00
2x
2T
(In(n) m(x)
(m(x) m(xs))
n
2m(())
dz
(xx
20 2x(x) (x)
The last cquality simply uses the definition of the CDF of the log-normal distribution. The second term is the partial expectation, so
(ate)) a(x))2
dz
(x(x)) mieo?
In(1)) - x(x) (x))
o(x)
Hence,
[((n Z)+]= 1/t6 (In(() (In(n)-1(x) -1(x)
pm(x))+ o(n)
[I((n) n(x) (x))
(xx)
o(x)
Setting 1) = Smi + C, we obtain the formula E[(Smin (Z (Z- i.e. SlogEl:
In + () - m(x))
a SlogEE xx; fmin) = (fmin + () &
(xx)
(x))+ 0300
2000
In In(fmin + p(x) 0 dx(x))
o(x)

---

Published in Transactions on Machine Learning Research (03/2025)
A visualization comparing SlogGP+SlogEI (Figure 17 with GP+EI (Figure 16 can be found in Appendix
A.3
Note that for := O, SlogEl becomes the acquisition function by Hutter et al. 2011 Furthermore, when the posterior mean and variance g(x) are differentiable with respect to x, the SlogEl acquisition function inherits this differentiability. The corresponding gradient calculations are also provided in Appendix A.2 along with an adaptation of the popular Probability of Improvement acquisition criterion.
In Figure we compare GP+EI and SlogGP-SlogEI empirically objective functions that are drawn from GP model, shown in the left plot, and SlogGP model as shown in the right one. The iyperparameters of both methods are fit to data. We observe that both methods perform similarly when the objective functions are sampled from a GP. When f is sampled from a SlogGP, the performance of GP+EI degrades substantially. This observation is consistent with the theoretical property shown above that a SlogGP model can approximate a GP sample, but not vice versa. We also performed a cross-validation test on the prediction error of the two surrogate models, see Table [ When the objective function is a GP, SlogGP and GP are equally good in prediction while when the objective function is SlogGP, SlogGP achieves better cross-va tross-validation error. Please see Appendix A.5 for details about the experimental setup.
Table Cross-validatae test. The column determines the test function and the row the choice of the surrogate model. The cells give the prediction error of the surrogate.
GP
SlogGP
GP
0.828(+0.0801)
6.21
SlogGP
1.414(+0.287) (0.0002))
GP 2D
SlogGP 2D
(6++)
dF+c Skg
Slag
simple regret
3.2
iteration
iteration
Figure Within-model Test. The left panel shows the performance of GP+EI and SlogGP+SlogEI on GP generated functions, while the right panel compares the two algorithms SlogGP generated functions. While both models work equally well on GP generated functions, only the SlogGP model works well on the SlogGP generated functions.
Next we show how the lower bound information /z model. Then we present novel acquisition function called SlogTEI that also leverages the lower bound f².
l can be incorporated into the SlogGP surrogate
3.4 Incorporating lower bound on the minimum into SlogGP
An advantage of the SlogGP model is that it allows to enforce a lower bound through its parameter Ç. On the other hand, the skewness introduced may not be very good fit to the observed data, creating possible conflict between setting to value that best represents the bound information, and setting Ç to value that best matches the observed data.
To solve this potential conflict, we use a maximum a posteriori (MAP) estimate for ( and include the bound information only as prior. Furthermore, we implemented some mechanisms that allow to quickly reduce the reliance on the prior if the observed mismatch between prior and fit to the data very large. The details of how choose Ç are explained in the following.
7

---

Published in Transactions on Machine Learning Research (03/2025)
When the lower bound fo is known, straightforward way is to set =- However, this method may suffer if the prior information is not accurate enough or would otherwise lead to a model mismatch. For instance, suppose that the objective function is SlogGP f(x) = eω(x) + 10, and the prior information is f(x) > O. In this case, the prior information is correct, but if we enforce Ç = O, the bound is too loose and the model skewness doesn't. match the skewness of the true function. Thus, any information about the lower bound should only guide, but not dictate Ç. A natural way is to use a maximum a posteriori probability (MAP) estimate for the parameter Ç. We choose the prior distribution to be a shifted log-normal distribution to guarantee that -S < fmin; where fmin is the current best value and - is the lower bound of the model, so that the model is well-defined:
~~ -mii +e?
(1)
where 2 ~ (In(Smii - f²), 2 If - + - 2 In(fminn - f)),, f)), and o1 is a positive hyperparameter, representing the gap between the mean and median of the prior distribution of ç, which implicitly determines its variance. We set the mean and variance of such that the median of - equals the known lower bound 9 and the mcan of Ç equals f - 51. The approach is not sensitive to the selection of 61, requiring only a small positive value. In our experiments, we set 61 0.1.
Sometimes, there can be prior-data conflict, where the prior information is inconsistent with observation data. Specifically, in SlogGP, a prior-data conflict is detected when the estimated lower bound - is unlikely under the prior distribution. Note that in this paper, we distinguish Ç and Ç: Ç is an unknown parameter while Ç is its estimator. To handle a prior-data conflict, we introduce an uncertainty level and variance threshold. When we observe the MAP estimator -c is unlikely under the prior distribution, we reduce our confidence through uncertainty level U. Specifically, we control the variance of Z to be U² (222(In(fmmn) - -f" + 61)_In(fmin - f*)) so that as the uncertainty level U increases, the prior bound information becomes weaker. Thus, we set Z in the prior in Eq. 1 to be Z [[[[ffmin - ff",, (2
f + 51) 21n(fmin f))))
Initially, the uncertainty level is set to be U 1. When Ç lies in the tails of the prior distribution, specifically when F'Gemar (3) < 82 or F'cprar(() (2) > 1- 82 (where Fserior is the cumulative density function of Sprior and 82 is hyperparameter that sets the threshold for detecting prior conflicts), then we interpret this a conflict between the prior information and the observed data, and re-train the surrogate model by maximum likelihood estimation (MLE) as the MAP depends on the prior information that we now consider unreliable. In the next iteration, we increase the uncertainty level U by multiplying it with the absolute value of standard score of the estimated č, which can be calculated as In(&+ Imin for the shifted log-normal Sprior
distribution (where Mprior and oprior refer to the mean and the variance of Z in the prior distribution). A large standard score absolute value indicates prior-data mismatch, prompting us to reduce reliance on the prior by increasing the variance in subsequent steps. 82 needs to be small probability. In this paper, we set = 0.01.
Additionally, we will exclude prior information when the estimated lower bound - significantly deviates from the known lower bound f³ (i.e., when - exceeds a threshold). However, setting a universal threshold is challenging due to the varying ranges of objective functions. Instcad, we leverage Theorem 3.1 which establishes that og - ôg (gy) - 0 as ( oo. Consequently, we use the estimated signal variance o3 as an indicator for incorporating bound information. Specifically, to prevent prior-data conflict, we disregard bound information when og < 63, where da is hyperparameter that determines when SlogGP's behavior approximates a GP closely enough. We set d3 = 0.25² in our experiments.
As we show in Section 5.2 our algorithm is not sensitive to the choice of &n, 52 and 83-
Figure 3 illustrates a GP model, a SlogGP model and a SlogGP model with the prior knowledge of s
3.5 Incorporating knowledge of f 2 fó into the acquisition criterion
A potential drawback of SlogEI that the improvement is calculatod over the range (-L. fmin), where the estimate - can be smaller than the known lower bound ft. Thus, the Shifted Logarithmic Trun- cated Expected Improvement (SlogTET) acquisition criterion truncates any impossible value below fo when calculating SlogEI:
SlogTEI (x: Jmin 1D) = a SlogEl (x; smin) - a SloggE (*)
8

---

Published in Transactions on Machine Learning Research (03/2025)
GP
SlogGP
SlogGP (known lower bound)
observation
observation
observation
95% confidence intervel
95% confidence interval
95% confidence interval
bound
lower
3
-2
0.2
3.6
0.6
0.2 1.
Figure 3: The predictive posterior distribution of the SlogGP model respects the known lower bound. (left) shows GP surrogate model; (middle) shows SlogGP surrogate model; (right) shows SlogGP surrogate model with known lower bound.
Note that if the estimate - is larger than the known lower bound, i.e - > f SlogTEl (x) becomes SlogEl (x). Additionally, SlogTEI is differentiable as it is the difference of two differentiable acquisition functions.
We summarize the BABO method that uses the SlogGp* surrogate and the Slog acquisition criterion in Algorithm where the superscript b means that the SlogGP is trained with lower bound information.
Algorithm BABO (SlogGP" + SlogTEI)
Initial data points Du and uncertainty level U Known lower bound f
for 0 to T do
4: Sct prior distribution Sprior: Eq.
Train surrogate model f(-) by MAP with D, and prior distribution Sprior
Feprior (2) < 82 or Pippiio (2) > - 62 then
Train surrogate model f(.) by MLE with D,
U- Ia($+ (m((+mmm))=pprr
8:
apvior
9:
end if
if f is trained by MAP and à < 63 then
10:
11:
Train surrogate model f(.) by MLE with D,, end if
12:
Find X,+1 = arg maxxxx fa (logTTE(()))
13:
14:
Evaluate the objective function Dat1 = ((xn+1)
15:
Update Dn+1 = D,, (xn+1.8nc Bn+13
16:
end for
Note that BABO can be straight extended to batch acquisition by adapting the qEI idea from Ginsbourger et al. 22008 The acquisition function value qloggTT (X; f fmins (o) for batch
of solutions X = (XI...., x,)) is calculated by Monte Carlo simulation: a AloogTEl (X; fmin: f")
Nare Dimi maxj= {Umin Exg) (f + where tenotes the positive part function, NMC is the number samples and & (Ff(() DD.
Similarly, we propose Truncated Expected Improvement (TEI) for vanilla GPs, an adaptation of Expected Improvement, (FI) that uses lower bound fo the global minimum. When know that the lower bound

---

Published in Transactions on Machine Learning Research (03/2025)
is f", we truncate the distribution of f(x) f and calculate TEI as
TEI (x: mmin: (4) Amiin [ (Jmin - =). (0-11(1) dz
o(x) p(x)
[[[[[[mmm - f(x))
TEI will serve as benchmark algorithm in our experiments.
4 Experiments
In this section, we compare performance of BABO with other benchmark algorithms across various test functions. The competing methods are:
Random
Bergstra et al. 2011
randomly choosing the next solution.
ET Mockus 1998
using Expected Improvement (EI) as the acquisition function
TEI: using Truncated EI, see Section 3.5
MES
(Wang al. 2018 adapted max-entropy search, see Section 2.2
OBCGP (Jeong & Kim
2021
OBCGP follows the paper by Jeong & Kim (2021). The original im-
plementation of Jeong & Kim
(021 lacks version information and depends deprecated packages
no longer available through pip. We have updated the codebase to utilize current package versions
and fixed existing bugs.
ERM Nguuyen & Osborne 2020 Our parabolic GP+ERM algorithm follows the method of Nguyen & Osborne (2020) and the code shared by the authors. Initially, we employ the regular GP+EI until the lower confidence bound (LCB) reaches the known lower bound. Subsequently, we switch to the transformed GP+ERM algorithm. This ensures a scamless integration between the two methods, allowing for an effective exploration-exploitaaoo trade-off during the optimization process. In addition, following their code, if the next evaluation X,+1 is too close to an existing observation (1-norm distance smaller than 3d 10-4), we will pick random instcad. The hyperparameter B of LCB is set be / [n(NN)
We also fix - in BABO understand the benefit of learning Ç. Note that when -< fh, SlogTEI will become equivalent to SlogEI, and BABO with fixed Ç can be viewed an extension of the method in Hutter et al. (2011) where in preprocessing step the function is shifted into the positive region based the information of the lower bound. We evaluate the algorithms on eight synthetic functions that are widely used in BO testing as well as three real-world problems. In addition, we compare qBABO with qEI on two synthetic functions with different batch sizes and the results are shown in Figure 18 in Appendix A.6 The
code available at https:: tttptt///ithub
http//aaaannggeee--aaann BBBBO. git)
Following the experimental setting in Nguyen & Osborne (2020); 3
Jeong & Kim 2021 we assumee noise-free observations in experiments. To ensure numerical stability, introduce small, adaptive noise variance proportional to the estimated signal variance (more details
be found in Appendix A.5). Additionally,
as discussed in Scction 3, the hyperparameters of BABO are set to be (51, 63) = (0.1,0.01 0..,0..0...2253. 0.25²). Wo will later show in Section 5.2 that BABO's performance is robust. to hyperparameter values. Plots show the simple regret (mean + one standard error) over 100 repetitions for all functions except 50 repetitions for the 10d function and 20 repetitions for the PDE Variance problem. For improved visualization, some results are presented using logarithmic scales on the y-axis, as indicated by their y-axis scale. Appendix A.5 gives more details of the experimental setup.
10

---

Published in Transactions on Machine Learning Research (03/2025)
4.1 Synthetic Test Functions
We compare the performances on eight synthetic functions with 2 to 10 dimensions where we have an exact bound on the minimum objective value. The known lower bound is thus set to the minimal value of the objective function, i.e., f = f", which is also the setting for which ERM was proposed.
Figure 4 summarizes the performances. Dashed lines indicate algorithms that do not use any information about fr, whereas solid lines correspond to methods that do leverage bound information. We observe that BABO (black line) usually achieves substantially better objective values the same iteration than the other algorithms. An exception is the Ackley (6D) benchmark where OBCGP achieves better values and BABO comes in second.
SkHHumpCamel 2D
Hartmann
14m
11-
iteraiion Resenbreck
131
Leration Ackdey ED
Reration Powell
Baration StyblinskTTang 10D
17-
154
114
Itestton
terstion
Reration
Ieatation
Random
....
GP+E1
GP+
GP+MES*
OBCGP
ERM
BABO (fxed 0
BABO
Figure 4: Experiment Results (Synthetic Functions): We observe that BABO works best except in Ackley (6D), where OBCGP works better.
4.2 XGBoost Hyperparameter Tunning Task
In the XGBoost Hyperparameter Tunning Task, the goal is to optimize six hyperparameters of an XGBoost model (Chen & Guestrin 2016) for various classification problems. We chose four: Skin Segmentation, BankNote Authentication, Wine Quality, and Breast Cancer. The six hyperparameters are min child weight, colsample bytree, max depth, subsample, alpha, and gamma. Notably, max depth is integer-valued. however, we conduct the search in a continuous space and round it to the nearest integer for evaluation. The objective value is determined by the modcl's classification crror rate on a hold-out dataset. Since we aim to minimize the error rate in BABO, the objective function has a natural lower bound of 0%. However, for visualization purposes, we present the results in terms of accuracy, this is the standard metric reported in classification problems.
Figure shows that BABO achieves the best performance in both the Skin Segmentation and Breast Cancer tasks. In the BankNote Authentication task, BABO and BABO (fixed <) demonstrate nearly identical performance, with BABO (fixed <) showing a marginally higher mcan at the final iteration. In the Wine Quality task, OBCGP exhibits a slight advantage over BABO. Overall, BABO demonstrates consistent and superior performance compared to competitor algorithms.
11

---

Published in Transactions on Machine Learning Research (03/2025)
Skón Segmeniaton 6D
BankNote Authertication 6D
Wine Quality 6D
Breast Cancer 6D
0.3
6.12
iteration
Figure An empirical evaluation of BABO and competitors on XGBoost hyperparameter tuning tasks. BABO achieves overall best performance.
4.3 PDE Variance Problem
The PDE Variance Problem (Maddox et al. 2021 aims to minimize the output variance of spatially-coupled Brusselator system by optimizing four continuous parameters that control its diffusivity and reaction rales. This problem is also used as benchmark problem in Li et al. (2024). The lower bound is zero as variance is non-negative. Since each function evaluation requires computationally expensive simulation, we limit our experiments to 20 repetitions.
Figure aa shows that BABO converges more quickly than other algorithms, followed by BABO (fixed ().
4.4 The Robot Push Problem
The goal of the 4D Robot. Push problem (Wang & Jegelka 2017 De Ath et al., 2021 | is to direct robot to push ball towards unknown target, minimizing the distance between the ball and the target. The distance is nonnegative and thus implics lower bound of zero for f" The first input variables determine the initial location of the robot, the third determines the angle of its rectangular hand, and the fourth sets the number time-steps that the robot to move.
Figure b() shows that BABO achieves better solutions the same iteration and converges more quickly, The runner-up is TEI that also uses our truncated El acquisition function but regular GP surrogate model.
PDE Variance
Robot Push 4D
158
variance
distance
12
16:
10
15
20
300
(6)
iteration
[b]
iteration
Random
TEI
MES
OBCGP
ERM
BABO (fixed
BABO
Figure 6: An empirical evaluation of BABO and competitors on the PDE Variance problem and the Robot Push problem. BABO achieves better solutions at the same iteration and converges more quickly.
To summarize the experimental results, we ranked the algorithms based on their mean at the final iteration, shown in Table 2
12

---

Published in Transactions on Machine Learning Research (03/2025)
Table 2: Performance Rank
Problem
BABO
Fixed
TEI
E1
MES
ERM
OBCGP
Random
Branin
1
6
4
5
2
7
Beale
1
8
4
7
3
5
SixmumpCamel
1
5
4
6
3
7
Hartmann
2
3
1
6
7
Rosenbrock
4
2
6
7
Ackley
2
4
7
3
1
Powell
3
5
6
StyblinskiTang
1
3
4
5
6
7
Skin
5
4
6
Bank
2
6
4
3
7
5
Wine
6
7
Breast
1
2
3
7
5
1
PDE
7
5
Robert
2
5
7
4
Average
1.3 3.1
4.4
4.4
4.7
5.0
5.1
7.9
5 Investigating the influence of prior information on the optimum
In this section, we study the influence of prior information about f". First, we conduct an ablation study and observe that leveraging prior information about the optimum model training and acquisition function is beneficial. Second, we conduct a hyperparameter sensitivity analysis demonstrate BABO's robustness across different values of ò1, d2, and 63. Then, we investigate why we need the uncertainty level and variance threshold when using bound information in model training. Finally, investigate how SlogGP" is influenced by its estimated lower bound - and lower bound information.
5.1 The Contributions of Different Components
To better understand how different components of BABO contribute its success, compare the following settings:
SlogGP-Sloge SlogEE:: Removing the bound information usage in both SlogGP and acquisition function. This will tell us how helpful the lower bound information really is.
+SlogE Removing bound information from the acquisition function only. This will show the relative importance of using bound information in the SlogGP model and the acquisition function.
SlogGP- Removing bound information from the model only. This will show the relative importance of using bound information in the SlogGP model and the acquisition function.
Looking at the results the eight synthetic test functions in Figure and two real-life problems in Figure the full BABO performs best, often followed closely by the version using the bound information only in the SlogGP model or the version using the bound information only in the acquisition function, and then the version not using bound information at all. The standard GP+EI usually did quite poorly, except for the Hartmann function where all methods share similar performance. An additional observation is that SlogGP+SlogEI (not using bound information) works better than GP+EI, which can be explained by SlogGP's larger expressive power, shown in Theorem 3.1 Also, it is worth noting that for the Ackley function, SlogGP+SlogEl is expected to perform similarly to GP+EL, the learned signal variance ô is nearly zero. However, in our experiments, +SooGI outperforms GP. This may be due to numerical issues. When optimizing the Ackley function, SlogGP learns a very small signal variance (approximately 10 10-5), and the resulting noise variance of 10 However, we observe that the effective noise (due to numerical issues) ends up be around 10-6 This seems beneficial for the Ackley function, as helps handle Ackley's numerous local optima. This phenomenon specific to Ackley due to the very small learned signal variance.
13

---

Published in Transactions on Machine Learning Research (03/2025)
Branin 2D
Bcale 20
SixHumoCamel 20
Harzmann 3D
single regret
147
187
301
Iteration Rosenbrock
Eeration
keration Powel: BD
Iteration StybliskiTang
zimple
GP+EI
GP+TEI
SlogGP +SlogEI
SlogGPt
Slog SEog
BABO
Figure 7: Different Components Comparison (Synthetic Functions): Generally, the complete BABO approach demonstrates superior performance, suggesting that incorporating bound information in both the modeling and acquisition function stages is beneficial.
Skin Segmentation
Robot Push 4D
29.25
13"
accuracy
distance
98.50
(a)
iteration
(b)
iteration
GP+EI
BloGG++SlogE
SlogGP+SIogTEI
GP+TEI
SlogGl*+StogEl
BABO
Figure 8: Different Components Comparison (Real-world Benchmarks): Generally, the complete BABO approach demonstrates superior performance, suggesting that incorporating bound information in both the modeling and acquisition function stages is beneficial.
5.2 Hyperparameter Sensitivity Analysis
BABO has three hyperparameters: 51, 82, and 8g. 6, represents the gap between the mean and median of the prior distribution, which implicitly determines the variance of the prior distribution of Ç. ó2 sets the threshold for detecting prior conflicts, while &3 determines when SlogGP's behavior approximates GP closely enough that we discard the prior information at the current iteration.
Figure 9 and Figure I0 demonstrate BABO's performance given different hyperparameter values across different objective functions. The Ackley function is a special case where the ó3 condition is consistently triggered, resulting in training without any bound information. Consequently, ón and 8, do not influence its performance, so only analyze its sensitivity to &z. The experiments demonstrate that BABO exhibits
14

---

Published in Transactions on Machine Learning Research (03/2025)
Branin 2D (6, comparison)
Beale 20 | comparison)
Hartmann (52 comparison)
131
10'
6. 0.05
0.05
5; = 0.05
1.7
6, 0.1
104
10-1
6 0.1
20-1
6, 0.5
0.5
01 = 0.5
simple
20-1
10"1
10-1
10'1
1010
10"1
20
60
60
80
100
Iteration
Iteration
Iteration
(6, comparison)
Beale 2D (Gg comparison)
Hartmann 3D (52 comparison)
101
0.001
6, 0.001
6, 0.001
regret
1.3
0.01
6; 0.01
10-1
6 0.01
8, 0.05
6g 0.05
62 0.05
10"2
100
20
60
100
Iteration
Iteration
Iteration Hartmann 3D (5, comparison)
Branin 20 (6, comparison)
Beale 2D (5g comparison)
107
16a
6g 0.12
6g = 0.12
6, = 0.12
0.25
6g - 0.25
6g - 0.25?
0.5?
8 0.5
6: 0.5?
simple
09
1010
20"1
10
300
100
Iteration
iteration
iteration
Figure 9: Hyperparameter sensitivity analysis shows that BABO is robust to different hyperparameter values except too small 63 may lead to marginally degraded performance on the Hartmann function.
robust performance across different hyperparameter values, with only marginally degraded performance observed when 03 is set too low.
This robustness can be explained as follows: Although d, determines the prior variance, which typically plays a crucial role in model training, our prior-conflict detection adjusts the prior variance when the data deviates from the prior bounds. For d2, we observe that as the number of observed data points increases, if the bound information does not align with the data, the probability that fo serves as valid prior information will quickly drop and reach 82, even when d2 is relatively small. Hence, even when 82 is set small, it still detect prior-data conflict. For dz, the choice does not matter for functions that can be modeled by SlogGP well (such as Branin and Beale), as their learned signal variance is usually large and will not trigger the 8g condition. For functions that are better modeled by a regular non-skewed GP (such as Hartmann and Ackley), when given the bound information, the learned signal variance will be small, usually in the range from 10-4 to 10-1, Therefore, to ensure we can discard all non-informative bound information, we recommend setting dy to be moderately larger. In the Hartmann and Ackley functions, we observe that too small a value (e.g., dg 0.1 can slightly worsen the performance.
5.3 Using bound information in model training
When the bound information is available, we MAP to allow use of this information, and uncertainty level and variance threshold to allow BABO to ignore the bound information if it would lead to model mismatch. A possible question is whether it is necessary to handle the prior-data conflict explicitly. To answer this question, we compare the following settings with SlogTEI:
Using MAP only
Using MAP and the uncertainty level
15

---

Published in Transactions on Machine Learning Research (03/2025)
Ackley 6D (63 comparison)
63 0.12
17.5
53 = 0.25²
15.0
63 0.5?
12.5
10.0
7.5
60
100
120
140
simple regret
Figure 10: Analysis on the Ackley function demonstrates that setting 03 excessively small values may lead to marginally degraded performance.
Branin 20
Ackley 6D
101
2x10
fixed
fixed {
10"
MAP
MAP
MAP+Uncertainty Level
MAP+ Uncertannyyy Level
BABO
BABO
10-1
10
10 20
4x
100
60
100
120
140
(5)
iteration
(b)
iteration
Figure 11: Ablation Study on How to Use Prior Information: For the Branin function, where the known lower bound quite accurate, all methods (MAP, MAP+Uncertainty Level, BABO) demonstrate comparable performance. In contrast, for the Ackley function, the optimal fitted value of - is substantially negative, creating a significant prior-data conflict with the known lower bound f = 0. In this case, disregarding the prior bound information in the surrogate model leads to superior performance.
We do experiments on two objective functions: Branin and Ackley. The results are shown in Figure 11 On the Branin function, their performance is almost identical at the beginning while MAP-only is slightly worse later on. This is because when we have enough data, we can estimate the underlying lower bound accurately even if the prior information can slightly misleading. For the Ackley function, MAP is worse than MmP-uncertannty level which in turn is worse than BABO. This is because the best fitting - of SlogGP very negative, so fo conflicts with observation data and not using the bound information is a better choice.
5.4 The influence of lower bound information in SlogGP- BO
We first test how different known lower bound information f influences BABO performance, We test on three objective functions: Branin, Beale and Hartmann. Our experimental results in Figure[1] demonstrato that BABO's performance consistently improves the known lower bound fo converges toward the true minimum value f
16

---

Published in Transactions on Machine Learning Research (03/2025)
Branin 2D
Beale 2D
Hartmann 3D
207
10P
=
simple regret
105
p= f"
12
Ø= l"
3
p= f" -
20
f*
100
10a
100
10-1
f 100
10-1
10-1
10-4
10
20
80
100
20
80
100
20
60
80
100
iteration
(b)
iteration
(c)
iteration
Figure 12: BABO with different f BABO's performance improves as the known lower bound fo approaches the minimum value f³.
Branin
Beale
Hartmann
4
-200
2.001 -11
-100 -200
-0011
-200 -200
(8)
(b)
Branin 2D
Beale 2D
Hartmann 3D
varfance of
-30 -30
51
Branin 2D
Beale
Hartmann
K - Xninl
-0.03
-2.011 -2.1
-10
-216
410011
-230
(g)
(h)
(0)
Figure 13: Exploitation and Exploration Analysis: When - is higher (towards the left of cach figure), BABO prefers more exploitation; When - is lower, BABO prefers more exploration. Top row shows E[ssase(xx+11 - Smin, middle row shows variance at next sample location, and bottom row shows distance of new sample location from current best.
17

---

Published in Transactions on Machine Learning Research (03/2025)
BABO consists of two parts: SlogGP" and SlogTEI. Although it is clear that the closer fo and f", SlogTEI should be better, it is not that obvious how the prior information influences the surrogate model. The influence of a given lower bound in SlogGP on BABO performance is clear when there is no model mismatch: when f close to the underlying -,, the prior information of the lower bound can help to learn the model lower bound quickly. On the other hand, when f is far away from the best fitting -S, this information can be misleading, which why we reduce the impact of the prior information if large gap is found.
If there is model mismatch (which is common in practice), the inffuence of lower bound information on the surrogate model is less clear. However, we found that the estimated parameter - influences the balance between exploitation and exploration. Specifically, higher value of - corresponds to greater emphasis exploitation, as the current best value in the underlying GP model 9min in(fmin + +)) will be more negative as - increases. Hence, BO more likely to search in close proximity Xmin, focusing more on exploitation. Conversely, if - is lower and far away from fmin, then BO will do more exploration.
In order to demonstrate this effect, we study the relationship between - and the properties of the next sample chosen. Our base model is SlogGP with - = fmin 1. 1.
Figures 13 13(a)(c) show that the gap between the mean of the next evaluation value and fmans i.e. Elfbase [x41)]] - fmin increases as - is lowered. Similarly, Figures 13 13fd)-(f show that the predicted variance of Jaase (Xa+1) increases as - is lowered.
Finally, Figures 13 gi-(1) present the relationship between Euclidean distance ||x+1 Xman |l and -G. A more local search near Xmin in case of larger - is an indication of more exploitation BO.
6 Conclusion
We proposed BABO, Bayesian optimization algorithm that is able to leverage prior information about the optimal value f to achieve better solutions and a higher sample-eficciney. BABO uses the tailored surrogate model SlogGP and the acquisition criterion Shifted Logarithmic Truncated Expected Improvement (SlogTEI), which an extension of Expected Improvement that takes the prior information into account. Experimental results demonstrate that BABO benefits from the prior information and outperforms previous approaches. Moreover, we find that even without prior information about f*, the SlogGP model often performs better than the commonly used GP model when combined with ET. For future work, we will investigate the potential of the SlogGP model further for wider range of scenarios.
We observe that BABO sometimes chooses a large absolute value for the parameter Ç of the SlogGP model, which means that the modeled function is 'pushed away' from the known lower bound on the optimal value. This is because the parameter fitting trades off model fit with using the prior information about fr, Although this effect is mitigated by incorporating the information about f* also in the acquisition criterion SlogTEI, we wonder if there is more suitable way of jointly modeling the unknown objective and the bound its optimal value.
Acknowledgments
We sincerely thank the anonymous reviewers for their valuable comments and constructive suggestions, which have greatly helped improve the quality of this paper. The first author gratefully acknowledges support by the Engineering and Physical Sciences Research Council through the Mathematics of Systems 11 Centre for Doctoral Training at the University of Warwick (reference EP /S02222444/
References
James Borgstra, Rémi Bardenct, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter opti- mization. Advances in Neural Information Processing Systems, 24, 2011.
Eric Brochu, Vlad M Cora, and Nando De Freitas. tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.20 2010.
18

---

Published in Transactions on Machine Learning Research (03/2025)
Roberto Calandra, André Seyfarth, Jan Peters, and Mare Peter Deisenroth. Bayesian optimization for learning gaits under uncertainty: An experimental comparison on a dynamic bipedal walker. Annals of Mathematics and Artificial Intelligence, 76:5 23, 2016.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 783-794, 2016.
Hyunghum Cho, Yongiin Kim, Eunjung Lee, Daeyoung Choi, Yongjae Lee, and Wonjong Rhee. Basic enhancement strategies when using Bayesian optimization for hyperparameter tuning of doep neural net- works. IEEE Access, 8:52588 52608,,52888---8 2020.
George De Ath, Richard M Everson, Alma AM Rahat, and Jonathan E Fieldsend. Greed is good: Exploration and exploitation trade-offs in Bayesian optimisation. ACM Transactions on Evolutionary Learning and Optimization, 111):1-2,, 2021.
Jose Pablo Folch, Shiqiang Zhang, Robert Lec, Behrang Shafei, David Walz, Calvin Tsay, Mark van der Wilk, and Ruth Misener. Snake: Bayesian optimization with pathwise exploration. Advances in Newral Information Processing Systems, 35:35226- (185226--5239,, 2022.
Roman Garnett. Bayesian optimization. Cambridge University Press, 2023.
David Ginsbourger, Rodolphe Le Riche, and Laurent Carraro. A multi-points criterion for deterministic parallel global optimization based on Gaussian processes. hal-00260579, 2008.
Tom Gunter, Michael A Osborne, Roman Garnett, Philipp Hennig, and Stephen J Roberts. Sampling for inference in probabilistic models with fast Bayesian quadrature. Advances in Neural Information Processing Systems, 27, 2014.
Frank Hutter, Holger H Hoos, Kevin Beywon-Brown, and Kevin P Murphy. An experimental investigation of mosel-based parameter optimisation: Spo and beyond. In Genetic and Evolutionary Computation Conference, pp. 271 1278, 2009.
Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Brown. Sequential model-based optimization for general algorithm configuration. In International Conference on Learning and Intelligent Oplimization, pp. 507- 523. Springer, 2011.
Bjorn Sand Jensen, Jens Brehm Nielsen, and Jan Larsen. Bounded Gaussian process regression. In 2013 IEEE International Workshop Machine Learning for Signal Processing (MLSP), pp. 1-6. IEEE, 2013.
Tacwon Jeong and Hecyoung Kim. Objective Bound Conditional Gaussian Process for Bayesian Optimiza- tion. In International Conference on Machine Leurning, pp. 4819 4828. PMLR, 2021.
Yueen Lily Li, Tim G. J. Rudner, and Andrew Gordon Wilson. A Study of Bayesian Neural Network Surro- gates for Bayesian Optimization. In The Twelfth International Conference on Learning Representations, 2024. URL https::/ wttp:eetteeoommmddddsss......
Wesley J Maddox, Maximilian Balandat, Andrew G Wilson, and Eytan Bakshy, Bayesian optimization with high-dimensional outputs. Advances in Neural Information Processing Systems, 34:19274-19287, 2021.
Jonas Mockus. The application of Bayesian methods for seeking the extremum. In L.C.W. Dixon and G.P. Szegö (eds.), Towards Global Optimisation, volume 2, pp. 117 129. North-Holland, 1998.
Vu Nguyen and Michael A Osborne. Knowing the what but not the where in Bayesian optimization. In International Conference on Machine Learning, pp. 7317-7326. PMLR, 2020.
Vu Nguyen, Mare Peter Deisenroth, and Michael A Osborne. Gaussian process sampling and optimization with approximate upper and lower bounds. arXiv preprint arXix:2110. 120887 2021.
Andrew Pensoneault, Xiu Yang, and Xueyu Zhu. Nomnegativityy Gaussian process regression. The- orelical and Applied Mechanics Letters, 10(3):182-188 187, 2020.
19

---

Published in Transactions on Machine Learning Research (03/2025)
Carl Edward Rasmussen and Christopher KI Williams. Gaussian processes for machine learning. Springer, 2006.
Binxin Ru, Michael A Osborne, Mark McLeod, and Diego Granziol. Fast information-theoreti Bayesian optimisation. In International Conference on Machine Learning, pp. 4384-4392 PMLR, 2018.
Eric Schulz, Maarten Speekenbrink, and Andreas Krause. tutorial on Gaussian process regression: Mod- elling, exploring, and exploiting functions. Journal of Mathematical Psychology, 85:1 16, 2018.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148 175, 2015.
Edward Snelson, Zoubin Ghahramani, and Carl Rasmussen. Warped Gaussian processes. Advances in Neural Information Processing Systems, 16, 2003.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable Bayesian optimization using deep neural networks. In International Conference on Machine Learning, pp. 2171 2180. PMLR, 2015.
Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating Bayesian optimization for biological sequence design with denoising autoencoders. In International Conference on Machine Learning, pp. 20459 20478. PMLR, 2022.
Zi Wang and Stefanic Jegelka. Max-value entropy scarch for efficient Bayesian optimization. In International Conference on Machine Learning, pp. 3627 3635. PMLR, 2017.
Zi Wang, Clement Gehring, Pushmeet Kohli, and Stefanic Jegelka. Batched large-scale Bayesian optimization in digensiiensional spaces. In International Conference on Artificial Intelligence and Statistics, pp. 745- 754. PMT.R, 2018.
20

---

Published in Transactions on Machine Learning Research (03/2025)
A Appendix
A.1 MES' (x I fo) shares the same maximizer with P(f(x) < f*)
MES shares the same maximizer with the acquisition function P(J(x)
, i.e.
argmaxxes x MES" (x fo) = x < fo). See examples in Figure 14
Figure 14: MES" and IP(((x) < f²)
Proof. For simplicity, we use y to represent 7 (x, fo) ~ (x)
for the following analysis.
Firstly, P(f(x) < f*) = 2 ( o(x)
= 1 - 277), so is monotonically decreasing with 7.
The next step is to prove that MES is also decreasing with 7. We look at a MES (x fh) For simplicity, we denote it by M()- The derivative of M(7) is:
-00(>) 20(77)
log ( (7)).
1
((7)
+ 71)) - 7 7.0(7)
x(7)
((7)
((7) 27>)²
(7))
1
((1) 0')) 7 o(7)²
((1) 4(7) 2())²
1
((7) 7 (0(1)
2 6(7)²
((1)
4(7)
2(7)²
((7)
(77)
7 -(1)))
((1)
4(7)
Φ(7)²
((7)
72
7 6(1)
+
((7)
4(7)
4(7)²
)())
(0(7) + + k(7))
(7)
To determine the monotonicity of M(),, we need to determine whether D(y) = (1)) 47. ((7)) is
positive or negative. It is easy to calculate D'(7) 2Φ(7)) 27) Given D² = -0(()) 0, D'(r))
increasing function in (0) and the minimal value is achieved when 7 - -00:
lim D'(2) = lim 2(0(7) + +)(())
20
x
2 lim
70 0c
4'())
2 lim
70 (2)
2 lim -
7-
0
21

---

Published in Transactions on Machine Learning Research (03/2025)
Hence, D'(y) for y E (-00,00), D(y) is an increasing function. Similarly, D(y) - 0 when + -00 so D(7) > 0 for (-x0, Hence, the derivative of M()) is negative, Lee. M(7) is a decreasing function of y.
Considering is a function of P(((x) < f*) and MES (x (a) share the same monotonicity with respect to x, they share the same maximizer.
Figure 15 shows the decreasing relationship between 7 and MES Consider MES' and P(f(x) ( are both decreasing with they share the same maximizer y
3.5
3.0
2.5
MES
1.5
1.0
0.0
-20
-15
-10
-5
0
5
15
20
V
Figure 15: MES is decreasing with 7. As MES and P((()) < <1) are both decreasing with 7, they share the same maximizer 7"
A.2 Adapting the acquisition functions El and PI for the SlogGP model
In the following section, propose adaptations of popular acquisition functions to the SlogGP model, Shifted Logarithmic Probability of Improvement (SlogPI) and Shifted Logarithmic Expected Improvement (SlogEI). If the currett best value is denoted by fmin: SlogPI is
SlogPl (x; smin) = P(f(() S (minn) In [In(fmin + 1xx)))
o(x)
The calculation process is shown as follows.
For a solution x and the current minimal observation fmins we denote the predicted mean of g(x) as p(x) and variance of g(x) (x().. Due to the normal distribution of g(x), eo(x) (denoted by Z) follows log-normal distribution. Hence, SlogPI can be calculated follows:
a SlogPl (x; fmin)
P(((() S Smin)
IPZZ ( fmin)
IP S smin + +))
2 In(ffmn + + - x(x))
(xx)
22

---

Published in Transactions on Machine Learning Research (03/2025)
The formula for SlogEI and its derivation presented in Section 3.3 and do cxist. The partial derivative is
SlogEI is differentiable with x if Ou
SlogE1
Ox Slggl
8p
da SlogEl
do 0x;
ix;
0x;
do
where
da SlogEi
(fmin +()
[n([fin +() +()-1(x))
o(x) o
o(x)
ea(x)) -i00
In( + +() (1(x) - 0² -02(x)
o(x)
o(x)
-x((x)++ 2ww
In(Smin + + u(x)x x²())
o(x)
and
da SlogEl do
(Jmin + x2(x)
(In[Smin + + +5)-m(x)) m(xx)
(In(Jmin + 5) - -x(x))
o(x)
el(x))+ 22w
In(fmin + -1(x) - o2(x)) ( [in(fmsn + +5) - -()-((()
+ +)
o(x)
x²(x)
xxx)c)(x)) p000 200 $
[In(Smin + + [x(x) - o² o2())
o(x)
A.3 Acquisition Function Visualization
Objective Function Diserved Points
Next Evaluation
Functtoo
BostEl
Acquisiion Function
3.0
Figure 16: Example data acquisition of first iterations of BO using GP+EL Upper figures shows surrogate model and true objective function, lower figures show acquisition function. Three imitial points.
23

---

Published in Transactions on Machine Learning Research (03/2025)
Step Oujedive onnction Doserved Po nix Next Evalustion
Step
Step
Function Velue
SiegEl
Acquisition Function
0.00
Figure 17: Example of data acquisition of first 4 iterations of BO using SlogGP+ Upper figures shows surrogate model and truc objective function, lower figures show acquisition function. Threc initial points.
A.4 Proof of Theorem 3,1
For convenience, we repeat Theorem 3.1 first before stating the proof.
Theorem 3.1. Define SlogGP f(x) e0x)+,, - with g(x) Gaussian process with zero mean, zero noise and a covariance function K with signal variance og and other hyperparameters 0g.
Then for any GP l(x) with mean Ag signal variance og. and other hyperparameters 0,, we can have the SlogGP f(x) converge to h(x) in distribution for any X' by setting
Mg = In + Ag)
0 0
0, = o
+ Pag
and letting Ç 00, i.e.,
lim Pr()(2) = (2(x)(2) Vz E (-),00), Vx E X, 800
where P(-) denotes the probability density function of a random variable.
Proof. Given SlogGP f(x) = ea(x)+H, with g(x) Gaussian process with zero mean, zero noise and covariance function K with signal variance of and other hyperparameters 0g- We set og O Mg such that the following equations:
Ng = In (5 + Agy
0 0
ōg
0g Ç + Bg
satisfied for some constants (ā Pgg 0g) E O.
24

---

Published in Transactions on Machine Learning Research (03/2025)
zh According to the definition ex -k= kTI have
f(x)
cx -
1 2x00(x) ())))))))))))))))) + ...-- -
1 (x0
= etaa. + g(x) +
= =xx)g(x)) - et - 5 + eta ( (g0 + 69 gxx)) + +..).
1
In the formula, we distinguish three different parts: ex*g((x), et - and e (xx) + (x) +..).
In terms of the first part, g(x) is Gaussian process with zero noise, variance og and other hyperparameters 0g- Hence, e**g(() is Gaussian process with zero noise, zero mean and kernel with signal variance e² and other hyperparameters 0g.
and kernel with signal
The second part e** Ç is the constant Bg as we set Me = In (S + 11g)
For the third part, first look ex*) We have e#*g g²/g((x) (e "M g(x))² e*>gg (xx) is the square of Gaussian process with signal variance o and mean O. When - oo, we have el o + 0, so Vx
lim P (Ie")gp(x) - 0/ > 5) <00
For other terms et'gg in the third part, lim P (lettgg(x)) - 0/ e) = 0 for le 2 3, vx and Ve.
view them (e g(x)), similarly, we have
Therefore, for Vx and Ve,
lim (If(x) g(x) > E) = 0, <-000
where j(x) (x*)(()) elts - is Gaussian process with mcan Mg: zero noise and covariance function K with signal variance og and other hyperparameters 0g.
Therefore, we have
lim (x(x)()) = (z) Vz E (-), Vx E E C0000
where P()) denotes the probability density function of random variable.
Given that y and h share the same parameters, they share the same posterior distribution, i.e.
(2) (2)))(2) Vz (-C,,0, Vx X,
Hence, we have
lim ()))(2)= = P=oo)(=) V2 E (-C,,0))) Vx Ex So
A.5 Experimental Settings (Within-mood model Test, Synthetic Function Test and Real-world Benchmark Test)
The experimental sctting as follows. For each d-dimensional test function, we sample 4d initial points from Latin hypercube design. The input domain scaled to 1J, For GP-based methods, function values are standardized (scaled and centralized) and for SlogGP-based methods, function values scaled and the centralizing is done in model training. As kernel we use the squared exponential kernel: K (xa: xx)
o² eeep (- [x.. xx.l") For acquisition function optimization, we use restart L-BFGS-B in scipy. The restart time and initial samples (restart number 3d and initial sample 30d) and L-BFGS-B B options are the same for all acquisition functions. We compare the performance of BABO with other benchmark algorithms across various test functions. The benchmark algorithms we consider are Random, EI, TEI, MES', OBCGP and ERM.
25

---

Published in Transactions on Machine Learning Research (03/2025)
We consider noiseless setting in our theoretical analysis. In practice, small positive noise variance is typically introduced to ensure numerical stability. Since the estimated signal variance o in SlogGP can vary substantially, ranging from near-zero to several hundred, we adopt an adaptive noise variance proportional to the estimated signal variance: mniis = 10-5 2n The initial noise variance is set to 6 10-0. We maintain these noise parameter settings across all comparative methods to ensure fair comparison.
Details of test functions are shown in the table below.
Table 3: Test Function Information
Test Function
Optimal Value
Search Space
Ge-ggeneatedd functions (2D)
[0,
MlogGP-generatee functions (2D)
00,, 1.22
Beale (2D)
4.5, 4.5)2
Branin (2D)
0.397887
5., 10.
[0., 15.
SixHumpCaml (2D)
-1.0316
3.J3.,
22, 2.
Levy (2D)
10., 10.
- 10., 10.
Hartmann (3D)
33.6278
1.1
DixonPrice Price (4D)
10., 10.]
Rosenbrock (4D)
-2.04, 2.048
Ackley (6D)
-32.768,
32. 7687
Powell (8D)
-.,, 5.1%
Styblinski Tang (10D)
-391.6599
5., 5.1%
PDE Variance (4D)
00.1., 5.
0.1, 5.].
00.01, 5.]. 0.0
Robot Push (4D)
5.1,
55,, 5.
00,, 2m
0.,
300.
XGBoost Hyperparameter Tuning (6D)
10., 10.],
[0., 10.]. [5., 15.]
1., 20.
0.5. 1.). 0.1,1.]
In addition, in terms of within-model in Section 3.4 for GP--generaed functions, signal variance of = 2, functions, signal variance of = 1.2, lengthscale
lengthscale = 0.1 and mean 0. For 1= 0.1, shift Ç = 30 and mean 0.5.
We also had cross-validation test for GP and SlogGP. The repetition number is 50. In each experiment, 40 initial training points are sampled randomly. We test the gap between model prediction mean at random x and its value fxx).
A.6 More experiments: batched BABO
26

---

Published in Transactions on Machine Learning Research (03/2025)
Levy 2D
DixonPrice 4D
(q=1)
qEi (q=1)
qel (q=S)
qEl (q-5)
(q=10)
qEi (q=10)
QBABO (q-1)
QBAB0 (q=1)
qRABO (q=5)
4BAB0 (q=5)
simple regret
qBABO (q)
QBABO (q-10)
107
05
140
number of solutions
number of solutions
Figure 18: Batched BO: We evaluated the performance across varying batch sizes (q = 1, 5, and 10) for both test problems. The experimental results demonstrate that qBABO consistently outperforms qEI across all batch size configurations.
27


Reviews:
[{"review_id": "lXtMSWg27Q", "reviewer": "Reviewer_RefA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper proposed a novel modification of a GP able to handle lower bound information and proved its higher expressiveness compared to a classic GP. \n- The paper proposed an adaptation to the EI acquisition criterion that incorporates lower bound information; these adapted acquisition functions can be used with both SlogGP and classic GP.\n- The proposed method is shown to outperform classic GP in the considered experiments, which may motivate wider adoption of SlogGP (pending further investigation in the appropriate use cases). This would represent a significant impact to the field.\n- The paper provides a set of ablation studies to evaluate various components of the model.\n- The paper is well-written and clear.\n\nWeaknesses\n- There is limited discussion on what it means when estimated ζ is far away (or lower) than the known lower bound (it is only touched upon at the end of *Conclusion*).\n- Section 3.4. has many highly specific steps that would benefit from providing more intuition.\n- The latter benchmarking sections (starting from 5.1) become less clear and appear more rushed in presentation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a novel Bayesian optimization (BO) framework BABO specifically tailored to scenarios when a lower bound on or the minimum value of the objective function is known; though BABO can also be used when such information is not available. BABO utilizes a modification of a Gaussian process, SlogGP, as a surrogate model, which combines a log-transformed GP with a learnable shift parameter. To use SlogGP, whose predictive distribution is no longer normally distributed, the authors adapt the Expected Improvement acquisition function and additionally incorporate the lower bound information, resulting in a novel acquisition function SlogTEI. The authors show that SlogGP with SlogTEI (which together constitute the BABO framework) outperforms GP-based BO and other existing methods on a range of experiments. The authors also perform a range of ablations evaluating the contributions of leveraging bound information. The authors further prove that SlogGP is a more expressive model than the GP.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Requested Changes\n\n- Please briefly state if there are any commonly used conditions under which Theorem 3.1 does not hold. For example, can a SlogGP work in a noisy setting?\n- 3.4: Please show that the parameters of the $ζ_{prior}$ result in the median / mean of $ζ$ as specified. Why is it desirable to have the median equal to $f_b$ and mean be $\\delta_1$ away?\n- 3.4: The criterion for $\\hat{ζ}$ being far away from the lower bound appears to specify two regions of the CDF of $ζ_{prior}$, a thin slice below $δ_2$ and above $1-δ_2$. Can you please elaborate how this compares the estimated $\\hat{ζ}$ to $f_b$?\n- 3.4: Please give more intuition for why U is multiplied by the provided expression and define $\\phi^{-1}$ after estimating MLE of $ζ$.\n- 3.4: Why does signal variance being too small indicate the data-prior conflict? Is it because that would happen when $ζ \\rightarrow \\inf$, which would indicate it being far away from $f_b$?\n- 3.4: How were $\\delta_1, \\delta_2, \\delta_3$ chosen?\n- 3.5: Please show why SlogTEI becomes SlogEI for  $-ζ > f_b$.\n- Fig. 7: Which method estimated the $\\hat{ζ}$ shown with the \"no bound information\" curve? Plotting the true lower bound on each panel would also be instructive.\n- Fig 7: \"No bound information\" curve often converges to a lower bound than the other curve (which I am assuming to be BABO). Does that mean that the \"no bound information\" method found a better optimum? In most scenarios, both no-bound and bound methods converge to the same value; what is the interpretation of that? Does this mean that using bound information doesn't confer any utility?\n- Please provide a discussion on the difference between $ζ$ being far away from $f_b$ and model mismatch and how this connects to (i) BABO with fixed $ζ$ performing worse than having to learn $ζ$, (ii) BABO learning estimated $ζ$ that is lower than $f_b$ (e.g. Ackley in Fig. 7) and (iii) absorbing both skewness and lower bound information into a single parameter.\n- There is inconsistent use of $\\hat{ζ}$ and $\\tilde{ζ}$ in the manuscript.\n- Fig. 10: \"Fixed C\" in the legend is not referenced anywhere else.\n> However, for functions like Beale where the estimated bound converges near the true minimum, this additional bound information proves more beneficial.\"\n\nWhat indicates this, the superior performance of SlogGP over GP? But the previous statement suggests that this wouldn't hold on Ackley, yet BABO still performs best on that function.\n\n### Remaining questions:\n- Table 1: Why is SlogGP error on the SlogGP test function higher than that of SlogGP on the GP test function? Is a GP function easier to fit that a SlogGP function?\n- Fig. 8a: Both GP and SlogGP using TEI perform worse than all other methods. Do you have an intuition about this performance of TEI?\n- Can other acquisition functions be modified to be used in SlogGP?\n- It could be instructive to also show SlogGP without lower bound information in Fig. 3\n- How often is the lower bound information available in practice? Does incorporating more \"trivial\" bounds such as \"distance > 0\" still advantageous based on your observations?\n\n### Minor:\n- Fig. 1: \"with with\" typo\n- Fig. 3: \"l\" reads as \"1\". Might be clearer to spell out left/right\n- p. 9: typo in \"codebase\"\n- p. 12: \"In the Beale function ... the convergence value of 3\" should say \"value of -3\" according to Fig. 7\n- Eq. 3.4 in Algorithm 1 links to Section 3.4 and the equations are not numbered\n- Fig. 4: specify GP+EI and GP+TEI in the legend", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bXWEnbL7db", "reviewer": "Reviewer_oC82", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- Novel Contribution: The introduction of SlogGP and SlogTEI is original and addresses a practical need in BO applications where bound information is available.\n- Theoretical Analysis and Practical Methodology: The flexibility of SlogGP is proven. The proposed mechanisms (e.g., MAP estimation for ζ, variance thresholds) are practical for handling prior-data conflicts.\n- Comprehensive empirical experiments including ablation studies.\n- Clarity: The paper is well-structured, with clear explanations of the methodology and results.\n\n# Weaknesses\n- Limited Hyperparameter Analysis: The selection and sensitivity of hyperparameters (e.g., $\\delta_1, \\delta_2, \\delta_3$) are not extensively analyzed, which may affect reproducibility.\n- Lack of discussions of why BABO outperforms standard BO. My intuition is that a known optimal bound can help achieve a better balance between exploitation and exploration. For instance, it encourages exploration when the current solution is far from the optimal bound, while promoting exploitation when the current solution is close to the optimal bound. \n- Comparison with Broader Models: As mentioned in the paper, the proposed SlogGP model can be viewed as combining aspects of a parabolic $f(x) = \\frac{1}{2} g^2(x) - \\zeta$ and a log-transformed GP $f(x) = e^{g(x)}$. Thus, a comparison with these models using the proposed Truncated EI (TEI) could further illuminate the contribution and potential advantages of the SlogGP model.\n- Computational Complexity: The additional computational overhead of SlogGP and SlogTEI compared to standard GPs is not quantified.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Bound-Aware Bayesian Optimization (BABO), a novel approach to Bayesian Optimization (BO) that leverages prior information on the achievable bounds of the objective function. Specifically:\n- The authors propose a new surrogate model, SlogGP, which incorporates a shift parameter (ζ) to encode bound information, thus improving model expressiveness.\n- A tailored acquisition function, SlogTEI, is developed to exploit bound information effectively, improving upon standard Expected Improvement (EI).\n- The paper discusses mechanisms for handling mismatches between bound priors and observed data, ensuring robust performance.\n- Empirical studies on synthetic and real-world benchmarks validate the proposed method, demonstrating significant performance gains over existing techniques.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Critical Adjustments\nAs mentioned in Weaknesses, the following modifications will enhance the contributions of this paper.\n- Analyze hyperparameter sensitivity including experiments to evaluate how variations in $\\delta_1, \\delta_2, \\delta_3$ impact BABO's performance, and discuss practical guidelines for selecting these hyperparameters in diverse scenarios.\n- Some discussions and experiments on why BABO outperforms standard BO when the prior bound exactly matches the real bound\n- Comparison with parabolic and log-transformed GPs, as mentioned in Weaknesses.\n- Provide computational complexity analysis and results. \n\n# Minor Suggestions\n- Modifying some typos. e.g., Sec. 4 the citation of OBCGP is wrongly inserted into the word \"codebase\".\n- Provide additional real-world examples where bound information is naturally available and impactful.\n- Discuss limitations in scenarios where the proposed might fail.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "db9pkIWZdw", "reviewer": "Reviewer_bJPZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n* Paper builds on reasonable assumptions in practical applications.\n* Strong overall performance.\n* Good ablation studies.\n\n## Weaknesses\n\n* The submission does not come with code, and code is only promised upon acceptance.\n* Rather limited set of experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an extension of Bayesian optimization that combines log transformations of the target function with incorporating a bound on the optimum value in the model. The bound on the optimum value is a realistic assumption, because in machine learning we often know the best possible value, for example, accuracy is bounded between 0 and 1. The paper demonstrates superior results compared to standard Bayesian optimization on 8 synthetic functions, one XGBoost hyperparameter optimization problem and the 4D robot push problem. Furthermore, the paper contains a thorough ablation study that shows the benefits of the individual contributions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Please provide code for your method and the experiments.\n* Please make the paper more self-contained:\n    * Provide the mean and variance prediction equations of the SlogGP.\n    * Why is the acquisition function derivation in the Appendix? This is a journal paper, I would assume such central information in the main paper. Additionally, it would be great if the new acquisition functions could be contrasted with general EI and PI as described by [Brochu et al. (arXiv, 2010)](https://arxiv.org/abs/1012.2599).\n* Please add more benchmark tasks. In the past years, researchers have published and used various new benchmarks, see for example [HPOBench](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/93db85ed909c13838ff95ccfa94cebd9-Abstract-round2.html) or the recent comparison of [BNN](https://openreview.net/forum?id=SA19ijj44B).\n* The paper should discuss the related work [unexpected improvement to expected improvement](https://proceedings.neurips.cc/paper/2023/hash/419f72cbd568ad62183f8132a3605a2a-Abstract-Conference.html)\n* The method introduces a total of 3 additional hyperparameters. However, there is no further analysis of this, and the paper should contain a sensitivity analysis on them.\n* The Ablation misses SlogGP+EI.\n\n## Minor / Questions\n\n* Page 2: instead of \"For Max-value Entropy Search (MES) introduced by Wang & Jegelka (2017)\" I suggest writing \"For Max-value Entropy Search (MES; Wang & Jegelka, 2017)\", which introduces the reading flow because one does not have to focus on the author names, which add little additional information.\n* How does the statement \"In addition, we show in Appendix A.1 that MESb shares the same maximizer with the acquisition function P(f (x) < f b), i.e. argmaxx∈X αMESb (x | f b) = argmaxx∈X P(f (x) < f b).\" relate to SlogEI?\n* The text describing Figure 1 states \"The hyperparameters of both methods are fit to data.\", and I am wondering to which data the hyperparameters are fit? If we have a known GP, why don't we fit the BO surrogate to the GP hyperparameters?\n* Regarding the experimental setup (A4) I have the following questions: is the fixed variance a standard procedure in BO? So far, I have not seen this, and especially in the case of noiseless functions I wonder about the effect of adding a fixed noise. Why not estimate the noise too, or set it to zero (or near-zero) in the case of synthetic functions?\n* Figure 11 is horribly typeset, and the caption is below the page number.\n* The ticklabels are hard to read and their font size should be increased.\n* Section 3.5 contains pseudo-code and various additions to the acquisition function. I suggest moving the pseudo-code to its own subsection, as I find it a bit confusing and did not expect it here. Also, I would appreciate line numbers for algorithms.\n* The equation in 3.1 that gives the negative log likelihood should be abbreviated by NLL and not L, and the equation should end with a comma.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is about Bayesian optimization when knowledge about the optimal value is available. The authors provide a method to incorporate such knowledge into a new surrogate model (based on shifted log-normal GPs) and derive the expected-improvement acquisition function for their model. 

The reviewers all agree that the paper should be published. I agree. The paper is well crafted with a substantial amount of empirical evidence.
Decision: Accept (poster)