MALIBO: Meta-Learning for Likelihood-free Bayesian Optimization
Authors: Jiarong Pan
Stefan Falkner
Felix Berkenkamp
Joaquin Vanschoren
Venue: Submitted to ICLR 2023 (2023)
Keywords: Bayesian Optimization
Meta-learning

Abstract: Bayesian Optimization (BO) is a popular method to optimize expensive black-box functions. Typically, BO only uses observations from the current task. Recently proposed methods try to warm-start BO by exploiting knowledge from related tasks, yet suffer from scalability issues and sensitivity to heterogeneous scale across multiple datasets. We propose a novel approach to solve these problems by combining a meta-learning technique and a likelihood-free acquisition function. The meta-learning model simultaneously learns the underlying (task-agnostic) data distribution and a latent feature representation for individual tasks. The likelihood-free BO technique has less stringent assumptions about the problems and works with any classification algorithm, making it computation efficient and robust to different scales across tasks. Finally, gradient boosting is used as a residual model on top to adapt to distribution drifts between new and prior tasks, which might otherwise weaken the usefulness of the meta-learned features. Experiments show that the meta-model learns an effective prior for warm-starting optimization algorithms, while being cheap to evaluate and invariant to changes of scale across different datasets.

Under review as a conference paper at ICLR 2023
MALIBO: META-LEARNING FOR LIKELIHOOD-FREE BAYESIAN OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Bayesian Optimization (BO) a popular method to optimize expensive black- box functions. While BO typically only optimizes single task, recent methods exploit knowledge from related tasks to warm-start BO and improve data-efficiency. However, these methods either not scalable or sensitive heterogeneous value scales across multiple tasks. We propose a novel approach to solve these problems by combining meta-learning with likelihood-free acquisition function. Specifically, our meta-learning model simultaneously learns the underlying (task- agnostic) data distribution and latent feature representation for individual tasks be used as the acquisition function inside BO. The likelihood-free approach has less stringent assumptions about the problems compared regression based methods and works with any classification algorithm, making it computation efficient and robust different scales across tasks. Finally, we use gradient boosting as residual model on top to adapt to distribution drifts between new and prior tasks, which might otherwise weaken the usefulness of the meta-learned features. Experiments show that the meta-model learns an effective prior for warm-starting optimization algorithms, is cheap to evaluate, and invariant under changes of scale across different datasets.
INTRODUCTION
Bayesian Optimization (BO) is widely used method optimize expensive black-box functions (Shahriari et al., 2016) and has been successfully applied in different fields, including automated machine learning (ML) (Hutter et al., 2019). Given small amounts of data, traditional BO uses Gaussian Process (GP) surrogate model together with an acquisition function to quickly optimize black-box function. However, most BO techniques start from scratch for each new optimization problem, instead of leveraging information from previous runs for similar tasks to further improve data-efficiency.
To warm-start BO, exploiting additional task information has been explored in the context of transfer learning (Weiss et al., 2016) and meta-learning (Vanschoren, 2018). Prior knowledge can be used to build informed surrogate models (Schilling et al., 2016; Wistuba et al., 2018; Feurer et al., 2018b; Perrone et al., 2018), to restrict the search space (Perrone et al., 2019), or to warm-start the optimization with configurations that generally score well (Feurer et al., 2014; Salinas et al., 2020).
However, these approaches have three important issues:
(i) GPs scale poorly due to their cubical computational complexity (Rasmussen, 2004).
(ii) The standard BO framework requires a surrogate model with well-calibrated and tractable predictive uncertainty, which is challenging in high-dimensional problems (Tiao et al., 2021; Song et al., 2022).
(iii) Regression models, including GPs, struggle with different scales and noise levels across tasks, which hurts warm-starting and optimization efficiency (Feurer et al., 2018a).
We propose a new meta-learning BO approach that can effectively transfer knowledge from re- lated tasks and scales to large datasets. Our method is inspired by the idea of likelihood-free BO (Bergstra et al., 2011: I; Tiao et al., 2021; Song et al., 2022), which replaces the surrogate model with meta-leamed classifier that directly balances exploration and exploitation without modeling the

---

Under review as a conference paper at ICLR 2023
objective function. That way, we elegantly avoid both the scalability and the scale sensitivity issues simultaneously. We make the following contributions:
(i) A novel probabilistic meta-learning model that uses Bayesian logistic regression and a probabilistic approach to learn feature representations from prior tasks.
(ii) A scalable BO technique with good anytime performance that combines meta-learning classifier and a likelihood-free aeeeeeiiuuiiii acquisition function.
iiii) Robust adaptation to new tasks by combining the meta-learned classifier with gradient boosting to correct prediction errors and Thompson Sampling for more explorationn.
2
RELATED WORK
Meta-learning Various methods have been proposed that improve the data-efficiency of Bayesian optimization (BO) by leveraging the information of previous observations from similar tasks. They apply meta-learning (Vanschoren, 2018) or transfer-learning (Weiss et al., 2016) depending on the context and have been proven effective in various applications (Andrychowicz et al., 2016; Finn et al., 2017). We refer to Vanschoren (2018) for an in-depth overview.
One line of work adapts the initial design to warm-start BO, either by reducing the search space (Perrone et al., 2019; Li et al., 2022) or reusing good configurations from similar tasks, where similarity can be based on hand crafted features (Feurer et al., 2014) or learned with Neural Networks (NNs) (Kim et al., 2017). Alternative approaches estimate the usefulness of a given configuration both the current and prior tasks based on heuristics ( (Wistuba et al., 2015) or learning-han -based (Volpp transfer-learning to modify the probabilistic surrogate
al., 2020) methods. Other approaches model, for instance using a multi-task GP (Swersky et al., 2013; Tighineanu et al., 2022), an additive GP model (Golovin et al., 2017; Marco et al., 2017), or weighted combinations of independent GPs for different tasks (Schilling et al., 2016; Wistuba et al., 2018; Feurer et al., 2018a).
Several methods sinultaneously learn the initial design and modify the surrogate model. Springenberg et al. (2016) apply task-specific embeddings for BO and use Bayesian NN as the surrogate model, which are computationally expensive and hard to train. Perrone et (2018) propose Adaptive Bayesian Linear Regression (ABLR), which uses a NN to learn a shared feature representation across tasks with task-specific BLR layers improve scalability and adaptability. However, both these methods are sensitive to changes in the scale and noise level across datasets. To tackle this, Salinas et al. (2020) propose Gaussian Copula Process Plus Prior (GC3P), which transforms the data response values via the empirical CDF, and fits a NN across all prior tasks. This NN is used to warm-start the optimization and predict the mean for a GP on the target task. Despite its robustness, the use of a GP surrogate still limits its applicability on high-dimensional problems.
Our meta-learning method is closely related to Bayesian optimization with NNs and embedding reasoning (BANNER, Berkenkamp et al. (2021)), which uses meta-learning model based NN to learn a latent representation and a task-specific BLR layer, similar to ABLR. However, the model output is divided into iesk-independent mean and task-specific residual prediction learned by BLR layer. In this paper, we introduce a classifier variant of this meta-learning model and combine it with likelihood acquisition function.
Likelihood-fri Bayesian Optimization Bayesian optimization does not require an explicit model of the likelihood of the observed values (Gamett, 2022). Tree-structured Parzen Estimators (TPE, Bergstra et (2011)) phrase BO as a density ratio estimation problem (Sugiyama al., 2012) and use the density ratio over 'good" and 'bad" configurations as an acquisition function without probabilistic regression model. Tiao al. (2021) estimate the density ratio through class probability estimation (Qin, 1998), which is equivalent modeling the acquisition function with binary classifier. Likelihood-Free BO (LFBO, Song et al. (2022)) improves upon this by weighting the observations.
Likelihood-fiee free BO approaches address two drawbacks of traditional Ga-based BO methods: the computationally expensive inference and the lack of flexibility due to the strong assumptions of most kernel methods. Rather than modeling the objective function, likelihood free BO methods can use deterministic classifiers to separate good and bad configurations resulting in scale-invariant models
2

---

Under review as a conference paper at ICLR 2023
and allows the application of any binary classification method (Tiao al., 2021; Song et al., 2022). We leverage this flexibility to create new meta- leatanlnnning classifier, which yields a scalable method that is robust to heterogeneous scales across datasets.
3 PROBLEM STATEMENT AND BACKGROUND
In this section, we introduce our problem setting and introduce related methods.
3.1 BAYESIAN OPTIMIZATION
Bayesian optimization (BO) aims to optimize black-box function f(x) X + R over æ E X. At each step n. BO proposes a nn, obtains a noisy observation yn. = f(x) - Cn The proposal is based on probabilistic surrogate model M and all previous observations Pn = { Vi 1=1 The models posterior prediction p(y æ, 1 combined with the acquisition function a Dn quantifies the utility of each input and 2, = arg max (x;;; Dn-1)- Typically, BO algorithms use Gaussian Process model and assume 6, ~ ~N(0,, (?), for some unknown but fixed variance.
Setwwlege-Grrdieen (Frazier et al., 2009), Predictive Entropy Search (Hernández-Lobar Lobato et al., 2014) and Max-value Entropy Search (Wang & Jegelka, 2017) are defined as an expected utility function Uw: T) usually with a threshold that heuristically balances exploration and exploitation. For example, the prevalent Expected Improvement (EI, Močkus (1975)) acquisition function has ((r := max(r- - y, 0) while the Probability of Improvement (PI, Kushner (1964)) has U (y;:) := 1 > 0). The expected utility over the posterior belief from the surrogate model p(y æx Dn) is given by
Most acquisition functions, including
" (z:; Dn, T) Eywptvl a. (U = Ulu:
U(r:)))(u f)y x, Dr)dy,
(1)
where the threshold is usually chosen the lowest observed function value, i.e., minp, yi-
3.2 LIKELIHOOD-FEE ACQUISITION FUNCTIONS
Likelihood- acquisition functions model the belief of a candidate being promising instead of explicitly computing the likelihood of outcomes via the model's posterior p(y x., DN) ). One of the first likelihood- free BO algorithms, called Tree-structured Parzen Estimators (TPE, Bergstra et al. (2011)), dismisses the surrogate for the outcomes and models the two densities ((x) p( S Dn) and g(x:) p( ly > T, Dx) instead. The threshold relates to the 7-th quantile of the observed y values via 7 = (T) == ply S T Dr,). The density ratio (DR) serves as the acquisition function a DR (x; T (())((((())))))
Tiao et al. (2021) propose to improve several aspects of TPE by directly estimating the DR rather than solving the more challenging problem of modeling two independent densities an intermediate step. Their approach, dubbed BORE, rephrases the DR estimation as a binary classification problem when using the training loss
CBORR DN Σ (kin log Co(ar,) + (1 k..) log(1 1 (o()))))
Here kin = 1y, < T) represents the binary class labels estimated by the classifier Co with learnable parameters 0. Specifically, they show DR (a:; DN,7) 7) x Ca(x).
Although Tiao et al. (2021) argue that BORE resembles El, assigning the same label to all obser- vations with y 7 regardless of the magnitude of improvement conforms to the definition of PI rather than EI, and might lead to conservative optimization with little global exploration (Garnett, 2022; Song et al., 2022). In practice, El has exhibits stronger and robuster performance than PI. Song et al. (2022) convert the expected utility of El into an optimization problem using variational representation and reformulate as classification problem with the following objective:
CLFBO (0: TN = -E -= p)) Px) [max(T
0) log (z:) + log(1 - Ca(x)))
(3)
The resulting method, dubbed Likelihood -Fee Bayesian Optimization (LFBO), can be seen as a weighted classification problem with noisy targets for only class k = 1, where the El utility function
3

---

Under review as a conference paper at ICLR 2023
Objective function and metta-data
Meta-learned acquisition function and Its Thompson semples
observations
peediction
ciscrvation y
relatad taok
farget function
30
sample
1.0
0.4
0.8
Illustration of meta-learning the acquisition function. Left: Observations (crosses) from 10 related tasks and the target task. The top performing observations (7 1/3) in each task shown in red, the rest in blue. Right: The top panel shows the approximated predictive distribution (see Eq. (7)) while the others show Thompson samples. MALIBO successfully identifies the promising areas, and the Thompson samples show variability in the meta-learned acquisition function.
Figure I:
max(T y, 0) weights the importance for observations below by their improvement. The minimizer of Eq. (3) is shown to be equivalent to the acquisition function (Song et al., 2022).
METHODOLOGY
In this section, we introduce our MetA-learning for L.kelihood-free BO method, dubbed MALIBO. incorporates a probabilistic meta-learning approach derived from BANNER into the LFBO framework. In particular, show how to convert BANNER's 's regression model into meta-learning classifier that warm-starts BO and combine it with gradient boosting (Friedman, 2001). This yields good anytime performance, more flexibility in selecting the model, and better adaptation to new tasks. We first explain the different components and then summarize our algorithm in Algorithm 1.
META-LFARNING
Meta-learnnng for optimization strives to extract information from previous tasks and use this to accelerate the optimization of new one. As all methods discussed in Section 2, we only consider the case of identical input spaces X' across all tasks, which simplifies the learning problem. Our goal is to learn a probabilistic model for the likelihood-free acquisition function from the meta-data, as illustrated in Fig. for a one-dimensional function.
Following the approach from Berkenkamp et al. (2021) and Perrone et al. (2018), our meta-learning classifier uses deterministic, task-agnostic model to map the input space into feature space, h(x)) = Φ of predefined dimensionality. In this feature space, we seek to find simple model able to classify the data on prior tasks using task specific adaptation. We propose to use Bayesian logistic regression with a minor modification. In addition to the standard linear model zz, where z, represents prior task the learned feature space, and the sigmoid function converting the linear model into class probabilities, introduce task agnostic mean function m (()) that acts non-linear bias. We provide an overview of our method in Fig. 2.
The meta-learning model g(-) m((())) learns global prior for the classification problem, and the specific embedding vectors 2, representing the necessary adaptations. For better inference on new tasks, we regularize the distribution of the values during training. We follow the technique used by Berkenkamp et al. (2021) and Saseendran et al. (2021) to bring 2; close the prior distribution p(2) (O, I), using modified Kolmogorov-Smino test and the covariance calculate the
4

---

Under review as a conference paper at ICLR 2023
Task agnostic learing g()
ResFFN h(-)
Mean prediction layer m(-)
Sigmoid function a(-)
Class prediction
Residual prediction layer z:
speciip embedding
Figure 2: Schematic representation of our meta-learning classifier. A Residual Feedfoward Network (ResFEN) maps the input æ; to a latent feature representation Φ. From this, global mean prediction m(()) and task-specific embedding 2 combined and translated into class prediction via sigmoid function.
disparity of two distributions. The loss used for training on the meta-data reads:
meeta
min
Σ T 1m1
CLFBO ((x,), xn) +
(4)
where we average the contribution of all prior-tasks and R the regularization term outlined by Berkenkamp et al. (2021) weighted by X.
4.2
TASK ADAPTATION VIA BAYESIAN LOGISTIC REGRESSION
After training the motelm we need adapt our predictions to each new task. Hence, we must estimate a value for z that yields good classification results' for the predictions C(x) (g(x) + z h(=)). While we could greedily optimize Eq. (3), we pursue a Bayesian approach, Bayesian logistic regression in particular, to capture the uncertainty in the task-specific z. Especially with few observations on the new task, using a single z vector can introduce a strong bias
Besides stronger exploration in the early phases of optimization, the Bayesian approach allows us to employ Thompson sampling (Thompson, 1933) for x, as shown in Fig. 1. We believe this to be valuable strategy for parallelization briefly explored in Appendix E. Kandasamy et al. (2018) showed that this bypasses the sequential scheme of traditional BO, without introducing the common computational burden of more sophisticated methods. See Garnett (2022) for more details.
Bayesian logistic regression is simple yet powerful classification model with Bayesian treatment. Although exact Bayesian inference is intractable, the Laplacian and probit approximation allow us to approximate the weights' posterior and the predictive distribution respectively while remaining reliable and scalable (Bishop & Nasrabadi, 2006: Murphy, 2012). The Laplacian method fits Gaussian distribution around the and matches the second order derivative at the optimum.
(MAP) estimate of the weights distribution
In the first step, we obtain the MAP estimate by maximizing the posterior of our classifier C parameterized by To be consistent with the regularization used during meta-traien veta-training, we use standard, isotropic Gaussian prior for the weights: p(z) N= mo, Eo), with mean mo 0 and covariance matrix Eg Given observations DN. the log posterior likelihood w.r.t. z given by
N
MALIBO (z mo)
- (m- +
Σ (kn(T - u) log C(x.) + log(1 ((x))) (5)
==1
and defines the MAP estimate of the weights via ZMAP arg min m MALIBO, As second step for the Laplace approximation, we compute the negative Hessian of the log posterior
Σy' VV InVlnp2
E
Σ
k
(6)
which serves the precision matrix for the approximated posterior q(z) AV zAAP, EN). To finally compute the approximate predictive distribution, we need to marginalize w.r.t. p(2 (N)):
Cl:; g(x), ( DN) p(k = g(()),) 2)p(2 (D )dz ~ f | f m)()))((2)))))
In contrast the meta-training, we only infer but keep the task agnostic model g()) fixed.
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1: MALIBO: Meta-learning for likelihond-free Bayesian optimization
Meta-learning:
Input: Meta-datasets Di for tasks & 1, ....., T, proportion 7 E (0,
Generate binary class labels for meta-data using ?: 1(v S 7), where 2 (1))
Learn the meta-lcaming model C(:; ( by optimizing L (Eq. (4))
Bayesian optimization with Meta-learning:
Input: Fixed meta-learned model g())
Obtain the first best guess æo from the meta-learned model = (( ((x) + c)}
For the gradient boosting variant, use C(-) <- GB(C(-)) as the classifier
while Has budger de
Estimate ZMAP by optimizing [MALIBO (Eq. (5)) w.r.t. z
Update precision matrix D,, (Eq. (6))
if T'hompson sampling then
Sample ~ MVN(ZMAP Σ)
x' arg max ((n: (()),
11
else
12
x. arg max C(z: D) with probit approximation (Fq. (25))
D - DU DU 1 f(x') + e)}
Although no closed form solution for the integral in Eq. (7) is available due to the logistic activation, we can evaluate the predictions with probit approximation, which makes use of the similarity of the sigmoid function and the probit function. See Appendix B for further details.
4.3 GRADIENT BOOSTING AS RESIDUAL PREDICTION MODEL
While the learned feature representation warm-starts the optimization, the model gains more knowl- edge about the new task with growing data and needs to adapt. Eventually, the accuracy of C(xx) will saturate, as the parametric model has only limited flexibility and is ultimately limited by the amound and the quality of the meta-data. We expect suboptimal classification performance especially in cases of little meta-data or when a large discrepancy between the training data and the meta-data distribution exists?. More depth studies are shown Appendix D.2.
To counteract this, we propose to combine our method with gradient boosting (GB) (Friedman, 2001). In every iteration, after updating the Bayesian logistic regression model, we train gradient boosting model (regression trees) on D, without leveraging any meta- meta-leamed features and use our classifier as the first one in the ensemble. This way, GB only corrects where our meta neta-model inaccurate and allows for fine-tuning and better convergence.
EXPERIMENTS
In this section, we describe the experiments conducted to empirically evaluate our method. For the choice of problems, we focus automated machine learning (AutoML), i. e. hyperparameter optimization (HPO) and neural architecture search (NAS). Besides that, we evaluate our methods on synthetic functions to show the robustness against multiplicative noise. Throughout all benchmarks, we show that MALIBO clearly improves upon LFBO's performance.
We compare our method against state-of-the-art baselines across all problems. We picked random search (Bergstra & Bengio, 2012), the gradient boosting variants of BORE (Tiao al., 2021) and LFBO (Song et al., 2022) and BO with GPs as baselines without meta-learning and chose ABLR (Perrone et al., 2018), RGPE (Feurer et al., 2018a) and GC3P (Salinas et al., 2020) as representative consider the performance of our
algorithms with meta-learning component. Additionally, method without any additional components (MALIBO), with gradient boosting (MALIBO (GB)), with Thompson sampling (MALIBO (TS)), and with both (MALIBO (GB-TS)).
A large missmatch between traning and test data can arise here when the all cluster around an optimum after many iterations, but the meta-data consisted of IID points all tasks.

---

Under review as a conference paper at ICLR 2023
101
10F
Inmasiiaie segurnt (agl)
1.0
10%
10"4
70
10"4
10"1
Stage
(a) Protein
(b) Parkinsons
Randect
MALLIB
BOPS
MALIRO
0.55
Innushiite ropest (eg)
10-7-
10-
Iunedaate
10
10-1-
10"1
(c) Slice
(d) Naval
Figure 3: Immediate regret for different BO algorithms on the HPOBench neural network tuning
problems (D = 9) for 4 datasets. The optimization objective in this benchmark is the validation loss.
We use immediate regret to quantitatively measure performance of different methods, which by definition is the absolute error between the global minimum and the lowest function value obtained so far. For all benchmarks, we report the results by mean and standard error across 100 random runs.
We implemented ABLR, RGPE and MALIBO in Py Torch (Paszke et al., 2019), and used scikit- learn (Pedregosa et al., 2011) for gradient boosting. As the meta-learning model for MALIBO, considered a 4-layer, 64-umit residual feedforward network ResFFN-4-64, For more details we refer Appendix G. For GC3P, we used the authors' open source implementation" The algorithm samples five candidates from a meta-learned NN model before building task-specific Copula process while BORE and L.BOO sample random configurations for gathering global information from the target task before training model. Thanks to the meta-learned acquisition function, MALIBO starts the optimization at the point with highest acquisition function value. For all likelihood BO methods the required hyperparameter y, we set 7 1/3, following Tiao et al. (2021) and Song et al. (2022).
Neural network tuning (HPOBench) This benchmark represents joint NAS and HPO for two-layer feed-forward regression network on four popular UCI datasets (Dua & Graff, 201 7). tabular benchmark (Klein Hutter, 2019;
The search space is dimensional and available Eggensperger et al., 2021) with total of 62,208 unique configurations. The optimization objective in this benchmark is the validation mean squared eтror after training with the corresponding network configuration (see Appendix H.1 for more details). As meta-data for each dataset, we randomly sampled 512 configurations from each the remaining three. Fig. 3 shows the strong -aamtsttrring performance of all MALIBO variants. For the protein and Slice dataset, MALIBO (TS-GB) exhibits the best final performance. For Parkinsons and Naval, the problem characteristies seem favour the most basic MALIBO variant. GC3P performs very competitively often being the best method once the Copula process fitted, but its final performance usually matched or surpassed by BORE and LFBO. For all datasets, ABLR performs poorly, presumably due to the small number of tasks and the fact that the meta- tetenirriinng seems to overfit to the meta-data the first BO iterations. The GP based methods barely outperform RS, indicating that GPs fail to model the loss landscape.
Neural architecture search (NASBench201) NASBench201 (Dong & Yang, 2020) considers designing a neural cell with discrete parameters totaling 15, 625 unique architectures. evaluated on CIFAR-10, CIFAR-100 (Krizhevsky, 2009) and ImageNet-16 (Chrabaszez et al., 2017). The

---

Under review as a conference paper at ICLR 2023
38"
3271
12
(a) CIFAR-10
(b) CIFAR-100
(c) ImageNet-16
Figure 4: Immediate regret for different BO algorithms the NASBench201 neural architecture
search problems (D 6) on datasets. optimize the validation accuracy for this benchmark.
optimization objective is validation accuracy after training the corresponding network architecture. We refer to Appendix H.3 for more details. We randomly sampled 512 configurations from each of two other datasets for meta-training. From the results shown in Fig. 4, we that methods start again with the lowest average regrets, and the gradient boosting variants converge the fastest compared all baselines across all datasets. The graphs show that Thompson sampling (TS) varaints shows slightly slower convergence in the beginning, but they can outperform their non-TS counterparts. GC3P remains strong competitor, but seems converge prematurely. ABLR again suffers from few prior-tasks, and the other GP based methods struggle again with the loss surface.
ML algorithms tuning (MLBench) We picked algorithms (SVM, LogReg, XGBoost, Random- Forest and MLP) from the ML benchmark suite provided by Eggensperger et al. (2021) evaluated on 20 OpenML tasks (Vanschoren et al., 2014), except for MLP with only tasks. The search spaces dimensions range from 2 (SVM) to 5 (MLP) with the validation error as the objective. We refer to Appendix H.2 for more details. For meta-data, we randomly sampled 128 configurations from each task except the current target one (which changed across repetitions). As seen in Fig. 5, all MALIBO variants continue to demonstrate strong warm-starting performance and similar performance on all benchmarks, except for RandomForest, where gradient boosting variants better. ABLR finally shows its potential with more meta-tasks and is competitive on some benchmarks, but fails to converge on of them. The warm-starting of GC3P fluctuates from on-par with MALIBO (SVM, LogReg) to essentially random guesses (MLP, XGBoost). RGPE demonstrates quick adaptation, but the GP seems limit convergences again.
MACER
608
MN WWTCCCTTT
Sxeee
5uge
Stage
(a) MLP
RandomForest
XGBoost
1074
(d) SVM
(e) LogReg
Figure 5: Immediate regret for different BO algorithms on the HPOBench hyperparamter tuning problems for 5 different machine learning algorithms. We specifically optimize the (1 accuracy) for this benchmark.
8

---

Under review as a conference paper at ICLR 2023
10P
10P
10
UTEO
(a) Branin (E 0.0)
(b) Branin (e 0.1)
(c) Branin (E = 1.0)
Figure 6: Immediate regret for different BO algorithms Branin function ensembles (D = 2) with
different levels of multiplicative noise.
Robustness against different noise level To test the robustness of our method against different noise levels in the data, we use synthetic function ensembles introduced by Berkenkamp et al. (2021) with multiplicative noise. We focus on the Branin function ensemble (Dixon, 1978), which is a two-dimensional problem with 3 local minima. Their location and the global minimum varies across different functions in the ensemble. See Appendix H.6 for more details.
To avoid biasing this experiment towards single method, we decided to use a heteroscedastic noise that is incompatible with any assumptions about the noise of any method. In particular, this violates the GP methods' and ABLR's assumption of homoscedastic, Gaussian noise. GC3P makes a similar assumption, but after the nonlincar transformation applied to the y-values, which does not translate to well-known noise model. BORE, LFBO and by extension all MALIBO variants, make no explicit noise assumptions, but will optimize for the best mean. We choose a multiplicative noise, i.e. evaluate E {0, 0.1, 1.0). The y = f()) (1 + n)), where n N(O, 1). To test the robustness, noise corrupts observations with large values more, while having a smaller effect on lower function values. For mneta-training, we randomly sampled 128 noisy observations from 256 functions in the ensemble. We show our results in Fig. 6, where we can see across all magnitudes of multiplicative noise, our method still learns meaningful prior for the optimization. As the noise level increases, the performaces of all methods degrade. with ABLR and GC3P performing worst for the largest noise. The GP based methods, especially RGPE do well on such smooth functions and handle the noise surprisingly well. In comparison, BORE, LFBO and MALIBO show the least performance losses with growing noise on a function ensemble where GP methods excel.
CONCLUSION
We introduced Meta-learning for Likelihood-free BO (MALIBO), a novel meta-learning optimiza- tion algorithm that is computationally efficient and robust to varying scales of the observations and heterogeneous noise. By directly modeling the acquisition function from observations, the method makes fewer assumptions about the data and noise distributions. Coupled with meta-learning, it leverages information from prior tasks for more sample efficiency. To ensure adaption to new tasks, possibly different from prior ones, incorporate our model into gradient boosting, transitioning from a meta-learning driven model towards a specialized one on the current task. Empirical re- sults demonstrate superior performance on both HPO and NAS benchmarks, as well as synthetic benchmarks with heteroscedastic noise.
Despite the promising experimental results, some limitations of the method should be noted. As discussed by Tiao et al. (2021), the threshold parameter t in likelihood- -ree BO algorithms, which controls the exploitation and exploration trade-off, should be treated more carefully. One might consider probabilistic treatment for this hyperparameter. We also observed over-confident predictions in our experiments, for example the Forrester ensemble in Appendix F, where not all runs find the true optimum. This is especially important for problems with more complicated loss landscapes, but mitigation strategies exist.
Directions for future work include extensions to parallel BO with Thompson sampling (Kandasamy et al., 2018), multi-fidelity optimization for HPO problems (Falkner et al., 2018), multi-objectivve optimization (Hemandez--obarl et al., 2016), BO with automatic stopping (Makarova et al., 2022) and Bayesian deep active learning (Gal et al., 2017).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gómez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas Learning learn by gradient descent by gradient descent. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 3988-3996 Curran Associates Inc., 2016.
James Bergstra and Yoshua Bengio. Random search for hyper-parametce optimization. Mach. Learn. Res., 13(null):281-305 2012.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for timization. In Advances in Neural Information Processing Systems, volume 24, Curran Associates,
op-
Inc., 2011.
Felix Berkenkamp, Anna Eivazi, Lukas Grossberger, Kathrin Skubch, Jonathan Spitz. Christian Daniel, and Stefan Falkner. Probabilistic meta-learning for bayesian optimization, 2021.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning. Springer, 2006.
Richard H. Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing, 16(5):1190-1) 1208, 1995.
Patryk Chrabaszez, 1ly Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets, 2017.
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learn- ing by exponential linear units (elus). 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, 2016.
Laurence Charles Ward Dixon. The global optimization problem. an introduction. Toward global optimization, 2:1 2:1-15, 1978.
Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations, 2020.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL htttttt////cchtve ics.uci www.uuci..euunnu.
Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, Rene Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter. HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO. In Thirty-fitth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter opti- mization at scale. Proceedings of the 35th International Conference on Machine Learning, pp.
2018.
Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Using meta-learning to initialize bayesian optimization of hyperparameters. In Proceedings of the 2014 International Conference on Meta-Learning and Algorithm Selection Volume 1201, MLAS' 14, pp. 3-10, Aachen, DEU, 2014. CEUR-WS.org.
Matthias Feurer, Benjamin Letham, and Eytan Bakshy, Scalable meta-learning for bayesian opti- mization using runking-weighter gaussian process ensembles. In ICML 2018 AutoML Workshop. 2018a.
Matthias Feurer, Benjamin Letham, Frank Hutter, and Eytan Bakshy. Practical transfer learning for bayesian optimization, 2018b.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1126-1135. PMLR, 06-11 Aug 2017.
10

---

Under review as a conference paper at ICLR 2023
Peter Frazier, Warren Powell, and Savas Dayanik. The knowledge sooledgg-gradiiet policy for correlated normal beliefs. INFORMS journal on Computing, 21(4):599 613, 2009.
Jerome H. Friedman. Greedy function approximation: gradient boosting machine. The Annals of Statistics, 29(5):1189 - 1232, 2001.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1183-1192. PMLR, 2017.
Roman Gamett, Bayesian Optimization. Cambridge University Press, 2022.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. Google vizier: service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, pp. 1487-1495. Association for Computing Machinery, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Daniel Hernandez-Lobenn Jose Hernandez-Lobato, Amar Shah, and Ryan Adams. Predictive entropy search for multi-objective bayesian optimization. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1492-1501. PMLR, 20-22 Jun 2016.
José Miguel Hernández- Viernández-Lobato, Matthew W Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. Advances neural information processing systems, 27, 2014.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated Machine Learning: Methods, Systems, Challenges. Springer Publishing Company, Incorporated, 2019.
Kirthevasan Kandasamy, Akshay Krishnamurthy, Jeff Schneider, and Barnabas Poczos. Parallelised bayesian optimisation via thompson sampling. In Amos Storkey and Fernando Perez-Cruz (eds.), Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pp. 133-142, PMLR, 2018.
Jungtack Kim, Sachoon Kim, and Seungjin Choi. Learning to warm-start bayesian hyperparameter optimization, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Aaron Klein and Frank Hutter. Tabular benchmarks for joint architecture and hyperparameter optimization, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. pp. 32-33, 2009.
H. Kushner. A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise. Journal of Basic Engineering, 86(1):97-106, 03 1964.
Yang Li, Yu Shen, Huaijun Jiang, Tianyi Bai, Wentao Zhang, Ce Zhang, and Bin Cui. Transfer learning based search space design for hyperparameter tuning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, pp. 967-977, 2022.
Anastasia Makarova, Huibin Shen, Valerio Perrone, Aaron Klein, Jean Baptiste Faddoul, Andreas Krause, Matthias Seeger, and Cedric Archambeau. Automatic termination for hyperparameter optimization. In First Conference on Automated Machine Learning (Main Track), 2022.
Alonso Marco, Felix Berkenkamp, Philipp Hennig, Angela P. Schoellig, Andreas Krause, Stefan Schaal, and Sebastian Trimpe. Virtual vs. real: Trading off simulations and physical experiments in reinforcement learning with bayesian optimization. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1557-1563. IEEE Press, 2017.
11

---

Under review as a conference paper at ICLR 2023
J. Močkus. On bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference Novosibirsk, July 1-7, 1974, pp. 4000-00.. Springer Berlin Heidelberg. 1975.
Kevin P. Murphy. Machine Learning: A Probabilistic Perspective. The MIT Press, 2012.
XuanI Nguyen, Martin J. Wainwright, and Michael I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11)55847-5661 nov 2010.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, segh-performacce deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Valerio Perrone, Rodolphe Jenatton, Matthias W Seeger, and Cedric Archambeau. Scalable hyper- parameter transfer learning. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Valerio Perrone, Huibin Shen, Matthias Seeger, Cédric Archambeau, and Rodolphe Jenatton. Learning search spaces for bayesian optimization: Another view of hyperparameter transfer learning. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., 2019.
Jing Qin. Inferences for case-control and semiparametric two-sample density ratio models. Biometrika, 85(3):11-6600 1998.
Carl Edward Rasmussen. Gaussian Processes Machine Learning. Springer Berlin Heidelberg. 2004.
David Salinas, Huibin Shen, and Valerio Perrone. A quantile-based approach for hyperparameter transfer learning. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8438 8448. PMLR, 2020.
Amrutha Saseendran, Kathrin Skubch, Stefan Falkner, and Margret Keuper. Shape your space: A gaussian mixture regularization approach to deterministic autoencoders. Advances in Neural Information Processing Systems, volume 34, pp. 7319-7332. Curran Associates, Inc., 2021.
Nicolas Schilling, Martin Wistuba, and Lars Schmidt-Thieme. Scalable hyperparameter optimization with products of gaussian process experts. In ECMIJ/KDD, 2016.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1): 148-175, 2016.
András Sobester, Alexander Forrester, and Andy Keane. Engineering design via surrogate modelling: practical guide. John Wiley & Sons, 2008.
Jiaming Song, Lantao Yu, Willie Neiswanger, and Stefano Ermon. A general recipe for likelihood free Bayesian optimization. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp. 20384-20404. PMLR, 2022.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
Masashi Sugiyama. Taiji Suzuki, and Takafumi Kanamori. Density Ratio Estimation in Machine Learning. Cambridge University Press, 2012.
12

---

Under review as a conference paper at ICLR 2023
Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimization. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-294 1933.
Louis C' Tiao, Aaron Klein, Matthias W Seeger, Edwin V. Bonilla, Cedric Archambeau, and Fabio Ramos. Bore: Bayesian optimization by density-ratio estimation. In Proceedings of the 38th
International Conference Machine Learning, volume 139, pp. 10289-10300, PMLR, 2021.
Petru Tighineanu, Kathrin Skubeh, Paul Baireuther, Attila Reiss, Felix Berkenkamp, and Julia Vinogradska. Transfer learning with gaussian processes for bayesian optimization. In Proceedings of The 25th International Conference Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 6152-6181, 2022.
Joaquin Vanschoren. Meta-learning: A survey, 2018.
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. Openml: Networked science in machine learning. SIGKDD Explor. Newsl., 15(2):49-66 60, jun 2014.
Michael Volpp. Lukas P. Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta- Meta-leaming acquisition functions for transfer learning in bayesian optimization. In International Conference on Learning Representations, 2020.
Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient bayesian optimization. In International Conference on Machine Learning, pp. 3627-3635. PMLR, 2017.
Karl R. Weiss, Taghi M. Khoshgoftaar, and Dingding Wang. A survey of transfer learning. Big Data, 3:9, 2016.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Learning hyperparameter optimization initializations. 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pp. 1-10, 2015.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thiene Scalable gaussian process-based transfer surrogates for hyperparameter optimization. Mach. Learn., 107(1):43-78, 2018
13

---

Under review as a conference paper at ICLR 2023
A LIKELIHOOD-FREE ACQUISITION FUNCTION
For completeness, we provide the proofs and derivations from Bergstra et al. (2011), Tiao et al. (2021), and Song et al. (2022). Recall from Eq. (1) that the expected utility function is defined as the expectation of the improvement utility function U (y; T) over the posterior predictive distribution ply z, D.). For the specific expected improvement (EI) acquisition function, where the utility function is (g: T) = max(T - y. 0). the function reads:
Uns x, D,)dy ax;; Dz, 7) =- Bp(y|a,D.) =
(T - y)ply x. Dr)dy
(8)
J.
pc (T- v)p(> y. Dr)p(y! Dx)dy,
p( (Dn)
We follow the prove from Bergstra et al. (2011) and Tiao et al. (2021) and consider {(x) = p(a y Pn) and g(x) p(æ T, Dn). The denominator of the above equation can then be written as:
p(a D,) = | p(æ ly, Dr)p(y) Dn)dy
= =((x) [" p(y dy + g(x) [. plu Dn.dd
(9)
=>((a)) + (1 )))(())
where : ((T)) ply S Da). The numerator can be evaluated as:
max(T v/ 0)p(x y, Dr)p(y) DN)dy = {(ar) J.- max(T v. -y.0)p(1 Dn.ddy
(10)
= =((x)) { p(y (Dn)dy - f(xz) 1.- up(y P,)dd
(11)
{ yp(y DD))dy
=TT((x) E(xx)
(12)
K ((x))
(13)
where K = - yp(y Da)dy. Therefore the El acquisition function is equivalent to the y-relative density ratio up to constant K,
((x))
c:(:: DN, T)
(14)
y((x) (1 7)((x)
exprectdd impravemment
density ratio
Intuitively, one can think of the configuration æ whose corresponding y S T as good configurations, and the those with y > 7 are bad configurations. Then the density ratio can be interpreted as the ratio between the model's belief that configuration being good and a bad configuration.
For Tree-structured Parzen Estimators (TPE, Bergstra et al. (2011)), they first select the 7 as hyperpa- rameter and estimate this density ratio by explicitly modelling ((x) and y(x) using kernel density estimation. As for BORE by Tiao et al. (2021), they model the density ratio by class probability, le 0).
where l(x) p( k 1) and g p(x
Song et al. (2022) proof that density ratio acquisition function is not always equivalent to EL. Bergstra al. (2011) and Tiao al. (2021) claims that Eq. (10) hold true, which is ((x) times value independent of x. Bergstra et al. (2011) assumes {(x) independent of y once S 7 and therefore can take {(x) directly out of the integral.
Nevertheless, p(x y S T, D.)) is still dependent on y even if y < T. because it is a conditional probability condition y. They confused with depending on and depending on y S where these
14

---

Under review as a conference paper at ICLR 2023
two statements are different and mean differently. By just satisfying the condition it does not necessarily mean that it become independent on y. From the definition conditional probability:
P(x, p(z, y D p y S T,,,) -- plu Dn)dy
+ p(x y, Dn).
(15)
they are not equivalent. Intuitively, to understand this, is for example, the probability of the configu- ration æ still depends on the its corresponding value y even though y < т. Another interpretation for this would be, density ratio acquisition function would treat all (æ,, y) pairs above it with equal importance, because does not depend on any more if y S T according to the independence assumption. In fact, expected improvement weighted the importance of (æz y) pairs by how much lower than T.
To tackle this issue, Song et al. (2022) propose to estimate the density ratio via variational f-divergence estimation (Nguyen et al., 2010). They provide a variational representation for the expected utility function at any point æ; provided the samples from some distribution ply æ),, which replace the integration with a variational objective function:
Ep(ula) (U (v:
arg max Ep(l UU (6)
(16)
=E(0,oc)
where the utility function U is non-negative, and f: [0, oc) IR is a strictly convex function, and f is the convex conjugate of f. This acquisition function does not model distributions with probability but only samples from the observations Dn.-
They consider their acquisition funetion ! SD., (x), and state that the acquisition function can be written as:
SD., (z) = arg max ED,
(17)
This means by optimizing a variaitional objective the search space X', we can recover an expected utility acquisition function over which makes LLPBO equivalent to expected utility acquisition function. For practical purpose, they choose a specific convex funciton f: f(r) = r log - +og + for all r > 0. For their acquisition function a L.RO they consider:
"
LFBO ( Dn: = (a)0(()))))
(18)
where Čo,,, is the maximizer of some objecetive over C : + 1). By applying this f and Eq. (18) to Eq. (17), the objective for maximization objective C become:
E (a.y))p(a..)) [U r) log C(x) + log(1 C(+)))
(19)
15

---

Under review as a conference paper at ICLR 2023
B PROBIT APPROXIMATION
Let's set a = कT + (Φ)) and the distribution of a would be a Gaussian N(a Ma, oa), where pa and oa are respectively:
=fooaadd== f Na = RRaa] = p(a)ada- = f ((=)(==(d) +
m(()) dz = =aPP + m(()
(20)
0² = varla] = pla))a²-E[eer da
a(2)((2² ob m(Ø))² ) (m ml m( dz
(21)
b'ENd
Thus our approximation to the predictive distribution becomes:
p(k (())) o(a)N(a Na, (a.o%) da
(22)
To evaluate the integral in Eq. (22), obtain good approximation by making of the close similarity between the logistic sigmoid function (a) and the probit function, which given by the cumulative distribution the standard Gaussian ((a) = - N/O 0, d0:
f oa) d
/ (X-2 + 02)1/2
(23)
We apply the approximation a(a) = ((Aa) to the probit functions appearing on both sides of the equation:
/ a(a)N((a (د da a(1 + NO mo² /8)-
(24)
Therefore we obtain predictive distribution in the form:
p(k 19(x), DN) a(1 /82 pa)
(25)
where is Eq. (20) and o is Eq. (21).
16

---

Under review as a conference paper at ICLR 2023
C RUNTIME ANALYSIS
The experiments in Section 5 show the performance over optimization steps. To be complementary. we demonstrate the same results from different perspective, namely, we report the immediate regrets as function of estimated wall-clock time. To obtain the realistic wall-clock time, we accumulate the time to optimize for corresponding BO methods and the recorded runtime for configurations in the benchmarks. Notice that, all the methods run for the same number of steps in an experiment.
As the results are shown in Figs. 7-9, MALIBO and its variants attain the best warm-starting performance across all benchmarks and constantly achieve the lowest regrets with the same amount of time. LFBO and BORE are two competitive methods in terms of end performance, but both need quite some time to catch up the regrets of MALIBO and its variants. GC3P is the method with closest time performance as MALIBO most of the benchmarks. However, their performance is not stable as MALIBO, especially for NASBench201 and some of the MLBench problems, where the meta-learning fail to warm-start the optimization. Similarly, the meta-learning of RGPE and ABLR do not deliver any advantage over the non-meta-learning baselines and thus end up with close performance as normal GP.
To further investigate the time efficiency of MALIBO, we illustrates the runtime of the optimization algorithms for each step in Fig. 10. The runtime for MALIBO and its variants is the fastest among all the meta-leaming methods, while only slightly slower than the non-meta-learnin likelihood- methods, namely BORE and LFBO. Due to the increasing amount of observations, the runtime of almost all the methods grow over iterations, especially for RGPE and GP, where the growths are the most significant. Although ABLR and GC3P are around a order of magnitude slower than MALIBO the beginning, but their runtimes remain stable over steps.
100
Inmeedatte sogre Oogl
10'
Inmesthate rogertt Oog))
10
70
VALEER MALERSOMM)
2007
1823
10
Botimated wall clock Eie
Extinated clock
(a) Protein
(b) Parkinsons
104
Inttediale regret (lg))
Inmedatto papettatttrttttoo..o. (1gg)
10-2
10
10
10-1
10-4
(c) Slice
(d) Naval
Figure 7: Immediate regrets of different BO algorithms on the HPOBench neural network tuning problem. Each algorithm runs for 500 iterations and we show the corresponding estimated wall-clock time on the x axis in log scale.
17

---

Under review as a conference paper at ICLR 2023
00
109
(a) CIFAR-10
(b) CIFAR-100 100
(c) ImageNet-16
Figure 8: Immediate regrets of different BO algorithms on the NASBench201 neural network architecture search problem. Each algorithm runs for 200 iterations and we show the corresponding estimated wall-clock time on the a axis in log scale.
(a)
(b) RandomForest
(c) XGBoost
1750
Dateaaied wallchec
Datiuated walceet
(d) SVM
(e) LogReg
Figure Immediate regrets of different BO algorithms on the HPOBench hyperparameter tuning for machine learning algorithms. Each algorithm runs for 72 iterations and we show the corresponding estimated wall-clock time on the x axis in log scale.
Optimization used time (5)
109
10-1
10P
10-2
Optimization
10
Bunkat
MALIBO
RCPE
100
200
300
400
500
10
20
30
40
50
00
70
Stepe
Steps
(a) HPOBench Slice
(b) MLBench MLP
Figure 10: Runtime of different BO algorithms over optimization steps. We show the typical results for benchmarks and plot the medial inter-quantiles to remove outliers.
18

---

Under review as a conference paper at ICLR 2023
D ABLATION STUDIES
We conduct ablation studies to show how the different components of MALIBO help for the optimization. To be specific, first demonstrate the meta-learned features representation for two different function in Appendix D.I to provide an intuitive visualization for the expressiveness of our meta-learning model. To answer the questions of what components and how they help the optimization, we show the experiments of MALIBO and its variants running on 4 different synthetic function benchmarks without any meta-learning Appendix D.2.
D.I LATENT FEATURE ANALYSIS
We show how meta-learning model learns feature representation from meta-data. The latent features Φ can be considered as basis functions and are supposed to represent the structure of the meta-data distribution. With the properly learned features Φ, the mean layer and the task embedding layer can combine them to obtain a function with similar structure to the meta-data while matching the shape of target function.
In order to learn a effective feature representation, one should capture both the local and global structure of the function. Therefore, we select two types of function study the effectiveness of features learning for MALIBO: i) Forrester functions (Sobester et al., 2008) with two very likely positions for the global optimum, which has rich local structure. ii) quadratic functions. where the functions share certain global shape, but the optima could be located anywhere in the search space. For more details on the synthetic functions and the generation of meta-data, we refer to Appendix
The results for these two synthetic functions are shown in Fig. 11 and Fig. 12 respectively. On one hand, we can see that Fig. 11 the features learned by MALIBO has either maximum or minimum around the two likely optima, which means the model successfully infers the local structure from meta-data. On the other hand, Fig. 12 shows that, even without a clear location optima, the features still learns the shape of quadratic functions.
Orjecttre Dinction: and meta data
Eeature
Stattrro
L5
observctions
cleersttions 2
toka
GU
300
0.25 0.50
1/0
0.25 0.5
1/0
Dhattre 3
L.S
L.D
0.5
-10
60-
-20
-0.5
Tuacioon
2000 predicton
0.8
0.4
2.3
0.2
034 3.5 0.5
1000
0.25
039
0.55
1002
0.25
030
0.75
3.00
Figure 11: Left: Forrester functions with two very likely optima as target function and related-taske The learned acquisition function is shown below. Right: Meta-learned latent features from related- tasks. The latent features show the model successfully infer the location of two optima, resulting in a acquisition with modes around the same locations.
19

---

Under review as a conference paper at ICLR 2023
Dbjecttre finction and mela-data
Seaturn 1
featume
1.0
0.2
tack
Seatorn
feature
0.5
-0.55
-1.0
-3.5
prodictio
0.5
0.-
-0.5
-1.0
-1.00
-0.75
-9.30
-0.25
0.00
0.25
0.70
0.75
1.00
-1.0
-35
0.0
05
1.5
-1.
-03
00
05
1.0
Figure 12: Left: Quadratic functions with varying optima as target function and related-tasks. The learned acquisition function is shown below. Right: Meta-learned latent features from related-tasks. The latent features show the model can learn the global structure shared across all the meta-data, even though there no clear position for optima.
20

---

Under review as a conference paper at ICLR 2023
D.2 EFFECTS OF GRADIENT BOOSTING AND THOMPSON SAMPLING
The vanilla MALIBO uses Bayesian logistic regression for task adaptation, which leverages the meta-learned feature. However, the performance depends heavily the quality of the data and the latent features learned from the data. In practice, the amount and the quality of the data are often not guaranteed. We introduce gradient boosting as residual model safeguard the optimization when little meta-data or large discrepancy between the training data and the meta-data distribution exists. Further, we apply Thompson sampling encourage exploration, which enables MALIBO to collect more information about the target function by exploring the search space efficiently. To show the usefulness of these components in MALIBO, we remove the effects of meta-learning by optimizing the target synthetic functions without any meta-training, and therefore the experiments will focus only on the effects on the gradient boosting and Thompson sampling.
We use four synthetic benchmarks for the experiments, namely quadratic, Forrester, Branin and Hartmann3 functions. We refer to Appendix F for more details of the synthetic functions. As the results shown in Fig. 13, the MALIBO (GB) variant consistently demonstrate better performance than the vanilla MALIBO except Forrester, where the non-meta-learning MALIBO can easily stuck in local optima. This indicates that, when there is little meta-data, gradient boosting can help the model to converge toward lower regret and the Bayesian logistic regression fail to optimize. For MALIBO (TS), the results show the model achieve lower immediate regrets than vanilla MALIBO across all the benchmark, because it encourages the exploration in the search space. However, due to the fact that Thompson sampling improve only exploration. for example Branin and Hartmann3 function, the MALIBO (GB) outperforms the MALIBO (GB) variant. Last but least, the results in experiments show that, by combing the exploitation of gradient boosting and exploration of Thompson sampling, MALIBO (GB-TS) achieves the best performance across all benchmarks.
MALIBO
MALIBO (TS)
MALIBO (CB)
MALIBO (GB-TS)
Immediate regret (log)
10º
rogret (log)
30*
10
6
10
12
(a) Quadratic
(b) Forrester
10
immediate regret (log)
Immedtate rogret (log)
201_
20
25
30
10
20
50
60
(c) Branin
(d) Hartmann3
Figure 13: Results of MALIBO and its variants on four synthetic function benchmarks without meta-train toeannraining. We report the standard error of immediate regrets over 100 runs.
21

---

Under review as a conference paper at ICLR 2023
E STEP-THROUGH VISUALIZATION
For illustration purposes, we provide step-through visualizations on a Forrester and a quadratic func- tion. For details of the synthetic functions, we refer to Appendix H.5 and Appendix H.4 respectively. We use the same meta-trained model for the visualizations as the one used in Appendix D.1 for the corresponding problem.
We demonstrate the advantage of using Thompson sampling in two parts. First, by showing the MALIBO (TS) variant optimize functions sequentially, which is in correspondence to normal BO pipeline, we can see how the algorithm explore the space efficiently. The illustrations are demonstrated in Fig. 14 and Fig. 15. Thereafter, we show two toy examples of synchronous parallel BO (Kandasamy et al., 2018 8) using MALIBO (TS) on the same functions. To be specific, we use three Thompson samples as acquisition functions in each iteration, and evaluates the three proposed points for the next optimization step. We demonstrate that, MALIBO (TS) can be easily extended to parallel BO with the help of Thompson sampling.
Lerge Leadan cormen spary
Exceet cany
1.0
Ar Accutattze Frecles
Accutaation furetize
Acguhellea bracttur
onediction Bhupooe xegiy
predictin
Usi puns HuRgtr
Ibvegove susgpy
(a) Iteration
(b) Iteration 2
(c) Iteration
bayent Foetas
dbcendeerya
ciszrvatica
-esemnssns you cermecappryy
dermttasyyo commit yarry
carervation you curnert qaary
66 Accuratues Iens
60
ranns prodictis
mIn prod
cares preckttic
02
(d) Iteration 4
(c) Iteration
(f) Iteration
Figure 14: Sequential Thompson sampling with MALIBO optimizing Forrester function. We
show the mean prediction together with the Thompson samples of the acquisition function in the lower part of each sub-figure. At the first iteration, MALIBO (TS) picks the point with highest mean prediction of the acquisition function, which is already close to the global optimum. Thereafter, chooses the point according to the maximum prediction of the Thompson sample, where explored another location of interest on the left-hand side. After the second step, the model believes that the promising region is the right-hand side and start exploiting that region, where the true optimum locates. Notice that, almost all the Thompson samples remain the shape close to the mean prediction of acquisition function, which means it will explore the search space according the current belief rather than simply random samples.
22

---

Under review as a conference paper at ICLR 2023
Elee' 4h4E7RT
densisayar
022
02
0.2
6.0
-1.00 -2.75 -3.50 -025
OZE
025
E53
0.5
148
-1.00 -0.75
-253
-0.25
0.00
0.25
038
375
1.0
-1.9
-0.5 -030
-233
0.03
135
0.5
07
10
You prodicton
YAM prodichi
00
-1.00
-2.1
10.0
0.70
116
00
-609
-203
-0.0
000
2.50
1/0
0.
40.5
4000
-236
100
(a) Iteration 1
(b) Iteration
(c) Iteration
0.5
000
0.0
0.
10
Accrsitoo frection
1.0
Accuiittonn Baxtoon
Aspultbor Trtetion
sieple
The sampln
00
-2.1 -3.0
035
+2.3
335
1.00
+000 +235 833
100
(d) Iteration 4
(c) Iteration
(f) Iteration 6
Figure 15: Sequential Thompson sampling with MALIBO optimizing a quadratic function. We show the mean prediction together with the Thompson samples of the acquisition function in the lower part of each sub-figure. Since the meta-data of quadratic functions do not have distinctive promising region but only sharing a similar global structure, the first point MALIBO (TS) picks is not informative for the problem. However, after exploring the search space with Thompson samples of the acquisition function, MALIBO (TS) gradually forms belief that the optimum locates around x 0.25. Notice that, even though the meta-learned acquisition function is not informative about the location of the optimum, but the shape of the mean prediction and the Thompson samples of the acquisition function remain similar to quadratic function over iterations, which conforms the global features learned from the meta-data shown in Fig. 12.
23

---

Under review as a conference paper at ICLR 2023
-2
Acrieittrr Furciion
Arppiction foretior
Aeputitter Brantven
warr proditine
rear pradition
(a) Batch iteration
(b) Batch iteration 2
(c) Batch iteration 3
tarpe Daa
Surge: Dante
Laget Kantas
yuumtrusyer you
Accydettinn fircttinn
Acpulittirn Arection
their puin crrpies iand preditie
Daripece sraples Emmal pedlicoo
Eata poodicuas
(d) Batch iteration
(e) Batch iteration
(f) Batch iteration
Figure 16: Synchronous parallel Thompson sampling using MALIBO (TS) to optimize Forrester function. Every iteration we draw three samples as acquisition functions and utilize the resulting query points observations for the next optimization step. In the first iteration, MALIBO (TS) already acquires three observations which covers both the likely positions for the optimum thanks to the parallelism. Subsequently, MALIBO (TS) exploits more often around the where the true optimum locates than exploring the other area of interest. At the last iteration, all of the three Thompson samples have already been skew toward the right-hand side, which shows MALIBO (TS) converges to the correct region.
24

---

Under review as a conference paper at ICLR 2023
275
2.9
KITS
2.5
your
charveten
200
2.0
3.03
2
123
1.00
130
180
-1.0 -0.5 -238
-233
0a0 0.5
116
-1.90
-0.00
-033
12x
140
-1.8
-279 40.00
2.01
100
08
-077
4.58
0.50
0.5
100
-0.5
-050
-025
000
3.7
1.00
-118
00.0
002
0.00
022
Batch iteration
(b) Batch iteration 2
(c) Batch iteration 3
335
335
3.55
2.00
g
0
133
1.5
145
125
075
-0.7 -128
-1.90
100
-1.3
1.0
Aequeells Bascke
Acquiatizn Turelee
Acpunia Buadie
03
1458
030
0.7
110
-0.5 1EAE
1006
025 375
1/00
11588 -57> +0.90 405 3.00
300
(d) Batch iteration 4
(e) Batch iteration 5
(I) Iteration 6
Figure 17: Synchronous parallel Thompson sampling using MALIBO (TS) to optimize a quadratic function. Every iteration we draw three samples acquisition functions and utilize the resulting query points as observations for the next optimization step. The exploration from the Thompson sampling helps to discover the boundary of the promising region, i.e. the observations y S r (blue crosses), in fewer iterations and the range of promising region keep reducing as more unfavorable points close the true optimum were acquired.
25

---

Under review as a conference paper at ICLR 2023
F ADDITIONAL BENCHMARKS
In this section we show more results for experiments with multiplicative noise as in Section 5. With the settings remaining the same, perform our experiments on Forester (Sobester et al., 2008) and Hartmann3 (Dixon, 1978) function ensembles. We refer to Appendices H.S and H.7 respectively for more details.
Forrester function ensemble For meta-training in Forester ensemble experiment, we randomly sampled noisy 32 observations in 64 prior-tasks. The results is shown in Fig. 18. MALIBO and its variants keep showing strong warm-starting performance and stay robust to noise compared to other methods. However, all the likelihood-fii free BO methods, namely LFBO, MALIBO and its variants, seem to stuck in local minimum in runs, resulting in almost improvement over the optimization process. The performance of all MALIBO variants is on-par with GC3P in all cases. Although most of the GP-based methods, namely GP, RGPE and ABLR, all outperform the other likelihood-free based methods, however after increasing the noise level, their performances degrade significantly.
201
Ineredair www.e
LTEO
MACEO
(a) Forrester 0.0)
(b) Forrester 0.1)
(c) Forrester 1.0)
Figure 18: Immediate regret for different BO algorithms Forrester function ensembles (D 1)
with different levels of multiplicative noise.
Hartmann3 function ensemble We randomly sampled noisy 512 observations in 256 prior-tasks in the Hartmann3 ensemble experiment. The results is shown in Fig. 19. All MALIBO variants shows the strongest meta-lcarning performance in all settings, and the gradient boosting variants report the lowest regrets together with ABLR in noise-free function. All MALIBO variants remain robust to noise, especially the vanilla MALIBO. The Ga-based method, although they have strong performance in the noise-free case, especially RGPE, but they degrade significantly after the noise level increased.
10'
Teuratate INBEEE
17R0
124
(a) Hartmann3 (c 0.0)
(b) Hartmann3 (e 0.1)
Hartmann3 1.0)
Figure 19: Immediate regret for different BO algorithms on Hartmann3 function ensembles (D = =3) with different levels of multiplicative noise.
26

---

Under review as a conference paper at ICLR 2023
G EXPERIMENTAL DETAILS
Consider a Residual Feed Forward Network (ResFFN) (He et al., 2016) architecture ResFFN-4-64, which contains 4 residual feedforward layers with 64 units. We ResFFN-4-64 to learn the latent feature representation, with 4 hidden layers, each with 64 units. For the mean layer m(-) and task embedding layer 2, we use a fully connected layer with 50 units for each. We use ELU (Clevert et al., 2016) as activation function for our problem following Tiao et al. (2021).
During meta-training, we optimize the weights with ADAM (Kingma & Ba, 2015) using batch size of B = 256, and polynomial decay for learning rate, with the initial learning rate Ifmitill = 10 end learning rate Irimal 2-1 and the exponent set 2. The model trained for 2048 epochs with early stopping. We set the regularization factor X = 0.1 in Eq. (4) and follow the approach in (Berkenkamp et al., 2021) to estimate the weights for modified Kolmogorov- -emiroo test and covariance regularization. In task adaptation, optimize the task embedding with L-BFGS (Byrd et al., 1995).
For the gradient boosting applied to BORE, LFBO and MALIBO,
the implementation in scikit-learn (Pedregosa et al., 2011) with default settings. The only difference is that we use the meta-leamnd MALIBO classifier the initial estimator for gradient boosting variants of MALIBO.
27

---

Under review as a conference paper at ICLR 2023
H DETAILS OF BENCHMARKS
H.1 HPOBENCH
The hyperparameters for HPOBench and their ranges are demonstrated in Table 1. All hyparameters are discrete and there are in total 66,208 possible combinations, More details can be found in Klein Hutter (2019).
Table 1: Configuration spaces for HPOBench
Hyperparameter
Range
Initial LR LR Schedule
{ 10 10 10-3.5.
10
10 3.5
3,5
10
211x 10 1011
}
{ cosine, fixed
Batch size
{23, 24, 23, 26
Layer 1 Width
24, 25, 26, 27,22. relu, tanh
2 2
Activation
Dropout rate
0.0, 0.3, 0.6
Layer 2 Width
21, 25, 26, 2 28, 29 relu, tanh } 0.0, 0.3, 0.6
Activation
Dropout rate
H.2 MI ALGORITHMS HPOBENCH
The hyperparameters for machine learning (ML) algortihms in HPOBench (Eggensperger et al., 2021)
and their ranges are summarized in Table 2. More details can be found in Eggensperger al. (2021).
Table Configuration spaces for ML algorithms in HPOBench
Benchmark
Hyperparameter
type
Log
Range
float
[2 10, 2 10, 2
SVM
gamma
float /
[2 [2-10,
alpha
float
/
[le 1.0] [1e-5, 1.0]
LogReg
etaO
float /
colsample_
float
X
1.01
XGBoost
eta
float
[2-10,100]
max max_depth reg lambda
[1,50]
float
[2 12-10,2 20,2-10]
max_depth max_fcaturee
int
[1,50]
RandomForest
float
X
[0.0, 1.0
min samples Ieaf int
X
[1,
min hinnssamlle.sppp
int
[2, 128
alpha
float
V
[1.0e-8,14
MLP
batch_size
int
V
[4, 256 ]
depth
int
X
[1,3]
ceariingrrae__nit
float
[1.0e-5 1.0]
width
[16, 1024]
H.3 NASBENCH201
The hyperparameters for NASBench201 and their ranges summarized in Table 3. All hyparameters are discrete and there are total 15,625 possible combinations. More details can be found in Dong & Yang (2020).
28

---

Under review as a conference paper at ICLR 2023
Table 3: Configuration spaces for NASBench201
Hyperparameter
Range
ARCO 0
none, skip-connect, conv-1 conv-3 avg-pool-3
ARC
none, skip-connect, conv-1 x 1, conv-3 x 3, avg-pool-3 x none, skip-connect, conv-1 conv-3
ARC 2
avg-pool-3
ARC 3
none, skip-connect, conv-1 1, conv-3 x 3, avg-pool-3
ARC4 4
none, skip-connect, conv-1 1, conv-3 avg-pool-3 x
ARC
none, skip-connect, conv-1 1, conv-3 3, avg-pool-3 3
H.4
THE QUADRATIC ENSEMBLE
The function for the quadratic ensemble is defined as:
f(z, b, = (a (x b)) cc
[0,
(26)
To form the ensemble, we choose the distribution for the parameters as:
a ~ -00.5 (1.),,,...) 1.5) b (.0., 0.9) c~ c~u((11,1
(27)
This distribution of parameters ensures that the search space contains the minimum of the quadratic function at x² = b with f(z") = c. The location of the optimum has broad distribution over the function space, which is intended highlight algorithms that learn the global structure of the ensemble rather than restricting some small regions of interest.
H.5 THE FORRESTER ENSEMBLE
The original Forrester function (Sobester et al., 2008) is defined following:
f(z, a, b, c) = a (6z - sinsin 4) + 6(x - 0.5) - E [0,
(28)
The function has one local and one global minimum, and zero- eero-gradient inflection point in the domain x E [0, 1]. To form the ensemble, we choose the distribution for the parameters as:
a a~ 2(0223)) b~ u(- 15) c~ (5,,5)
(29)
Let fa, b, c} and p(T) is three dimensional uniform distribution. The ranges are chosen around the usually used fixed values for the parameters, namely 0.5, b = 10, = -5.
H.6 THE BRANIN ENSEMBLE
The function for the Branin ensemble the following:
(xr a, b) a(x2 b CT1 r) s(1 cos +
x1 [-5, 10), 02 [0,15] (30)
The distribution for the parameters are chosen as:
a ~ U(0.5, 1.5) 6~ (0.10000.155))
e~ ((.00,))
(31)
m~.00 7,0) ((8.0,0,12.))
t~ t~000 0.05)
Let {a, b, r, t, and P(T) is six dimensional uniform distribution. The ranges chosen around the usually used fixed values for the parameters, namely a = 1, b 5.1/(4x?)) c = 5/π. n, 6, 10 and 1/(8x).
H.7 THE HARTMANN3 ENSEMBLE
The function for Hartmann3 (Dixon, 1978) ensemble reads:
4
3
f(r 01 az O3, 04)
Σ a; exp
Σ
An, p2
x E [0, 1]
j=l
3.0
10
307
T368
1170
26737
(32)
0.1
10
35
4699
4387
7470
P 44
3.0
10
30
1091
8732
5547
0.1
10
35
381
5743
8828
29

---

Under review as a conference paper at ICLR 2023
To form the ensemble, we choose the distribution for the parameters
2.0,0, 02 ~ ((000, 2.0) 03 ~ (2.00, 4.0) 04 ~ (.2.0,44.0)
(33)
Let = {ai, 02. 0z, 01) and p(7) is a four dimensional uniform distribution.
30


Reviews:
[{"review_id": "cmqqIeTeZAj", "reviewer": "Reviewer_YSQy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work proposes a likelihood-free Bayesian optimization strategy with a meta-learning scheme. Given multiple tasks, it trains task-agnostic and task-specific components in order to predict a probability that measures how likely a solution is, inspired by BORE and LFBO. By utilizing Bayesian logistic regression, it determines a query point. In addition, the authors use a gradient boosting model to predict a residual of the model. Eventually, the experiments demonstrate the effectiveness of the proposed method, compared to other existing methods.", "summary_of_the_review": "Please see the above text boxes.", "strength_and_weaknesses": "Here I describe the strengths and weaknesses of this paper.\n\n### Strengths\n\nIt solves an interesting topic where multiple historical tasks are given, by applying a likelihood-free framework.\n\nThe proposed method is quite novel. In particular, a combination of some components such as mean prediction layer, residual prediction layer, and gradient boosting is interesting.\n\nThe purpose of the respective components and the respective loss functions is well-described.\n\n### Weaknesses\n\nPresentation and writing can be improved. The current version is okay, but I think it can be polished more.\n\nIteration budgets for the experiments are too low. I think that you should give a larger iteration budget in order to show the convergence of the algorithm tested.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZtjIEt8YRYF", "reviewer": "Reviewer_Xog3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new approach to perform transfer-learning for Hyperparameter optimization where offline evaluations are used to fasten the tuning on a new task. The method proposes to leverage recent work that uses classification to train surrogates by learning to classify good/bad configuration. In particular in this work, the classifier takes a combination of global (task agnostic) and local (task specific) features and is learned with a Bayesian Logistic Regression layer which is made possible with the use of several approximations. \nIn addition, a gradient boosting approach is used to improve the final fit of the model at each iterations. Experiments are conducted on several tabular benchmarking suites  (hpobench, mlbench) and also artificial examples to study the effect of noise levels. The method proposed is show to be competitive or better than the baselines proposed.", "summary_of_the_review": "The paper provides an interesting and novel method to a relevant problem with several ingredients that may be leveraged by future work (for instance using GB to improve surrogate quality). However, as it stands the experimental section has only too little baselines to really its performance against state-of-the-art (only 2 baselines for transfer) and some part of the models are unclear. I believe those could be potentially addressed before camera ready.\n\nAdditional details:\n* many typos, it would be valuable to run a spell-check:\n  * meta-leared\n  * observaitons\n  * This benchmarks\n  * Coppola\n  * ensmeble\n* Eq (3): interestingly, the scale is back in the loss (but the surrogate dont have to predict values proportional to it)\n* A potential ablation would be to use BORE to illustrate the benefit of using LFBO in your case, other possible ablations I was wondering about were: using only local/global part of the model, fitting (3) directly as mentioned in the begining of 4.2. The latter would be important to have given that otherwise all the complexity of the approximations (laplace and logits) would not be justified.", "strength_and_weaknesses": "The strengths of the paper are:\n* tackle a very relevant and impactful problem\n* provides a novel technical contributions with several ideas that could be leveraged in future work (refinement with gradient-boosting, use of EI equivalent likelihood-free criterion)\n* good coverage of experiments: a reasonable range of benchmarks/different blackboxes are considered \n\nWeaknesses:\n* lack of runtime analysis: no runtime is given for how long the method takes to return suggestion (the appendix mentions 2048 epochs on top of that, a gradient boosting tree is fitted). However, this can significantly worsen the results if they were reported against wallclock time (as proposed in other works) since then almost all time will be spent in fitting models and the method would likely underperform random-search. Without additional details, one cannot assess if the method will have any practical relevance (a method taking 5 minute to suggest the next candidate will likely not be very applicable).\n* the set of baselines is relatively considered is relatively small (only 2 transfer learning and 2 non transfer baselines). They are several baselines that could be easily added: BORE for non transfer, but also BORE/LFBO with search space pruned to bounding box of best previous evaluations [Peronne 2019]. Ideally, another additional transfer-learning baseline on top of this simple bounding-box approach would also be added in order to better assess the quality of the method regarding state-of-the-art (2 methods for transfer is a small number).\n* details are lacking, some part of the methods were not clear to me:\n  * 4.1: was difficult to get through while it is a key part of the paper, I would suggest adding the dimensions for the different variables and give the final expression of the classifier that sums its two input (which is given in 4.2). In any case, the text alone was not sufficient for me to get exactly how $\\phi \\dot z_t$ is obtained as I could see several ways to achieve this depending on input dimensions.\n  * 4.2: the expression of L^LFBO given in Eq. (3) does not take arguments as input\n  * 4.3: how exactly is the Gradient boosting combined with the classifier was not entirely clear to me. I would recommend to write it down formally rather than with words (given the current text description, there could be many ways on how a GB would be fitted to reduce the residuals)\n* the set of ablations is small and several complexity are not justified (for instance the approximation done instead of the direct optimization of (3), see additional details for a bigger description on this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CwzkuTpvZX1", "reviewer": "Reviewer_Xo11", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of meta Bayesian optimization (BO), which aims to warm-start the BO process by exploiting knowledge from related tasks. In this paper, the authors propose warm-starting the acquisition function, which takes the form of a classifier in the likelihood-free BO setting. Gradient boosting is further incorporated to combat distributional shifts. ", "summary_of_the_review": "The paper presents an interesting idea to tackle an important problem. However, the difficulty or challenges of the problem are not highlighted enough. The comparison with related works is lacking. And the writing can be improved. \n\nDetailed comments:\n\n- The idea of meta-learning an acquisition function has been explored by Volpp et al. (2020). Could the authors elaborate on the differences and advantages of the proposed algorithm to that of Volpp et al. (2020)? In addition, this should also be included as a baseline in the experiment section.\n\n- The paper is generally hard to follow since the readers assume knowledge from several other key papers. I have to read several other works on the BORE framework and BaNNER to understand several parts of the paper. I encourage the authors to make the paper more self-contained by reintroducing concepts from essential related works like Berkenkamp et al. (2021), Tiao et al. (2021), and Song et al. (2022).\n\n- When the input space is a simple Euclidean space without any structure (think of minimizing the function $f(x) = x^2 – 4x$), how does the feature embedding $h(\\cdot)$ works? I suppose feature embedding is only useful when there is some structure in the inputs (e.g., images).\n\n- Since BO is a black-box optimization algorithm, can the authors indicate more clearly what are the objective functions that we are optimizing in the experiment section? It is difficult for readers to comprehend and assess results without knowing what are we optimizing.\n\n- From my understanding, UCB is one of the most commonly used acquisition functions, with a nice theory in BO. However, the paper did not mention or compare with UCB. Is there a good reason for that? Can the authors compare the proposed algorithm with GP-based BO methods with EI/PI and UCB?\n", "strength_and_weaknesses": "Strengths:\n\n(+) The problem of (meta) Bayesian optimization is an important problem with several applications.\n\nWeaknesses:\n\n(-) The comparison with existing works is lacking, both in the techniques and in the empirical evaluations.\n\n(-) The lack of a theoretical guarantee (which is common in the BO literature).\n\n(-) The writing can be improved (as detailed below).  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CgKqhIRAf2o", "reviewer": "Reviewer_YYiE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a meta-learning method for likelihood-free Bayesian optimization. In particular, the proposed approach is a combination of two existing solutions, a meta-learning approach from BaNNER and a likelihood-free LFBO, together with the gradient boosting method. The proposed solution is able to work with high-dimensional inputs and handle heterogeneous scales and noises across different tasks.\n", "summary_of_the_review": "Due to the limited novelty, concerns about the proposed solution, and the lack of baselines in experiments, the paper may require further improvements to fit the conference.", "strength_and_weaknesses": "The main strength of the paper is in the empirical performance which is shown to outperform two of the existing works, ABLR, and GC3P. However, it has several weaknesses as follows.\n\n1. The technical solution is not novel since it is based mostly on the two existing works BaNNER and LFBO.\n\n2. Although the paper claims that the meta-learned classifier can balance between exploration and exploitation, the proposed approach requires a gradient boost method to correct the errors. It means that the meta-learned classifier does not allow exploration properly. This is unlike multi-task GP, where the correlation between tasks can be learned from the data without any additional ad-hoc method (such as gradient boosting) for correction.\n\n3. While Thompson sampling often relies on a good approximation of the posterior distribution, the proposed approach only uses the Laplace method to approximate the posterior distribution which is a very simple approximation method. Better approximation methods such as variants of MCMC and/or variational inference techniques should be applied.\n\n4. The numerical representation of the task (z) is simply optimized to be close to a standard (multivariate) normal distribution. I am wondering if the proposed method can work with a multi-modal task distribution, i.e., prior tasks form 2 clusters where tasks in a cluster are similar and tasks between clusters are different. In a principled Bayesian approach such as multi-task GP, it can correctly correlate a new task with an existing task.\n\n5. It is unclear to me about the choice of \\tau (for prior tasks and the current task) in the proposed algorithm.\n\n6. While the paper reviews a lot of related works on meta-learning for BO, the experiments only consist of 2 existing works while ignoring the others such as those using GP surrogates (e.g., running experiments on low-dimensional problems).\n\n7. Section 3.1 lacks many well-known BO methods such as GP-UCB, predictive entropy search, max-value entropy search, and knowledge gradient-based methods.\n\n8. Since BO is about sample efficiency, can gradient boosting work reasonably well with a small training dataset?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject