Transfer Learning for Bayesian Optimization on Heterogeneous Search Spaces
Authors: Zhou Fan
Xinran Han
Zi Wang
Venue: Accepted by TMLR ()

Abstract: Bayesian optimization (BO) is a popular black-box function optimization method, which makes sequential decisions based on a Bayesian model, typically a Gaussian process (GP), of the function. To ensure the quality of the model, transfer learning approaches have been developed to automatically design GP priors by learning from observations on "training" functions. These training functions are typically required to have the same domain as the "test" function (black-box function to be optimized). In this paper, we introduce MPHD, a model pre-training method on heterogeneous domains, which uses a neural net mapping from domain-specific contexts to specifications of hierarchical GPs. MPHD can be seamlessly integrated with BO to transfer knowledge across heterogeneous search spaces. Our theoretical and empirical results demonstrate the validity of MPHD and its superior performance on challenging black-box function optimization tasks.

Published in Transactions on Machine Learning Research (02/2024)
Transfer Learning for Bayesian Optimization on Heteroge- neous Search Spaces
Zhou Fan Horvard University
afan@@g.hrroan hww..adda.rd..ooooo edu
Xinran Han
zhoranhan@gghavvrr..com
Horvard Universiby
Zi Wang Google Mend
wanggif@google..com
Reviewed on OpenReview: https: // openreview. net/ forum? id= emXh4477yl
Abstract
Bayesian optimization (BO) is a popular black-box function optimization method, which makes sequential decisions based on a Bayesian model, typically a Gaussian process (GP), of the function. To ensure the quality of the model, transfer learning approaches have been developed to automatically design GP priors by learning from observations on "training" functions. These training functions are typically required to have the same domain the "test" function (black-box function to be optimized). In this paper, we introduce MPHD, model pre-training method on heterogeneous domains, which uses neural net mapping from domain-specific contexts to specifications of hicrarchical GPs. MPHD can be scamlessly integrated with BO to transfer knowledge across heterogeneous search spaces. Our theoretical and empirical results demonstrate the validity of MPHD and its superior performance on challenging black-box function optimization tasks.
1 Introduction
Many real-world applications require finding the best hyperparameter values by evaluating a series of configurations of those hyperparameters. Some examples include tuning machine learning (ML) models (Snock et al. 2012 Turner al. 2021), learning robot control strategies (Wang et al. 2021), synthesizing functional chemicals (Shields al. 2021). , and discovering new materials (Frazier Wang 2015). For these problems, there exists an underlying black-box function that scores the utility of hyperparameters. One popular way of formulating such problems is Bayesian optimization (BO): optimizing an unknown function by reasoning about the Bayesian beliefs about this function.
In BO, we often use Gaussian processes (GPs) as Bayesian beliefs for unknown functions. Given a GP prior and observations on the function, we can obtain a GP posterior and use its uncertainty predictions to make decisions on which datapoints to acquire. For example, one popular strategy is greedily evaluate the inputs that achieve the highest upper confidence bounds on function values (Srinivas et al. 2010).
A prerequisite of BO is to specify GP prior, which can be difficult to do in practice. To address this issue, much progress has been made to learn the GP prior using transfer learning based approaches (Swersky
ct al. 2013 Yogatama & Mann 2014 Wang et al. 2018 Perrone et al. 2018 Volpp et al.) 2019 Wistubal These approaches typically assume that we have data on a sel of "training" functions, and the goal is to generalize a learned GP model or a learned strategy to a "test" function, which has the same domain as the training functions. While these mothods have been shown to perform well on a variety of tasks, they cannot be easily used to generalize test functions that do share the same domain the training functions.
& Grabooka 2021 Wang el al. 2022

---

Published in Transactions on Machine Learning Research (02/2024)
Pre-training in domains A, B,
Using the learned model in domain T
domain
domain
0 estimated lengih-sccale
1 dimeusion of domaiin A
S context of k1
scale Bys of dimension of domain
Sri of lengthl
input
leaammed model
output
dri estimated prior distriburion pammeter Ori
Baz: estimated longth-scalc of
dimeusion of domain
Suzi context A2
Syz: context of length- scale f22 2nd dimension of domain
input
keamed model
output
Gzgz estimated prior distribution parumeter of'
0 estimatedd noise variance of domain A Sxy context of AA
Step ② of pre-tmining
g deleumed model
Syy: context of lengthl- scale Bys of 3 dimension of domain
input
llammed model
output
dry: estimated prior distriburion pammetter drg
domain
Pag: estimated length-sccale
Sya context of noise variance Bya domain
input
dlaamedd model
output
āzg: estimated prior distribution parameter of Byy
1 dimension of domain pp
Figure 1: Illustration of how the learned MPHD model is used for BO. Only the second step of the pre-training of MPHD is shown. In this example, domain A has dimensions so there are 2 length-scale parameters for it, while domain T has dimensions and there are corresponding length-scale parameters.
In practice, the data available for transfer learning might not have the ideal setups to ensure the domains of all functions are well-aligned. For example, we might have collected many datapoints by experimenting with several robot skills that have different (but potentially overlapping) sets of control parameters (Wang et al., 2021). Or, we might have ML model tuning data from a commercial BO tool (Golovin et al. 2017 Balandat et al. 2020), where different people might tune different hyperparameters and use different. names for the same type of hyperparameters. With current methods, can be difficult to transfer knowledge from one task to another in these cases.
In this paper, introduce Model Pre-training Heterogeneous Domains (MPHD), new transfer learning method for BO on heterogeneous search spaces. In the pre-training stage, MPHD Iearns model with a mapping from domain-specific contexls to specifications ions of hierarchical GPs. This allows transferring knowledge from a set of training functions with different domains to a new test function to be optimized on an unseen scarch space. Using the pre-trained model, MPHD can generate a customizod hicrarchical GP the prior for the test function, and then this hierarchical GP can be combined with an existing acquisition strategy to perform BO. An illustration be found in Fig. 1
Through theoretical and empirical case studies ($3 show that MPHD is asymptotically consistent, meaning that it converges to the ground truth solution as the number of training functions increases. We also show that the hierarchical GPs generated by MPHD can accurately capture test functions with new domains.
verify the usefulness of MPHD for BO ($4), we conducted extensive experiments on real world BO transfer learning problems with heterogeneous search spaces. We tested benchmarks including HPO-B (Pineda-Arange et al. 2021 and PD1 (Wang et al. 2022), which involve 17 search spaces in total. Our results have shown signilicant improvement made by MPHD sample efficiency for BO functions with unseen search spaces.
In this paper, we make three contributions: (1) We identify practical problem in BO, and propose a new problem formulation: the transfer learning problem for functions with different domains. (2) We propose a new method, MPHD, to solve this problem.
(3) We show the effectiveness MPHD both theoretically and experimentally, and prove the consistency of MPHD building on our new theoretical results for constructing sufficient statistics of training functions. To the best of our knowledge, MPHD is the first Ga-based framework that can be used to transfer knowledge for BO hetcrogencous scarch spaces.
2

---

Published in Transactions on Machine Learning Research (02/2024)
Related work. Researchers have developed methods for transferring knowledge between BO tasks. For
example, Swersky et al.] (2013) and Yogatama & Mann (2014) proposed to learn multi-task GP and use the similarity between tasks for generalization. |Feurer et al. (2018) aimed to learn warm-starting strategies from previons BO tasks using an ensemble. Recently, [Wang et al. (2018); Perrone et al. (2018): Wistuba &
Grabocka (001): Wang el. al. (2022) found that learning a GP prior from evaluations on training functions can be an effective approach to transfer knowledge for BO if all the functions share the same domain.
Another line of related work is end-to-end black-box function optimization. Chen et al. (2017) trained recurrent neural network (RNN) on a large mumber of BО trajectories. The RNN can then be used to generate the next point to evaluate for a new BO problem. Chen et al. [2022) introduced OptFormer, a transformer-based transfer learning method for hyperparameter tuning on universal search spaces. Opt trains a giant transformer model on millions of BQ trajectories to learn to propose lyperparameeerss in an end-to-end fashion. Note that Feurer et al. (2018) and Chen et al. (2017 2022) all require previous optimization runs, meaning that they cannot make use of raw evaluations on training functions without simulating BO trajectories. Our approach, MPHDD, focuses transferring knowledge about functions by pre-training surrogate model on raw evaluations, and does not require data in the form of BO trajectories.
MPHD can be naturally combined with other components of BO methods, e.g. acquisition functions, input and output warping, cross validation etc, to complete a practical BO software. For example, MPHD can be directly incorporated in BoTorch (Balandat et al. 2020 and Vizier (Golovin et al. 2017) by replacing their default hierarchical GP model.
2 MPHD: Model Pre-training in Heterogeneous Domains
We present MPHD, model pre-training framework for functions with heterogeneous domains. Given data collected on training functions, MPHD aims to learn distribution over functions to model unseen test function. The domains of all training functions and the lest function can have different numbers of dimensions and each input dimension can have different meanings.
2.1 Problem formulation
We first define terms on datasets. A super-dataset is a collection of data points from all training functions with different domains, along with the contexts associated with each domain. A datasel is a collection of data points from training functions with the same domain. A sub-dataset is a collection of data points from the same training function. Fig. illustrates super-datasel example.
Formally, we use D = [(D,,, S,),& to denote super-dataset, where D, is dataset and S, is domain- context. Each dataset D, consists of observations on collection of training functions F; = =fi; x - RR
where functions in F; share the same compact domain x00 C R2. The information about domain x²) is encoded into the context St. Let dataset D. = {Di where cach sub-datasct Dij = {= Viy
(1) (1) Lug
Ly is the number of observations on function fig perturbed by i.i.d. additive Gaussian noise, i.e., Dif ((ii(()!)) (1))), o²). ) Noise variance of is specific to each domain x(0)
(1)
We assume that all functions in F; with domain x0 are i. function samples from the same Gaussian process, GP, = GP(,, k;; 0,), where 14 is a constant mean function, ky is a stationary kernel, and 0, 1 R are GP p including parameters of the mean and kernel functions well as the noise variance of. Each GP parameter Oe is sampled independently from its prior distribution O(au)), where asi = 0(sa) E gd. Si is the context of GP parameter aak. and o maps from contexts to hyperparameters (parameters of the priors). The domain-specific context. S; is composed of the contexts of all GP parameters, i.e., S, = |
Example of the domain-specific context. For d-dimensional domain using the Matern kernel, there are d length-scale parameters. In the upcoming experiments in 4 each length-scale parameter is associated
In the classic GP literature (Rasmussen Williams 2006), 6; are called hyperparameters of GP. But from functional perspective, GP is parameterized distribution over random functions, and so 0 can also been seen as parameters of GP.
3

---

Published in Transactions on Machine Learning Research (02/2024)
M;
(I)
Example of super-dataset
comtinoous dimension
8ih
H
Fij
dimenzion
Domain X
Contert St
Functions F
Observations
of
x(
(
(1.0..13
f1t
fune,
Vij
20
Osh
fij
Lij
X
fix
fime
(1.0 2,33
k;s
X
N
fN1
fuMa
00.1
Figure 2: Graphical model of MPHD. Function & maps context 8th hyperparameter agh, which parameterizes the prior of GP parameter Os.-
Figure 3: An example of a super-dataset structure. Each dimension of domain x(()) corresponds either discrete continuous real-valued variable.
with a 4-dimensional context vector. This context vector comprises one-hot encoding for the first two dimensions, identifying the domain dimension that the length-scale parameter corresponds to as either discrete continuous. The remaining two dimensions of the context vector denote the counts of discrete and continuous dimensions within the domain respectively. For example, the domain marked as 2 in Fig. 3 includes discrete dimensions (first and third) and 1 continuous dimension (second). Consequently, the context vector for the length-scale parameter of its first domain dimension is expressed as (1, 0, 2, 1), indicating a discrete dimension in domain having two discrete dimensions and one continuous dimension. The context vector for the second domain dimension represented as (0, 2, 1). One advantage of this basic context representation is that it can be automatically generated for almost any domains, requiring only the essential meta description of domain for BO.
Experiment results in $ will showcase that this basic domain-specific context, although simple, is surprisingly effective in conveying information about prior distributions our model. Intuitively, the prior over a contimuous learning rate hyperparameter should be very different from the prior over a discrete hyperparameter which refers to an activation function type. Moreover, tuning problems on similar models tend to have similar number of hyperparameters to tune. For example, in НPО-B (Pineda-
et al. 2021). the "svm" search spaces have 6 to 7 dimensions, while the "rpart" search spaces have 29 to 31 dimensions. There could be other ways to include more sophisticated information about the black-box function, and MPHD is general method that is capable of incorporating any kinds of contexts in vector format.
Fig. 2 illustrates the graphical model. The goal of MPHD is to pre-train this probabilistic model so that for any new domain, the model can generate the prior distributions over the GP parameters to construct a domain-specific hicrarchical GP.
2.2 Our method
As shown in Fig. 2 the model can be compactly described by function d. Thus, model pre-training in MPHD is equivalent to training function á the super-dataset, D (D) such that the model generalize to test functions with new contexts. We define the pre-training objective to be the log marginal data likelihood as follows,
((0) Σ log logp(D; (.) Σ log / p(D, 1.)p((, | S.)
N
(2T))
log
p(0, o. S.)p(D, 1
(1)
M det
Mr, log logp(D; (DD))1)))lo-
log (D 0.) logp(0, 0 Si)
(0, S.)
(2)
4

---

Published in Transactions on Machine Learning Research (02/2024)
where 0, = arg m p(D: (1)%)) and 8, [0 The approximation in Eq. uses Laplace's method (see
derivations in S Eq. 2 removes terms irrelevant of optimizing &. To summarize, model pre-training in MPHD has two steps:
N log p(6, St).
I OVEE + arg.axx p(D; 0:), ② arg max Σ
Step 1 can be done using gradient-based optimization methods for each of the N likelihood functions,
M.
po, 0,)
Σ logg(Dss Dig 0,)) - -- Σ 3=1
(01))) (k! Vij
(0.)
k-(0.) + +2 log is
+
Lij log 2n
(3)
2
where vector y
/0 ( 2. and matrix k( = [k( 2.f 0,) 0.)
Step 2 requires computing the approximated objective
N H.
[ ~ L(0) = Σ log p(d (i) = log (Oin 0ss = ((8h)), ==1
(4)
==
which depends on the exact forms of the prior distributions O(aa). Morcover, we need to specify the function space for d such that optimization is possible. In this work, we use a neural network to parameterize function o, and Step 2 can be done by optimizing over the weights in o with gradient- based methods.
3 Case studies for validating MPHD
We validate MPHD via case studies. $3.1
presents theoretical analyses on convergence and consistency. $3.2 shows empirical results on synthetic data to compare pre-trained models with the ground truth.
3.1
Theoretical analyses
For the theoretical analysis in this section, assume zero mean and anisotropic Matérn kernels with known smoothness term v (e.g. == 5/2), and the GP parameters 0; = 10 to be learned only include length-scales. For simplicity, we also assume that contexts sih are the same across domains (²) i E [N] and the length-scale priors are normal or Gamma distributions. Under mild conditions, we show that (1) For each i E [N], as Mj (the number of sub-datasets) increases, the estimated GP parameters 6, for the respective domain 20 converge to the ground truth parameters 07 and consequently (2) For each h E [Ht,), as N (the number of datasets) increases, the learned hyperparameters åeg of the normal or Gamma priors converge to the ground truth values agg.
Background. The theoretical soundness of pre-training heterogeneous domains relies the quality of the estimated GP parameters from each domain. The asymptotic behavior of MLE for covariance parameters under a single GP has commonly been studied in two asymptotic frameworks with fixed or increasing domains [Bevilacqua et al. 2019 Zhang & Zimmerman, 2005). Under the fixed domain setting, observations are sampled a bounded set and thus become increasingly dense with more samples. the increasing domain setting, observations are collected with a minimal spacing in between data points and thus make the sampling domain unbounded as the number of observations increases; i.e. for sub-dataset {(x(1), ( with T observations, there exists fixed A > 0 such that
(x((
x (*) 2 A, VI ST.
(5)
Mardia & Marshall (1984) and Stein 19999 showed that, given the observations from a single function sampled from a zero-mean GP, MLE estimates of covariance parameters are consistent and asymptotically normal with mild regularity conditions for increasing domains. [Bachoc (2014) showed that while the MLE given finite observations may not be unique, it converges to unique global minimizer with probability converging to one as T goes infinity. Additionally, Bachoc al. (2020) discusses extensions of these results to GP models with non-zero mean functions.
5

---

Published in Transactions on Machine Learning Research (02/2024)
0.0
F2
e
Input X
Figure 4: Illustration of how to construct a pseudo sub-dataset in Lemma. 1 from observations on functions [1, ... We rearrange the original sub-datasets in the domain such that datapoints from different, sub- datasets have a distance at least Q > in the constructed pseudo sub-dataset.
...
Our results. The critical part of our proof is to show that the setup MPHD belongs to the increasing domain sctting. The key property of the incrcasing domain setting that there vanishing dependence between observations that are far apart (Bachoc 2014). Thus, a larger sample size would be more informative of the covariance structure.
Lemma 1. For any E [N], (P 0,) and its corresponding dataset D; = {Dig) there exists a pseudo sub-dataset D observed at inputs 2(1), x!2), .(..,,(()) function f GP such that D satisfies the increasing domain assumption in Eq. 5 and D is sufficient statistic of D; with p (D; 0,) = p(D) 0:).
Proof. Consider functions fij whose observations Dij [(x E M,].
(1) (
Vej BE is in dataset D; DDis |
1,44 where
Since the domain 20 of Jig is compact, and we define Q' = su || - xll + 1 < oo. Since the stationary covariance function only concerns the relative location between two points, we can shift all inputs jointly while the NLL, NLL((DdyH)
log p({ 0,), stays the same. More formally, let big)) B for some Q > 0, where d; is the input dimension and 1a, is a
(1) + 1a, (Q + Q')j, Vij dimimensoon vector filled with ones. Because the distance between any pairs of inputs stays the same, we have (I.I(((:;))) = (LT ((D,)) = log (Dig 0o)-
Dj ((=,)
We define an augmented sub-dataset D Đ= D..
As Q + oc, we can show that the sum of NLLs over set of sub-datasets and the NLL of the single augmented sub-dataset are equivalent. Without loss of generality, consider the Matérn covariance function with smoothness term == 1/2 and variance o², length-scale p. (1/2(d) = x² exp(-2). Since the covariance exponentially decays with the distance d between points, for any O, we can find a Q such that C(d) for d Q.
For the augmented sub-dataset カ= = D, ......D we have ||=' x1 Q for any Dj and x' E Dgy j # j'- As + 0, the covariance matrix for this augmented sub-dataset becomes block diagonal. As a result, we can express its NLL as a sum of NLL for Dat...., DiM, respectively, i.e., NLL((D)) = E log 0t). This gives us the equivalence of density in Lemma I and by definition, D is a sufficient statistic of the original dataset D,
Note that the distance between inputs in D will always be at least min, min || - x 4') > 6, since any two datapoints in each sub-dataset Dij are at least 8 apart by construction problem setup. As M, + 00, the augmented sub-dataset is unbounded and satisfies the increasing domain characterization in Eq.
Lemma I highlights the important connection between the increasing domain setting and our setups with an increasing number of sub-datasets. Intuitively, this lemma shows that observations from multiple independently generated functions from a fixed GP can be viewed as being sampled from one function, with infinitely large interval between subasstasets' observations. We illustrate this process on a 1-dimensional domain in Fig. 4 We prove Lemma in

---

Published in Transactions on Machine Learning Research (02/2024)
To show the asymptotic properties of MPHD, we make the following assumptions:
(1) Dataset D, = { (i E [N]) contains a finite number of observations in each of its sub- datasets. (2)) There exists a minimum spacing å > 0 such that | 2ij 2 o, (1 for all sub-datasets Diy
(2) The ground truth GP parameters 0 belong to C₁, the space of 0,, used for estimating 0 in Step !
(3) For each E [N], there is sufficient information in sampling locations xx1))x1 ,..,,,""""" of the pscudo sub-dataset D (Lemma 1 to distinguish covariance functions (z)) with k() (Bachoc 2014
That is, for any 0, E CC,,
Σ (1(==(0,x());;)))) x00),, :8)) O.
lim inf inf T-000 110 1012e T
(Asymptotic Identifiability)
t,0==1
Intuitively, the asymptotic identifiability condition mcans that no two distinct covariance paramcters exist such that the two covarinnce functions are the same on the set of randomly sampled points. In our case, as the number of observations T - in unbounded region (increasing domain), it is realistic to expect that this condition holds on the sec of observations.
Theorem Given assumptions (1)-(3), for dataset D2 with M₁ sub-datasets generated from the same mean and coveriance function, as M; + 0o, we have 8, B 07.
Theorem 3. Given assumplions (1)-(3), the number of dalasels, and sub-dalasels N, M, co, Vi, MLE for the prior distribution of each GP parameter ((aa), E [H,] is consistent; i.c., 6ih Dx a:p-
The proofs of Theorem and can be found in $ The above theorems complete the claims made at the beginning of $3.1 Moreover, MPHD has the following asymptotic MLE behaviors in Step ② with n estimated samples of GP parameters 0,
Remark. (Ye & Chen (017)) and Rice (0066)
(1) When the prior is assumed to be from a Gamma distribution parameterized by shape a" and rate by, the MLE bà is consistent and asymptotically normal with distribution N(aa", b²), I[(a" bb)=", where I is the Fisher information matria.
(2) Similar results hold when the prior is normal distribution parametrized by mean e* and standard deviation d' where the MLE satisfies (e, (c", dd))
A key observation from our theoretical analyses that with sufficient observations each dataset Di, increasing the number of sub-datasets can effectively improve the estimation of the covariance parameters and thus the parameters of the prior distribution. We show the empirical asymptotic behavior in $3.2
3.2 Empirical analysis with synthetic data
In this section, we present empirical results to further demonstrate the asymptotic properties of MPHD. For these empirical results, we assume constant mean and anisotropic Matérn kernels with known smoothness term v. The asymptotic propertics of learning the length-scale parameter are included this section, while the results for other GP parameters can be found in { We generated Iwo synthetic super-datasets following the generative process illustrated in Fig. 2 with two different settings where we use Gamma distributions the prior for length-scales.
Synthetic Super-dataset. (S) is a smaller and simpler one out of the two synthetic super-datasels, which uses the same Gamma prior for all domains (i.e., function @ returns constant hyperparameters). includes 20 datascts (cach with its own domain) with 10 sub-datasots each dataset. Each sub-dataset includes noisy observations 300 random input locations in its respective domain. The dimension of each domain is randomly sampled between 2 and 5.
7

---

Published in Transactions on Machine Learning Research (02/2024)
Estimated a of Lengtb-scale Prior
20.0
(a) Distribution parameter
(b) PDF of distribution
(c) NLL of test datasets
Estimate
Ground-truth
17.5
Ground-truth
Probability Density
5-
of Training Datasets
of TTss Dotasets
45-
of Traiiing Datasets = 16
15.0
40-
12.5
3-
35-
10.0
30-
7.5
25-
5.0
10
15
0.0
0.5 Length-scale
1.0
1.5
10
15
Number of Training Datasets
Number of Training Datasets
Figure 5: For Synthotic Super-datasct (S) with fixed one-dimensional length-scale prior, we plot (a) estimated shape parameter à of Gamina distribution prior for the length-scale GP parameter, (b) the PDF of the shared Gamma prior for length-scales ($3.2.1), and (c) NLLs of the test datasets on pre-trained priors w.r.t. the number of training datasets. We show the mean and violin plots over random seeds. The pre-trained Gamma distributions with 16 training datasets are more stable than those with 2 training datasets and match well with the ground-truth prior.
Synthetic Super-datasct (L) is larger and more complex super-datasct and is also the synthetic data used for BO evaluations in $ It has domain-specific Gamma priors where the hyperparameters linearly depend on the number dimensions of each domain (i.e., function & is a linear function of the domain dimension d.). We let the hyperparameters depend on the number of input dimensions order to simulate practical applications where tuning objectives with different numbers of input dimensions might need to be modeled with very different hyperparameters in MPHD. Additionally, this synthetic linear dependency allows for the comparison of learned hyperparameters against the actual ground truth. This super-dataset includes 20 datasets (each with its own domain) with 20 sub-datasets in each dataset. Each sub-dataset includes noisy observations at 3000 random input. locations in its respective domain. The dimension of each domain is randomly sampled between 2 and 14.
We split each of the synthetic super- datasesssssts into training data and test data: for Synthetic Super-dataser (S), we used 80% of datasets as training datasets and the remaining 20% as test datasets; for Synthetic Super-datasct (L), we used 80% sub-datasets within each dataset as the training sub-datasets and the remaining 20% the test sub-datasets. All experiments repeated 5 times with different random seeds. More details on the setups of the synthetic super-datased can be found in E.
3.2.1 Length-scales with the same Gamma prior
For Synthetic Super-dataset (S); Fig. 5(a) shows â of the pre-trained Gamma prior I(ā, 6) for length-scales, where à is the shape parameter and b the rate parameter. As the number of training datasets increases, the variance of the estimated à gradually decreases and the mean becomes closer the ground-truth prior. Fig. b(bb plots the PDF of both the pre-trained and ground-truth Gamma priors, showing that more training datasets help the stability of pre-training. These results are consistent with our theoretical analyses in $3.1
In Fig. ((c),, we show the NLL of test datasets (the negative of the objective L()) in 2.2 but applied on test data instead of training data) with an increasing number of training datasets. Both the mean and variance of the NLL. drop as the number of training datasets increases, indicating no overfitting.
3.2.2 Length-scales with domain-specifé Gamma priors
For Synthetic Super-dataset (L) with domain-specifie Gamma length-scales priors, we ran MPHD with two versions of the function Φh
Variants of MPHD: (1) MPHD Standard, which uses a neural net (NN) to represent the length-scale prior for any domain dimension taking as input the context sa that corresponds to the length-scale of that domain dimension and generating the length-scale Gamma prior for that domain dimension output. On
8

---

Published in Transactions on Machine Learning Research (02/2024)
Synthetic Super-dataset (L), MPHD Standard uses domain-specific context that specifies the number of dimensions as all domain dimensions are continuous. For each of the other GP parameter types such signal variance, MPHD Standard directly learns a shared prior distribution without NN for all domains. The NN-based length-scale prior of MPHD Standard consists of two hidden layers, each of size 16, and utilizes the hyperbolic langent as the activation function. When this NN takes as imput the context. vector related to a length-scale parameter, it outputs 2-dimensional vector. This output vector symbolizes the two hyperparameters (shape and rate) of the Gamma distribution that form the length-scale prior distribution. (2) MPHD Non-NN HGP: the model is simplified version that learns a shared length-scale Gamma prior for all search space dimensions without using an NN-based length-scale prior and is pre-trained with the two-step approach. Essentially, function & outputs a constant for every hyperparameter type, same as $3.2.1 Priors for the other GP parameter lypes such as signal variance are the same as MPHD Standard.
For both versions of MPHDD the underlying GP uses constant mean and anisotropic Matérn kernels, and function & outputs hyperparamoters in Gamma priors.
For the test datasets in Synthetic Super-dataset (L) with dopain-specific length-scale priors, Fig. presents the average KL divergence between the ground-truth Gamma priors for length-scales and the corresponding Gamma distributions generated by the pre-trained func- tion o, with an increasing number of training datasots. The results are presented for two versions of MPHD: the first empolying an NN-based prior and the second utilizing a non-NN prior. The KL divergence with re- spect ground-truth Gamma prior is averagod over repeats of experiments (with different random seeds) and all possible numbers of domain dimensions between 2 and 14. The KL divergence values of both versions of MPHD decrease as the number of training datasets increases, which shows that the pre-trained model grad- ually becomes closer to the ground truth. Moreover, MPHD with NN é achieves lower KL divergence values than MPHD with constant Φ, showing the advantage of using an expressive function in MPHD for complex problems.
12.5-
Average KL Divergence
Non-NN Prior
10.0-
NN-based Prior
7.5
5.0
2.5-
0.0
3
4
Number of Training Datasets
Figure 6: The average KL divergence between the ground-truth and the learned Gamma distribu- tions.
4 MPHD for Bayesian optimization
The goal of BO is to optimize black-box test function f with few evaluations as possible. BO works by optimizing series of acquisition functions to sequentially select inputs and observe their function values. In every iteration of BO, surrogate model is constructed based on all the observations on function f. and the acquisition function is defined over the predictions from the surrogate model.
MPHD can generate domain-specifie hierarchical GPs as surrogate models for BO on new search spaces. More precisely, to optimize function f over its domain x() (i.e., search space for BC we use the domain-specific context S = [syl] of domain x to obtain the prior distributions p(0,k ayn), where 0thh = (8)))), Yh E [ See Fig. for an illustration of how MPHD generates the surrogate model.
At each iteration of BO, and we compute the MAP estimate for the GP parameters 0f 10
Hy
0, = arg maxg, p(0, Dy. 8) arg max,, p(D, 10,, , Sy. Sy. x- = )) II H p(0 ay = )(ssn)); (6)
where Dy is the set of observations made on function f. We can then use GP parameterized by 0,, GP(ps, (P(py,kg;0p) 0,), to compute the acquisition function. See the full algorithm in $
Withhou loss of generality, we use "search spaces" and "domains" interchangesblly this paper.
9

---

Published in Transactions on Machine Learning Research (02/2024)
In the rest of this section, we present the experiment results and analyses to verify the usefulness of MPHD for decision making in BO. 44. introduces the 3 types of datasets we experimented with. 4.2 lists the compared methods including 2 variants of MPHD and 9 different baselines. 4.3 presents the results on transfer learning for Bayesian optimization.
4.1 Datasets
We used both synthetic data and real-world data in our experiments. The synthetic data used was the Synthetic Super-dataset (L) introduced in $3.2.2 where the ground truth GP parameters for all domains were sampled from their prior distributions. The real-world data were collected on hyperparameter tuning tasks for classic machine learning models (Pineda-Arango et al., 2021 and near state-of-the-art deep learning
models (Wang ct al. 2022
HPO-B Super-dataset (Pineda-Aranaa et al. 2021 is large-scale multi-task benchmark for hyperpa- rameter optimization. As a super-dataset, it consists of 16 different search spaces and more than 6 million evaluations in total. The dimensions of those search spaces vary from to 18.
PD1 Dataset (Wang et al., 2022) is a large hyperparameter tuning dataset collected by training expensive deep neural network models on popular image and text datasets, as well as protein sequence dataset. As a dataset (instcad of a super-dataset), it contains 24 sub-datasets of the same 4-dimensional search space.
For HPO-B Super-dataset and PD1 Dataset, we normalized the range of every domain dimension as well as function values to [0, 1]. Synthetic Super-dataset (L) was generated with every domain dimension in [0, 1], and its function values were kept unnormalized in order to test its ground-truth priors.
Train/test splits: For any super-dataset D [(D,, S of the two, we split every dataset D; in the super-dataset into a training dataset Dirain and a test dataset Di each containing disjoint subset of sub-datasets in D;. As mentioned in $3.2.2 we used 80% sub-datasets within each dataset the training sub-datasets and the remaining 20% the test sub-datasets for Synthetic Super-dataset (L). HPO-B Super-dataset comes with a pre-specified per-datasot train/test split and we used the same sctup.
To show the generalization capability across different real-world datasels, we evaluated the BO performances of MPHD on PD1 (Wang et al.) 2022), where the model was pre-trained only on HPÔ-B (Pineda- Arango ct al.) 2021). Although MPHD models were never pre-trained on PD1 during the evaluation, a train/test split: for PDI is still needed because the homogeneous meta BO methods have to be pre-trained on PD1. Out of the 24 sub-datasets in PD1, we abandoned the ImageNet ResNet50 1024 task it only has 100 datapoints. We randomly sampled 19 (~ 80%) of the remaining 23 sub-datasets as training sub-datasets and used the remaining 4 (~ 20%) sub-datasets as test sub-datasets. For convenience, we denote the entire PD1 Dataset by Dp, its training part. by Dirain and its test part by Djest.
4.2 Experiment setups and compared methods
To test the capability of MPHD to generalize to new tasks with both seen and unseen search spaces, we designed experiments to compare MPHD with competitive meta BO baselines including HyperBO (Wang ct al. 2022). which known to be the state-of-the-art GP prior pre-training method for BO. For MPHD, we tested the performance of both the two variants MPHD Standard and MPHD Non-NN HGP to understand different ways of setting function & in Fig. 2 As in the example shown in 2.. MPHD Slandard employs a 4-dimensional context vector for the length-scale parameter of each domain dimension. This context vector encodes information on whether the domain dimension is discrete or continuous, the number of discrete dimensions, and the number of discrete dimensions in the domain. The NN-based length-scale prior of MPHD Standard has the same architecture as in (3.222] with 2 hidden layers, each of size 16, and utilizes the hyperbolic tangent activation function.
Baselines: (1) Random sampling. (2) A Hand-specified Hierarchical GP prior, fixed across all search
spaces. (3) A Non-informative Hierarchical GP prior, fixed across all search spaces. (4) HyperBO Wang
et al. 2022). (5) ABLR (Perrone et al.) 2018). (6) FSBO (Wistuba & Grabockal 2021). (7) Base GP, a single GP that uses the MLE of GP parameters 0, of the search space that it is being tested in, which is
10

---

Published in Transactions on Machine Learning Research (02/2024)
the same MLE achieved in Step ! of the pro-training of MPHD. This baseline is essentially simplified version of HyperBO that does not have the ML.P base for its GP kernel and uses a constant mean function. (8) The tround-truth Hierarchical GP prior, which is only available for Synthetie Super-dataset (L). (9) The Ground-rruth GP for the test datasct, which is also only available for Synthetic Super-dataset (L).
HyperBO and FSBO use an anisotropic Matérn kernel (u = 5/2) with an MLP base, while ABLR uses a dot-product kernel with an MLP base. The sizes of layers in the MLP base are (128, 128) for experiments on HPO-B Super-dataset and (82,32) on Synthetic Super-dataset. All the remaining GP-base methods use anisotropic Matérn kernel (v 5/2) without an MLP base. HyperBO uses linear mean function with an MLP base ABLR and FSBO use zero mcan function. The rest of the GP-based methods all use a constant mean function. Please see $E.2 for more details of the configuration of baselines, including GP prior parameters, of all compared methods.
Based on whether a method requires pro-training and how it is pre-trained, we can group all compared methods by the following calegories: (1) Methods that are not pre-trained. This category includes Random, the Hand-specified HGP prior, the Non-informative HGP prior, the Ground-trus truth HGP prior, and the Ground-truth GP. (2) Heterogeneous meta BO methods, which are meta BO methods that can generalize over search spaces. This category only includes variants of MPHD. (3) Homogencous meta BO methods, which are meta BO methods that can only train and test on a single search space. This category includes HyperBO, ABLR, FSBO, and Base GP.
When testing the BO performances of compared methods in the search space X1 corresponding to a dataset Dj within super-dataset D (either Synthetic Super-dataset (T) or HPO-B Super-dataset), we used the test. part of that datasct, D!est, to run BO with every tested method. For every method, we report its average normalized simple regret on all test sub-datasets in all search spaces, i.e, all sub-datasets in [Dlet) J Methods that are not pre-trained can be directly tested. In order to test the ability of MPHD to generalize to new functions in seen search spaces as well as unseen search spaces, we designed two settings for its model pre-training. In the default setting, the MPHD models were pre-trained on training sub-datasets from all search spaces in the super-dataset, i.e., {(Djrain S,)) In the second setting denoted by NToT (Not Trained on Test Search Space), the MPHD models were pre-trained on the training super-dataset without the training datasct for the test scarch space, i.c., {(D) " \ {(D)rain S.)) Therefore, in the NToT setting, the pre-trained model of MPHD is tested on functions from an unseen search space. Because homogeneous meta BO methods need to be pre-trained in the same search space used for BO test, their models were pre-trained on {(Dtrain, S:))
For PD1, we report the average normalized simple regret on all test sub-datasets in PD1, i.e., sub-datasets in Djset Same as above, methods that are not pre-trained can be directly tested. As a special case of the NToT setting, when lesting the BO performance of MPHD on PD1 Dataset, the MPHD model were pre-trained on the entire HPO-B Super-dataset {(Dirain 3 but not PD1 Dataset. In this case, MPHD needs to generalize to an unseen search space that is not even in the same super-dataset that it is trained on. Models of homogencous meta BO methods were pre-trained on Deraan, the training sub-datasets in PD1.
For Step ① in the pre-training of MPHD and the pre-training of meta-BO baseline methods HyperBO, FSBO, and ABLR, the GP parameters in every domain were fit by minimizing the NLL of training data in that domain (Eq. 3 using the Adam optimizer (Kingma & Ba 2013 Wistuba & Grabocka 2021). For Step 2 in the pre-training of MPHD, the NN-based hength prior was optimized by minimizing Eq. across all training domains using the Adam optimizer. Hyperparameters of Non-NN priors were directly fit using standard library functions for MLE of Gamma and Normal distributions. When using pro-trained MPHD models for BO, the GP was re- re-trained on the current observations at every BO iteration. The re-training process used the pre-trained model as a prior, aiming to approximate the posterior as defined in Eq. 6. and utilized the L-BFGS optimizer. It is noted that L-BFGS was recommended by Wang et al. (2022) as the standard objective optimization method for HyperBO, while Adam was recommended by Wistuba & Grabockal (2021) that applied the FSBO method on HPO-B. More details of the training processes are available in E.2 and
11

---

Published in Transactions on Machine Learning Research (02/2024)
Random
FSBO (PI)
MPHD Non-NN HGP (PI)
Ground-truth GP (PI)
Base GP (PI)
Hand-specifid HGP (PI)
MPHD Standard (PI)
MPHD Non-NN HGP (NToT) (PD
HyperBO (PI)
Non- con-informative HGP (PI)
Ground truth HGP (PI)
MPHD Standard (NT6T) (PI)
ABLR (PI)
Default Setting
NToT Setting
3 10
Average Normallzed Simple Regret
Average Normallzed Simple Regret
2x
10-1
10
40 Number of BO Iterations
80
100
40
60
80
100
Number of BO Iterations
Figure 7: Results on Synthetic Super-dataset (L). The highlighted areas show mean = std for each method. Left: Results on the default setting where all meta BO methods can use the training data in the test search space. MPHD Standard obtained a similar performance to Ground-truth HGP, showing that the model learned in MPHD a good characterization of the ground truth prior. Note that in practice, BO methods do not have access to the ground-truth priors, it is not realistie to expect that our method can outperform those two ground-truth truth methods. Right: the NToT setting and only methods that do not require pre-training on the test search space are included. MPHD Standard (NToT) outperformed other methods, demonstrating the ability of MPHD to generalize to unseen search spaces.
4.3 Results Bayesian optimization
For all of the following experiments, the budget for BO is 100, and there are a set of 5 initial observations that are randomly sampled for each of the random sceds. The acquisition function used for all GP-basod methods is Probability of Improvement (PI) (Kushner, 1964) with target value max(ı:) + 0.1. As shown by WWan & Jegelka (2017), PI can obtain high BO performance by setting good target values. Results on other acquisition functions included in F
Synethetic Super-dataset (T). Fig. 7 (left) shows the BO performances of compared methods on Synthetic Super-datasel (I.). MPHD Standard, which has an NN-based length-scale prior model, outperformed all the baselines except for the Ground-truth HGP and Ground-truth GP. This demonstrates the ability of MPHD effectively Icarn good prior model across heterogencous search spaces during pro-training. Morcover, while Base GP should be more customized to the search space than MPHD Non-NN HGP, MPHD Non-NN HGP was able to achieve much better performance than Base GP: this shows the importance of Step ? in MPHD regardless of using domain-specific contexts. Another observation is that MPHD Standard outperformed MPHD Non-NN HGP by a large margin, demonstrating the effectiveness of using an NN to capture the dependence of domain-specific priors on the context features.
Interestingly, HyperBO achieved better performance in the initial few BО iterations but it fell behind other methods after about 10 BO iterations for the test functions in Synethetic Super-dataset (L). One possible reason is that the posterior GPs in HyperBO significantly deviated from the ground truth GP. As noted in Wang et al. 2022). the difference between the predicted posterior variance and the ground truth posterior variance increases with the number of observations. This can be a critical issue for complex datasets like Synthetic Super-dataset (L), though it was not a severe problem for HPO-B and PD1 (results below). FSBO and ABLR can be viewed as special cases of HperBO with zero mean function, which makes it difficult obtain a good prior or posterior compared to the ground truth. On the contrary, the posterior inference in MPHD) done for hierarchical GP instead of single GP in HyperBO. Since hierarchical GP
12

---

Published in Transactions on Machine Learning Research (02/2024)
Random
ABLR (PI)
Non-informative HGP (PI)
MPHD Non-NN HGP (NToT)
Base GP (PI)
MPHD Non-NN HCP (PI)
MPHD Standard (NToT) (PD
HyperBO (PI)
Hand-speciiied HGP (P1)
MPHD Standard (PI)
Default Setting
NToT Setting
Average Normallzed Simple Regret
Average Normallzed Simple Regret
10-1
10-1.
20
40
100
20
40
100
Number of BO Iterations
Number of BO Iterations
Figure 8: Results on HPO-B Super-dataset. Left: in the default setting, HyperBO had superior performance at first during the BO iterations, but MPHD Standard eventually achieved the lowest regret after 100 BO iterations. Right: in the NToT setting, MPHD Standard (NToT) achieved the best performance among all methods that do require pro-training on the test search space.
typically assign probability density to larger space of functions than single GP, MPHD was able to achieve much more robust performance than HyperBO.
Fig. 7 (right) shows the BO results on Syuthetic Super-dalasel (L) of compared methods the NToT setting where the search space used for BO test is not used for pre-training. HyperBO, ABLR, and FSBO cannot be tested in this setting as they require pre-training in homogeneous search spaces for each test function. MPHD Standard in the NToT setting achieved superior performance over baseline methods such the Hand-specified HGP prior and the Non-informaaive HGP prior, which demonstrates the ability of MPHD to generalize its pre-trained model to functions in new search spaces that are unseen during pre-training, Similar to the results in Fig. 7 (left), MPHD Standard again outperformed MPHD Non-NN HGP, showing the importance of using domain-specific priors.
HPO-B. Fig. (left) shows the BO performances on HPO-B Super-dataset. MPHD Standard achieved lower final regrets than all baseline methods. HyperBO had superior BO performance than MPHD Standard when the number of BO iterations was smaller than 80 but was eventually overtaken by MPHD Standard. As explained above, the performance plateau of HyperBO and FSBO could be related to the increase of the posterior approximation errors Wang et al. (2022). Similar to the observation Synthetic Super-dataset (L.,, MPHD variant with an NN-based lengah-scall prior model outperformed MPHD Non-NN HGP.
Fig. (right) shows the BО results on HPO-B Super-dataset of compared methods in the NToT setting. Notably, MPHD Standard (NToT) performed only slightly worse than MPHD Standard pre-trained in all search spaces even though MPHD Standard (NToT) was not trained the scarch space was tested in. The good performance of MPHD Slandard (NToT) further demonstrates the capability of MPHD to generalize across search spaces in real-world problems. MPHD Standard (NToT) outperformed all other methods in this setting, again demonstrating its ability to generalize to new test functions in unseen scarch spaces.
PD1. Fig. g (left) shows the BO results of methods valid in the NToT setting PD1 Dataset. Here the MPHD models were pre-trained on training datasets in HPO-B Super-datasel, but were not pre-trained on PD1. MPHD Standard (NToT) achieved the best performance in the NToT setting. Even though HPO-B and PD1 are two separately collected real-world hyperparameter tuning datascts, MPHD was capable of generalizing the model pre-trained on HPO-B to test functions in PD1. Fig. 9 (right) also includes the BO performances of the homogenous meta BO methods PD1 Dataset. MPHD Standard (NToT)
13

---

Published in Transactions on Machine Learning Research (02/2024)
Random
Non-NN HGP (NToT) (PI)
Base GP (PI)
ABLR
Hand specified HGP (PT)
MPHD Standard (NToT) (PI)
HyperBO (PI)
FSBO (PI)
Non- non-Informative HGP (PI)
NToT Setting
Including Homogeneous Meta-BO Methods
Average Normalizod Simple Regret
Average Normalized Simple Regret
10-1
10-1
10-2
10-2
10-3
20
60
20
100
Number of BO Iterations
Number of BO Iterations
Figure 9: Results on PD1 Dataset. Left: in the NToT setting, MPHD Standard (NToT) achieved the best performance among all methods that do not require pre-training on the test search space, demonstrating its ability to generalize to unseen search spaces. Right: we include homogeneous meta BO methods in addition NToT methods. Homogencous meta BO methods HyperBO, ABLR, and FSBO outperformed MPHD Standard (NToT), which expected since these methods were directly pre-trained on PD1 Dataset while MPHD Standard (NToT) was not.
outperformed by HyperBO, ABLR, and FSBO, which is not surprising as these singlo-searcb-space baselines were pre-trained training sub-datasets in PD1 while MPHD Standard (NToT) was not.
5 Discussions and conclusions
In this work, we propose MPHD, the first GP-based transfer learning BO method that works on heterogeneous search spaces. The key idea is to pre-train a hierarchical GP with domain-specific priors on training data from functions in different domains. MPHD does not need access to BO trajectories the format of an ordered list of data points. Instead, MPHD can effectively make of an unordered set of datapoints as long they are partitioned to different functions and different domains (i.e., super-datases in $2.1 Our theoretical and empirical analyses showed that MPHD enjoys appealing asymptotic properties and performs competitively on challenging hyperparameter tuning tasks, making it both theoretically sound and a practical transfer learning method.
Limitations and future work
For a BO task, MPHD only learns the prior model in the form of a domain-specific hicrarchical GP. While this allows a separation of model and decision making strategy, there are other components that can also be meta-learned, such as acquisition functions (Volpp al. 2020), to maximize the effectiveness of the BO system. One direction of future work is jointly pre-train all components of BO to allow more flexibility and further improve the performance.
Like most machine learning methods, MPHD is subject to assumptions, including stationary kernel function, a constant mean function and the availability of a super-datasel with domain-specific contexts. Possible future work includes relaxing assumptions on kernel and mean functions and incorporating architecture search to enrich the space of hicrarchical GP priors. The assumptions on data are naturally satisfied in our experiments such as HPO-B, since our context encodings only require input dimensions and whether the input is continuous. However, for other types of data, the domain-specific contexts might not be real
14

---

Published in Transactions on Machine Learning Research (02/2024)
vectors. Our work builds strong foundation for generalizing to those more complex contexts if they can be encoded as real vectors, and future work is to work with those more complex domain-specifee contexis.
Broader impact
Hyperparameter tuning for machine learning (ML) models, especially deep learning models, can be very costly if we repeatedly evaluate large number of hyperparameters. Each single evaluation of hyperparameter value requires training and testing new instantiation of the model. Our framework MPHD, with its superior BO performance discussed in $4.3 can help to reduce the number of evaluations needed for hyperparameter luning tasks and thus reduce their computational cost and carbon footprint potentially by a large margin.
Acknowledgments
We thank Richard Zhang for feedback, and Jasper Snoek and Eytan Bakshy for helpful comments on the previons version of this work (Fan et al., 2022), which was presented at NeurIPS 2022 Workshop on Gaussian Processes, Spatiotemporal Modeling, and Decision-making Systems. The koy difference to this previons work is the inclusion of domain--peccifi hierarchical Gaussian processes as opposed to using a universal model. Our work also benefited from Microsoft Azure credits provided by the Harvard Data Science Initiative, well Google Cloud Platform Credit Awards provided by Google.
References
François Bachoc. Asymptotic analysis of the role of spatial sampling for covariance parameter estimation of Gaussian processes. Journal of Multivariate Analysis, 125:1-35, 2014.
François Bachoc. Asymptotic analysis of maximum likelihood estimation of covariance parameters for Gaussian processes: an introduction with proofs. In Advances in Contemporory Statistics and Econometrics, pp. Springer, 2021.
François Bachoc, José Betancourt, Reinhard Furrer, and Thierry Klein. Asymptotic properties of the maximum likelihood and cross validation estimators for transformed Gaussian processes. Electronic Journal of Statistics, 14(1):1962 2008, 2020.
Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. Advances in Neural Information Processing Systems (NeurIPS), 2020. URT. http://arxiv. wttpe/ivvvvrrraas///eeeeee orgg/bss/119 06403
Moreno Bevilacqua, Tarik Faouzi, Reinhard Furrer, and Emilio Porcu. Estimation and prediction using generalized wendland covariance functions under fixed domain asymptotics. The Annals of Statistics, 47 (2):828-8566 856, 2019.
Yutian Chen, Matthew W Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy P Lillicrap, Matt Botvinick, and Nando de Freitas. Learning to learn without gradient descent by gradient descent. In International Conference on Machine Learning (ICML), 2017.
Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Qiuyi Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnand Doucet, Marc'aurelio Ranzato, et al. Towards learning universal hyperparameter optimizers with transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Zhou Fan, Xinran Han, and Zi Wang. Hyperbo+: Pre-training a universal prior for Bayesian optimization with hierarchical gaussian processes. arXio preprint arXio:2212. 0.tt2212..0538. 2022.
Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for Bayesian optimization. arXiv preprial Xrxi 1802 02219, 2018
Peter Frazier and Jialei Wang. Bayesian optimization for materials design. In Information Science for Materials Discovery and Design, pp. 45-75. Springer, 2015.
15

---

Published in Transactions on Machine Learning Research (02/2024)
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Elliot Karro, and D. Sculley. Google Vizier: A service for black-box optimization. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2017.
Toni Karvonen and Chris J Oates. Maximum likelihood cstimation in Ganssian process regression is ill-posed. arXiv preprint arXiv:2203. 09179, 17, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http::/arxiv..om abs/ mets//112..600 6980
Harold Kushner. A new method of locating the maximum point of arbitrary multipeak curve in the presence of noise. Journal of Fluids Engineering, 86(1):97-106 1964.
Erich L Lehmann and George Casella, Theory of point estimation. Springer Science & Business Media, 2006.
Kanti V Mardia and Roger J Marshall. Maximum likelihood estimation of models for residual covariance in spatial regression. Biometrika, (11):1355146, 1984.
Valerio Perrone, Rodolphe Jenatton, Matthias Seeger, and Cédric Archambeau. Scalable hyperparameter transfer learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 6846 6666,, 2018.
Sebastian Fineda-Arango Hadi S. Jomaa, Martin Wistuba, and Josif Grabocka. HPO-B: A large-scale reproducible benchmark for black-box HPO based openml. Newral Information Processing Systems (NeurIPS) Track Dalasels and Benchmarks, 2021.
Carl Edward Rasmussen and Christopher KI Williams. Gaussian processes for machine learning. The MIT Press, 2006.
John A Rice. Mathematical Statistics and Data Analysis. Cengage Learning, 2006.
Benjamin J Shields, Jason Stevens, Jun Li, Marvin Parasram, Farhan Damani, Jesus Martinez Alvarado, Jacob M Janey, Ryan Adams, and Abigail G Doyle. Bayesian reaction optimization as tool for chemical synthesis. Nature, 590(7844):89 96, 2021.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems (NewrIPS), 2012.
Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In International Conference on Machine Learning (ICML), 2010.
Michael Stein. Interpolation of Spatial Data: Some Theory for Kriging. Springer Science & Business Media, 1999.
Kevin Swersky, Jasper Snock, and Ryan Adams. Multi-task Bayesian optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
Ryan Turner, David Eriksson, Michael McCourt, Juha Kiili, Eero Laaksonen, Zhen Xu, and Isabelle Guyon. Bayesian optimization is superior to random scarch for machine learning hyperparancter tuning: Analysis of the black-box optimization challenge 2020. In NewrIPS 2020 Competition and Demonstration Track, pp. 3 26. PMI.R, 2021.
Aad W Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
Michael Volpp, Lukas Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisiiion functions for transfer learning in Bayesian optimization, 2019.
URL httph:tttp:rrtee wwwp://wwgg.b....ooooooo
16

---

Published in Transactions on Machine Learning Research (02/2024)
Michael Volpp, Lukas P Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning in Bayesian optimization. In International Conference on Learning Representations (ICLR), 2020.
Dennis Wackerly, William Mendenhall, and Richard I. Scheaffer. Mathematical Statisties with Applications. Congage Learning, 2014.
Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient Bayesian optimization. International Conference on Machine Learning (ICML), 2017.
Zi Wang, Beomjoon Kim, and Leslie Pack Kaelbling. Regret bounds for meta Bayesian optimization with an unknown Gaussian process prior. In Advances in Newral Information Processing Systems (Neur/PS), 2018.
Zi Wang, Caclan Reed Garrett, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Learning compositional models of robot skills for task and motion planning. International Journal of Robolics Reseurch (IJRR), 40 (6-7):866 19086669444,, 2021.
Zi Wang, George E Dahl, Kevin Swersky, Chansoo Lee, Zelda Mariet, Zachary Nado, Justin Gilmer, Jasper Snock, and Zoubin Ghahramani. Pro-trained Gaussian processes for Bayesian optimization. arXiv preprint arXiv:2109008211 2022.
Martin Wistuba and Josif Grabocka. Few-shot Bayesian optimization with deep kernel surrogates. International Conference on Learning Representations (ICLR), 2021.
Zhi-Sheng Ye and Nan Chen. Closed-form estimalors for the gamma distribution derived from likelihood equations. The American Statistician, 71721:177-181, 2017.
Dani Yogatama and Gideon Mann. Efficient transfer learning method for automatic hyperparameter tuning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2014.
Hao Zhang and Dale L Zimmerman. Towards reconciling two asymptotic frameworks in spatial statisties. Biometrika, 92( 4):921 -):92- 2005.
17

---

Published in Transactions on Machine Learning Research (02/2024)
Table Problem formulation notations
Notation
Description
X0
domain.
F. (fisy M.
functions in domain X(0),
GP; GP(pi, 04)
Gaussian process that generates functions Fj in domain X10
44
mean function of GP,,
&
kernel function of gP;,
0:
parameters of gP;-
hyperparameter (distribution parameters) for the h-th parameter On of GP.
S.
= hml
Hi
context for dataset i.
6
model, i.c., mapping from Sih to agh- sub-dataset.
D3 {(1gg Vij M
!)) (1
M. D. {
dataset.
= {(D;, S.
super-dataset.
D D2 U.... DM,
augmented pseudo sub-dalaset in Lemma
D (2010) (
datapoints the pesudo sub-dataset.
A Table of notations
Table provides a list of the notations used in this paper.
B Laplace approximation of integrals in the derivation of Eq
We derive Eq as follows.
N ((Ø)= Σ
log
p(D; 10)p(( 1 o 8,) d0,
8,
N
log p(0; o. J,)e d0;
1=1
8,
log / p(0; Φ, S.ye Me log (ogp(Di,)0.)) d0;
J..
N
log / p(0; d. Si)e M.)
Mi 2b, E/4 logg(Du)00) d0,.
For every i = N,...,N,1 1, 1....,N, 0, 10 For the following derivation, assume that there is reasonably large cap every dimension of 0, so that the domain of 0, is compact set K, C. R. Let h,(0,) = p(0, Φ S:) and f((O,) = 1 M, log logp(D;; Di (t). Here h; is continuous on K; and fa is twice continuously differentiable on K,. Let 0 = arg ming fe(O,) = arg max p( 02)- 0 is a global minimizer of fa on K2 and have (.,).) O. We also assume that f" (0.)) is a positive definite matrix. When
M, Σ
18

---

Published in Transactions on Machine Learning Research (02/2024)
N, we have M, for i ,,
N
p(O; I S, Mi Mi(- E (ogn(D))10.) d0.
[())
log
8,
log
h,(0,)e Nese((9)) dd:
((0.)e-MMss((1))
A.EK
N
(2T) H
M1 :i det (,.) he(0.)e (..(( d8,
log
N
(2T)
log
p(0: 10, S:)p(D; (.)
M. det a0 (-
1
Σラ log Dig 0:))
i=1
M.
(2x)
log
p(0: Φ. S:)p 0.)
M,a det d2 an=
M, logp(D; 0,))
Here the approximation step of Eq. uses the Laplace approximation of integrals.
C More details of theoretical analysis
Theorem Discussion Recall from Lemına 1 that the the pseudo sub-dataset D satisfies the increasing- domain configuration. Its collection of input locations comes from the sub-datasets {Dij))= (each Dij has
Lij observations) and are denoted as x((1) ,x(2) .... ,x( where T Σド Lig. Respectively, we denote the observations at those locations as ((²) yy12) .......))))) It obvious that M; + oo we will have T oc. Therefore, to show the asymptotic property of the maximum-likelihoon estimator 6, we leverage the properties derived in previous work on the increasing domain setting. That is, aim to show T - oc, it is true that 0, 4 03.
Similar to (3), we define the observations in D as vector y(a)) = [(y(1) (1) 0,) and use K (()) to denote the invertible covariance matrix in D: ((().) = (xx=(1((((')) ; Let (r,),) be function proportional to the negalive log-likelihood of T observations:
1 LT T log(IK + 7 (v1Oi)) T (A10))) -1 (y)a.))
Then the maximum likelihood estimator for 0g is given by
0, arg ming,ec, (0,).)
Suppose that assumptions (1)-(3) in section 3.1 hold for zero-mean and anisotropic Matérn kernel whose length-scale parameter 0, is to be estimated,
derive the following lemma:
Lemma 2.1 (proved as Lemma 2 in Bachoc 2021 For any 0, E C2, as T + co:
var var()r(0.) = o(1)
This lemma shows that for any e > 0, there exists T large enough such that the variance of the likelihood function on the set of T observations satisfies var(Sr(0.)) < é in probability.
Then we proceed with the proof for Theorem following Bachoe 20221):
Proof of Theorem 2 (Bachoc (2021) Theorem 1): From Lemma 2.1 we have that for any 0g E C2:
(r)), E[2r(0)) B 0 as T 00.
We can also obtain that
sup (2r(0,) - E[Sr((,)] (1). 8,EC,
(8)
19

---

Published in Transactions on Machine Learning Research (02/2024)
Previous work from Bachoc (2014) 0g C::
(Proposition 3.1) shows that there exists constant A >> 0 such that for
- 2 A
Σ f,==1
(k((<(0)<()) - k,(x(0), ,00)
(9)
Combining the above expression and the asymptotic identifiailliyy condition, we obtain that for 0, there is strictly positive constant B such that for T large enough,
inf 1.c.116.-8111222 (E(2r((0)] ((7))) B
(10)
With Eq.18
and since 0, belongs to the M-estimator class, according Theorem 5.7 of Van der Vaart we conclude that as M; we have T oo and thus the maximum likelihood estimator 0; 4 0r.
10
2000
Proof Theorem According to Theorem 2, as the number of sub-datasets M, + oc, we have 0, - 02, i.e. each maximum-likelhhod estimator O is a consistent estimator of the ground truth GP parameter 0r_ the problem formulation we assume that each GP parameter 0, is sampled i.i.d. from the prior distribution (aas)). Therefore, using previous results on properties of MLE given i.i.d. samples (Wackerly el al. 2014), we know that the maximum--ikellhoon estimator dgh in general satisfies the consistency and asymptotic normality properties. That is, with mild regularity constraint as N + xe, we have âss - ai
Note that the MLE can fail to be consistent when certain regularity condition violated. For instance, when the parameter is not identifiable, L.c. when multiple distinct ai cxists.
Meanwhile, the authors of Karvonen & Oates (2022) pointed out that the MLE for the length-scale parameter is indeed pll-posed when the data is noiseless and can be unstable to small perturbations. However, they found that regularization with small additive Ganssian noise docs guarantee well-posedness and this is equivalent to data corrupted by additive Gaussian noise. This is identical to our setting with the synthetic data and we found that in practice an accurate estimation for noise allows reliable inference for covariance parameters.
D Algorithm of MPHD
Algorithm MPHD pre-training and Bayesian optimization with acquisition function ac(:;
1: function MPHD (D = (.)...)
2: for = 1,.., N do
0. < PRE-TRAIN-STEP (D,)
3:
4: end for
3 PRE-TRAIN-STEP STANNNSTEE--(((,,, (10,,
Dj
< -0
for = -,.... . do do
0, arg maxo, p(0; D S, 3) (Eq. 6
8:
Dt arg maggmaxoo ac, (x;0, 0,) zEXu
10:
9t + OBSERVE(f(2:)
11:
D, < D2 ((1,,
12: end for
13: return Dj
14: end function
Algorithm 1 shows the model pre-training of MPHD and how the learned model & is used in BO on new function f. Here the acquisition function ac(:; 0,) is a specific type of acquisition function (e.g., PI) associated
Forr instance, this requires the log-likellhood function being twice differentiable w.r.t. 0gh. More thorough discussions other regularity conditions be found in Lehmann & Casella 2006
20

---

Published in Transactions on Machine Learning Research (02/2024)
with a GP parameterized by 8,. The GP serves the surrogate model for function f during BO. While the ground-truth GP parameter o for the function f is unknown, the learned model & of MPHD used to achieve an estimate Or and the estimate is used to parameterize the GP associated with the acquisition function.
More details of the experiment setups E
Our code for the experiments is built upon the codebase of HyperBO (Wang et al, 2022 and is available at https:///iitub.
E.1 Details on the two synthetic super-datasets
Both synthetic super-datasets were generated using a constant mean function and an anisotropic Matérn kernel with a known smoothness parameter. In our setting, each dataset D; (i = .,....N)) corresponds to GP, parameterized by 0, which includes the following parameters: constant mean (the value of the constant mean function), length-scale (which has the same number of dimensions as the domain 200),, signal variance, and the noise variance.
For the prior distribution types of each of these GP parameters, use Normal distribution for constant mean and use Gamma distribution for each of the remaining parameters. A Gamma distribution is parameterized by (shape) and b (rate), while a Normal distribution is parameterized by e (mean) and d (standard deviation).
Synthetic Super-dataset (S) a smaller super-dataset where the same GP prior is shared across all domains. It includes 20 datasets (domains) with 10 sub-datasets cach dataset. Each sub-datasct includes noisy observations 300 input locations in its respective domain. The dimensions of the domains were randomly sampled between 2 and 5. As all the datasets are i.i.d. samples, used the first 16 datasets to be training datasets and the remaining 4 datasets to be test datasets when testing the empirical asymptotic behavior of the pre-training of MPHD in $3.2.1 The underlying GP uses constant mean function and an anisotropic Matérn kernel with smoothness parameter y = 3/2. The specific prior hyperparameters for the GP parameters are as follows: constant mean sampled from Normal(c = 1.0, = 1.0), each dimension of length-scale is sampled from Gamma(a = 10.0, = 30.0), signal variance is sampled from Gamma(a = 1.0, 1.0), and noise variance is sampled from Gamma(a 10.0, 100000.0).
Synthetic Super-dataset (L) is larger super-datasct and has domain-specific GP priors. It includes 20 dalasets (domains) with 20 sub-datasets in each datasel. Each sub-dataset includes noisy observations at. 3000 input locations in its respective domain. The dimensions of the domains were randomly sampled between 2 and 14. For every datasct, we used 80% of its sub-datasets for training and the remaining 20% for testing. We used the first 16 datasets for training when testing the empirical asymptotic behavior of the pre-training of MPHD $32.22 while all 20 datasets were used for experiments I The underlying GP uses a constant. mean function and an anisotropic Matérn kernel with smoothness parameter v = 5/2. The prior for GP parameter length-scale is domaín-specifie and the prior hyperparameters have linear dependence number of domain dimensions. For a domain ( c Rdd, the length-scale GP parameter of every domain dimension is sampled from Gamma(a 0.07692d; + 0.8462, = -0.3539d, + 5.7077). The priors for other GP parameters (constant mean, signal variance, and noise variance) are not domain-specific, and the specific hyperparameters as follows: constant mean is sampled from Normal(c = 0.5, d = 0.2), signal variance is sampled from Gamma(a 15.0, 100.0), and noise variance is sampled from Gamma(e = 1.0, b 10000.0).
the
E.2 Details on the compared methods
In this section, we provide additional details on the compared methods in $
MPHD Standard, MPHD Non-NN HGP, Base GP, Non-informative HGP, Hand-specified HGP, Ground- truth GP, and Ground-ruuth HGP all use constant mean function and an anisotropic Matérn kernel with smoothness parameter 5/2.
21

---

Published in Transactions on Machine Learning Research (02/2024)
HyperBO and FSBO use anisotropic Matérn kernel (v = 5/2) with an MLP base, while ABLR uses dot-product kernel with an MLP base. HyperBO uses linear mean function with an MLP base. ABLR and FSBO use a zero mean function. The size of the MLP bases used for the mean and kernel in HyperBO, FSBO, and ABLR is (128, 128) on HPO-B and (32, 32) on Synthetic Super-dataset (L).
For the prior distribution types of each of these GP parameters, MPHD Standard, MPHD Non-NN HGP, Non-informative HGP, Hand-specified HGP all use a Normal distribution for constant mean and use Gamma distribution for each of the remaining GP parameters. Ground-trute truth HGP also has the same distribution types as these are the distribution types used when generating Synthetic Super-dataset (L).
MPHD Standard uses an NN-based length-scale prior. The sizes of the hidden layers in the NN are (16, 16) and the act activation function is hyperbolic langent. The output layer is 2-dimensional, which represents the two hyperparameters of the prior distribution for length-scale (shape and rate of the Gamma distribution). The context S; [8th H. used in MPHD encodes information on whether every domain dimension discrete or continuous and the total number of discrete and continuous dimensions in the domain (i (ifi=la 1,.... :.. N). For every domain dimension in a domain (0),, the context s;h associated with the length-scale of this domain dimension is 4-dimensional vector, where the first 2 dimensions are one-hot encoding to specify whether this domain dimension is discrete or contimuous, and the remaining 2 dimensions of the context are the number of discrete and continuous dimensions in the domain X1(0, respectively. This context can be automatically constructed for any given dataset to be used by MPHD Standard.
k=1
Hand-specified HGP uses a hand-specified (and therefore could be misspecified) prior for each GP parameter, and the same prior is used for all domains. The specific hyperparameters are as follows: constant mean is sampled from Normal(c 0.5, d 0.5), each dimension of length-scale is sampled from Gamma(a = 1.0, = 0.1), signal variance is sampled from Gamma(a = 1.0, 5.0), and noise variance is sampled from Gamma(a = 1.0, 6 = 100.0).
Non-informative HGP uses a Uniform distribution as the prior for cach GP parameter. The specific hyperparameters are as follows: constant mean is sampled from Uniform(0.0. 1.0), each dimension of length- scale is sampled from Uniform(0.00000) 30.0), signal variance is sampled from Uniform(0.0000)) 1.0), and noise variance is sampled from (0000rrm000000....
Ground-truth GP uses the ground-truth GP parameters of every domain in the Synthetic Super-datasel (L).
Ground-truth HGP uses the ground-trut prior hyperparameters for every GP parameter type in the Synthetic Super-dataset (L.,, including the domain--peccfic length-scale prior.
Optimizations. For Step 1 in the pre-training of MPHD and the pre-training of meta-BO baseline methods HyperBO, FSBO, and ABLR, we fit the GP parameters in every domain by minimizing the NLL of training data in that domain using the Adam optimizer. The number of iterations for the Adamn optimizer is 20000, and each sub-dataset is randomly sub-sampled to 50 observations at each iteration. The learning rate of Adam optimizer is 0.001. For Step ② in the pre-training of MPHD, the NN-based length-scale prior in MPHD Standard is also optimized using Adam optimizer. The number of iterations is 10000, and the learning rate is 0.001. Hyperparameters of Non-NN priors (including the priors for constant mean, signal variance, noise variance in MPHD Standard, and priors for all GP parameters in MPHD Non-NN HGP) are directly fit using standard library functions for MLE of Gamma and Normal distributions. When using pre-trained MPHD models for BO, the GP is re- re-trained on the current observations at every BO iteration. The re- re-training uses the pre-trained model as the prior and its purpose is to approximate the posterior defined in Eq. The optimizer for this re- re-training is L-BFGS and the number of iterations is 100. Furthermore, L-BFGS was recommended by Wang et al. (2022) as the standard objective optimization method for HyperBO, while Adam was recommended by Wistuba & Grabocka (2021) that applied the FSBO method HPO-B.
E.3
BO Setups
When testing the BO performances of compared methods, the BO budget is 100, and random observations are made prior to BO iterations for initialization. All methods are tested on 5 random seeds, while the set of
22

---

Published in Transactions on Machine Learning Research (02/2024)
Mean Estimate Ground-truth
Mcan Estimate
0.45
Ground-rren -arut
Longth Scaleeaaee
Constart
0.0
0.35
0.2
0.30
20
40
60
80
100
20
40
60
80
100
Number of Sub-datasets
Number of Sub-datasets
(a) Length-scale
(b) Constant Mean
3.0
Ground-truth
Ground-truth
Sighal Varanca
2.5
Noise Verfaniee
8.0000
2.0
1.0
108000
0.5
20
40
60
80
100
20
40
60
80
100
Number
Number sets
(c) Signal Variance
(d) Noise Variance
Figure 10: Estimated length-scale, constant mean, signal variance, and noise variance parameters (distribution plotted over 50 random runs) of a 1-dimensional GP as the number of sub-datasets increases. Each sub-dataset has 25 obsorvations. The variances of the estimated GP paramcters decrease as the number of sub-datasets increases.
the initial 5 observations is the same for all methods given the same random seed. HPQ-B Super-datasel provides 5 groups of initial observations, and each group contains 5 random observations for every test sub-dataset. Therefore, we just used the provided initial observations for BO on HPO-B. We randomly sampled 5 initial observations for each random seed and every test sub-dataset.
The PI acquisition function has parameter ç= 0.1, which means its target value is the maximum observation plus Ç. The El acquisition function uses the maximum observation the target value. The GP-UCB acquisition function has parameter 3 = 3.
F Additional experiment results
In this section, present additional empirical evidence on the asymptotic behaviors of the pre-training method of MPHD, well as analyses on the performance of Bayesian optimization.
F.1 Asymptotic behavior of fitting single GP
Fig. 10 demonstrates the asymptotic behavior empirically for fitting the parameters of a single GP using an increasing number of sub-datasots with simulations on synthetic data generated with 1-dimensional GP. The ground-trute touth GP parameters are shown in the figure. The variances of estimated GP parameters decrease the number of sub-datasets increases.
23

---

Published in Transactions on Machine Learning Research (02/2024)
2.3
Estimated é of Constant Prior
Mean Estimate Ground-Lruth
Ground-trute
* of Training Datasets
Probabilty Density
2.0
of Training Datasets 16
1.0
1.3-
0.5
1.0-
0.0
0.3
0.0
2.3
5.0
15.0
0.0 Constant
1.0
Number of Training Datasets
(a) Constant Mean Mean parameter c of Normal Distri- bution
(b) Constant Mean Normal Distribution
100
Ground
Estimated of Signal Variance
Ground-truth
Training Datasets
80
Probsbility Density
Training Datasets 16
4
60
40
2-
20
1.
0.
2.5
3.0
7.5
10.0
12.5
15.0
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Number Training Datasets
Signal Variance
(c) Signal Variance Shape parameter a of Gamma Distri-
(d) Signal Variance
Gammaa Distribution
bution
Estinated á of Noise Varianze Prior
Mean Estimate
Ground truth 12000
20
Ground-truth
of Training Datasets 2
Probsblity Density
10000
of Training Datasets 16
15
8000
6000
4000
5-
2000
01
0
2.5
5.0
7.5
10.0
12.5
15.0
0.0000
0.0002
0.0b04 0.0006 0.0008 0.0010 Noise Variance
Number Training Datasets
(e) Noise Variance Shape parameter of Gamma Distri- bution
(f) Noise Variance Gamma Distribution
Figure 11: For empirical asymptotic analysis on Synthetic Super-dataset (S) with a fixed ono-dimensional length-scale prior, we plot on the left the estimated distribution parameter for GP parameters constant mean, signal variance, and noise variance as the number training datasets increases, and on the right. the probability density functions of the distributions that model each of the GP parameters for N = and N = 16 together with the ground-truth distributions. Results with 5 random seeds are shown.
24

---

Published in Transactions on Machine Learning Research (02/2024)
F.2 Asymptotic behavior of the pre-training method of MPHD
Fig. shows on the left the estimated prior distribution parameter (mean c for Normal distribution and shape a for Gamma distribution) for GP paramcters including constant mean, signal variance and noise variance w.r.l. the number of training datasets (the results for length-scale are shown in Fig. 5 in the main paper). This is demonstrated only on Synthetic Super-dataset (S) since there is no known ground-truth prior for real-world super-datasets such as HPO-B.
In addition, the estimated prior distributions for each GP parameter with 2 training datasets and 16 training datasets along with the corresponding ground-truth prior distribution are shown on the right of Fig. 11 (the results for length-scale are shown in Fig. 5 in the main paper). For most of the GP parameters (length-scale, constant mean, and signal variance), the estimated prior distribution parameter shows decreasing variance and gets closer to the ground-truth value as the number of training datasets increases.
The estimated prior distribution also gets much closer to the ground-truth prior distribution when 16 training datasets are used compared to only using 2 training datasets. For the noise variance parameter, the a parameter of Gamma distribution is estimated to be a different value than the ground-truth value, and the estimated prior distribution shows a different shape than the ground-truth prior distribution even when using 16 training datasets. Note that each dataset of Synthetic Super-datasel (S) contains only 10 sub-dat asels. This small number of sub-datasets in each search space is likely part of the reason for the difference between the estimated noise variance prior and the ground-truth prior. The variance of the estimated parameter of Gamma distribution for noise variance still decreases the number of training datasets increases. The visualization shows that the estimated prior distributions for noise variance also put most of the probability density over values between [0, 0.0002] as the ground-truth prior distribution does despite the difference in their shapes.
F.3 BO performances with different acquisition functions
4.3 shows the BO results with the Probability of Improvement (PI) acquisition function, and here we present additional BO results with other acquisition functions including Expected Improvement (ET) and GP-UCB (UCB).
Synethetic Super-dataset (L). Fig. 12 (left) shows the BO performances of compared methods on Synthetic Super-dataset (L) with acquisition functions UCB and EI. Echoing the results with PI in $4.3 MPHD Standard outperformed all the baselines except for the Ground-truth HGP and Ground-truth GP with either UCB or EI as the acquisition function. Moreover, the final regret of MPHD Standard matches that of the Ground-truth GP, which is rather impressive.
Fig. 12 (right) shows the BO results on Synthetic Super-dataset (L) of compared methods with UCB and EI in the NToT setting where the search space used for BO test is not used for pre-training. With either UCB or EI, MPHD Standard (NToT) outperformed baseline methods in the NToT setting. It can also be observed that the performance MPHD Standard dominates MPHD Non-NN HGP in both the default setting and the NToT setting, with either UCB or El.
HPO-B. Fig. 13 (left) shows the BO performances on HPO-B Super-dataset with acquisition functions UCB and EI. With EI as the acquisition function, MPHD) Standard achieved lower final regrets than all bascline methods. With UCB as the acquisition function, HyperBO has better performance than MPHD Standard. Notice that MPHD aims to recover the ground-truth prior distribution, and as we have shown here, need a good acquisition function to unleash the best performance of BO given pre-trained prior. Using misspecified priors, some sub-optimal acquisition functions may actually lead to better performance than an acquisition function that achieves better results with the ground-truth prior.
Fig. 13 (right) shows the BO results on HPQ-B Super-dataset of compared methods with UCB and El in the NToT sctting. With either UCB or EI as the acquisition function, MPHD Standard (NToT) outperformed all other methods in the NToT setting. Similar to the observation on Synthetic Super-dataset (L), MPHD variant with NN-based length-scale prior model outperformed MPHD Non-NN HGP.
25

---

Published in Transactions on Machine Learning Research (02/2024)
Random Base (UCB)
FSBO (UCB)
MPHD Non-NN HGP (UCB)
Ground-truth (UCB)
Hand-specified HGP (UCB)
MPHD Standard (UCB)
MPHD Non-NN HGP (NToT) (UCB)
HyperBO (UCB)
Non-informative HCP (UCB)
Ground-truth HGP (UCB)
MPHD Standard (NTrT) (UCB)
ABLR (UCB)
Default Setting
NToT Setting
3
30-1
Average Normalzzd Simple Regret
Average Normalized Simple Regret
x
10
20
40
60
80
100
20
40 Nunber of Iterations
60
80
100
Number of BO Iterations
(a)
UCB
Random
FSBO (EI)
MPHD Non-NN HGP (EI)
Ground-truth GP (EI)
Base GP (EI)
dland-specified HGP (EI)
MPHD Standard (EI)
MPHD Non-NN HGP (NToT) (ED)
BO (EI)
Non-infor con-informative HCP (EI)
Ground-truth HGP (EI)
MPHD Standard (NToT) (EI)
ABLR (EI)
Default Setting
NToT Setting
3x -1
30-0-1
Average Normalized Simple Rogret
Average Normalized Simple Regret
2 10-1
2
10-1
6x
20
40
60
80
100
20
40
60
80
100
Number of BO Iterations
Number BO Iterations
(b) EI
Figure 12: Average normalized simple regret of each method during BO for the Synthetic Super-datasel (L) in the two settings. The averages were taken over random seeds, and the highlighted areas show mean + std for cach method. Results with the UCB and EI acquisition functions for each method are shown in respective rows. The figure on the left of each row shows the results in the default setting. The figure on the right of each shows the results in the NToT setting and only methods that do not require pre-training the test. search space are included.
26

---

Published in Transactions on Machine Learning Research (02/2024)
Random
ABLR (UCB)
Non-informmative HGP (UCB)
MPHD Non-NN HGP (NT6T) (UCB)
Base GP (UCB)
FSBO (UCB)
MPHD Non HGP (UCB)
MPHD Standard (NToT) (UCB)
HyperBO (UCB)
Hand specified HGP (UCB)
MPHD Standard (UCB)
Default Setting
NIoT Setting
Average Normalized Simple Regret
Average Normalized Simple Regret
10-1
10-1.
40
80
100
20
60 Number BO Iterations
100
Number of BO Iterations
(a) UCB
Random
ABLR (EI)
Non-Informative HGP (EI)
MPHD Non-NN HGP (NToT) (EI)
Base GP (ED)
FSBO (EI)
MPHD Non-NN HGP (EI)
MPHD Standard (NTvT) (EI)
HyperBO (EI)
Hand-specifled HGP (EI)
MPHD Standard (EI)
Default Setting
NToT Setting
Average Normalized Simple Regret
Average Normalized Simple Regret
10-1
10-1_
20
40
60
80
100
20
40
60
80
100
Number Iterations
Number BO Iterations
(b) EI
Figure 13: Average normalized simple regret of each method during BO for the HPO-B Super-dataset in the two settings. The averages were taken over 5 random seeds, and the highlighted areas show mean + std for each method. Results with the UCB and El acquisition functions for each method are shown here. The figure on the left of each row shows the results in the default setting. The figure on the right of each row shows the results in the NToT setting and only methods that do not require pre-training on the test search space included.
27

---

Published in Transactions on Machine Learning Research (02/2024)
Random
MPHD Non-NN HCP (NToT) (UCB)
Base GP (UCB)
ABLR (UCB)
Hand cand-speriffed HGP (UCB)
MPHD Standard (NTTT) (UCB)
HyperBO (UCB)
FSBO (UCB)
non-nformative HGP (UCB)
NIoT Setting
Including Homogeneous Meta-BO Methods
Normalizod Simple Regret
10-1
Average Normalized Simple Regret
10
10-2
10
20
40
60
80
100
20
40
80
100
Number of BO Iterations
Number BO Iterations
(a) UCB
Random
MPHD Non-NN HGP (NToT) (EI)
Base GP (EI)
ABLR (EI)
Hand-speciine specified HCP (EI)
MPHD Standard (NToT) (EI)
HyperBO (EI)
FSBO (EI)
Non-nnfomativve HGP (EI)
NToT Setting
Including Homogeneous Meta-BO Methods
Average Normalized Simple Regret
Normalizzed Simple Regret
10-1
10-1
10
10-2
20
40
60
80
100
20
40
60
80
100
Number of BO Iterations
Number BO Iterations
(b) EI
Figure 14: Average normalized simple regret of each method during BO for the PD1 Dataset. The averages were taken over random seeds, and the highlighted areas show mean + std for each method. Results with the UCB and El acquisition functions for each method are shown here. The figure on the left of each row shows the results in the NToT setting. The figure on the right of each row includes homogeneous meta-BO methods in addition methods that do not require pre-traiiee ve--rainiig on the search space.
28

---

Published in Transactions on Machine Learning Research (02/2024)
PD1. Fig. 14 (left) shows the BO results of methods valid in the NToT setting PD1 Dataset with acquisition functions UCB and EI. Here the MPHD models were pre-trained on training datasets in HPO-B Super-dataset, but were not pre-trained on PDI. With either UCB or El as the acquisition function, MPHD Standard (NToT) achieved the best performance in the NToT setting. Fig. 14 (right) also includes the BO performances of the homogenous meta BO methods on PD1 Dataset. With UCB as the acquisition function, MPHD Standard (NToT) outperformed all methods including single-search-spaace baselines HyperBO, ABLR, and FSBO, With the El acquisition function, HyperBO and FSBO outperformed MPHD Standard (NToT), which is not surprising as HyperBO and FSBO were pre-trained on training sub-datasets in PD1 while MPHD Standard (NToT) was not.
In sum, we can see that with any of the three acquisition functions PI, UCB, and EI, MPHD generally outperforms the baselines (except the Ground-truth GP and Ground-truth HGP) in most cases across the two super-datasets and PD1, and also across the default setting and the NToT setting. This further demonstrates the effectiveness of the pro-training of MPHD and its ability generalize to new test functions in both seen and unseen search spaces.
29


Reviews:
[{"review_id": "i6d5GVzaFi", "reviewer": "Reviewer_Y6Xy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The approach is novel and very well motivated. In fact, it is in practice quite common that previous search spaces are not the same as future search spaces. How to handle this case efficiently is an open research question and the authors provide an interesting step toward solving this problem.\n1. The approach is very well mathematically motivated and presented.\n1. The benchmarks are extensive and thorough and provide a clear picture of the strength and weaknesses of MPHD and other baselines.\n1. Related work is well covered.\n\nWeaknesses:\n1. First of all, I got the impression that the authors tried to squeeze the paper into the 12 pages for TMLR. Unfortunately, many very important explanations and details can only be found in the appendix s.t. I had to go forth and back all the time. This makes it very hard to read the paper. Actually, I would strongly prefer to have 16+ page paper here without needing to jump to the appendix again and again. \n1. Similarly, I had to jump to the figures forth and back. At the very beginning, the authors refer to Figure 6 which is at the end, and at the end of the paper they refer to Figure 1. Also very annoying.\n1. Also the conclusion/discussion at the end is very brief and misses to fully discuss the strengths and weaknesses of the approach and future directions.\n1. (to continue with readability): The plots use too many colors without any marking (other than solid and dashed lines). It makes it very hard for me to distinguish the lines and match them with the baselines. People with color blindness would have an even harder time.\n1. Very late, I fully understand what the “context” actually is. On Page 9, they say that they use “whether the domain is continuous or discrete, the number of continuous dimensions and the number of discrete dimensions in the domain”. On the one hand, I can understand that there is not much known about a new black-box function; on the other hand, I wonder how much can be learned about this little information. In fact, I wonder whether the entire approach is only working fairly decently because of the hierarchical GP that needs to be adjusted according to these context features. \n1. I missed a lot of details on the training of the deep neural networks used for the pre-training. \n1. How does the approach compare against a simple hyperparameter tuning of BO’s own hyperparameters on previous functions (e.g., as done by https://ieeexplore.ieee.org/document/9913342) and transferring that to similar functions? \n1. Most of the above can be fairly easily improved, but the results are my main concern. Only on the artificial data (which is well designed for the use case), MPHD has a very convincing performance. On the other two HPO benchmarks, MPHD is either only marginally better (maybe not statistically significant) or even worse. Furthermore:\n    * In the results, which of the results is a standard BO-GP approach as one would implement it typically (e.g., the old but well-known Spearmint)? \n    * Why are baselines missing on PD1?\n    * I was surprised that the authors highlighted the results with PI since it is a rather uncommon choice. Also checking the results of Wang and Jegelka 2017, PI is not superior in their results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Bayesian Optimization is used a lot to optimize black-box functions, e.g., in Hyperparameter Optimization. However, it is not always easy to specify the kernel of a GP since there is a priori only very little knowledge about the black-box function at hand. The authors propose MPHD that meta-learns based on domain-specific contexts how to specify the parameters of a hierarchical GP and thus improve the optimization performance. The big advantage compared to previous approaches is that the search space does not need to be the same on previous functions. They demonstrate the strength of MPHD on artificial functions (with known optimal behavior) and on well-established HPO benchmark (HPO-B and PD1).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper needs a complete restructuring s.t. it is easy to read and understand. \n\nFurthermore, the results are not convincing in more than one dimension and need improvements (see above).", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "O7n0FsZdWK", "reviewer": "Reviewer_MKMS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n+ The approach of using previous datasets that include the optimization data of related optimization problems to aid the optimization of a new problem is important and is worth to investigate.\n+ The paper’s idea of using these prior datasets to pre-train a probabilistic model so that for any new domain, the model can generate the prior distributions over the GP parameters to construct a domain-specific hierarchical GP, is interesting and seems reasonable.\n\n+ The paper derives theoretical analysis to show that under some assumptions the proposed method can find the true GP hyperparameters when the number of sub-datasets converges to infinity (although I have various questions regarding the theoretical analysis).\n\n+ The paper conducts experiments on both synthetic and real-world datasets and shows that the performance of the proposed method is better than existing baselines\n\n\n# Weaknesses:\n+ The concept of “context” (S) is very important and is used throughout the paper but it is never explained or defined in detail. There are examples of the context in Figure 2 but it’s hard to catch the definition of this concept from these examples. I’m really confused what exactly “context” means here.\n\n+ The paper uses a lot of assumptions in order to derive the theorems presented in the paper, however, the implications of these assumptions are not described in detail. Are these common assumptions used in related research? Do these assumptions occur in practice? Some assumptions are really unclear me, for example, in the first paragraph in Section 3.1., it assumes that the contexts s_{ih} are the same across domain X^{(i)}, but there is definition of context, so I don’t really understand what this assumption means in practice. Furthermore, there are various assumptions used in the proof (in the appendix – Section C) but these are not mentioned in the paper – they should be explicitly mentioned in the paper. \n\n+ The proofs of Theorems 2 & 3 are very vague. There are only 2-3 lines for the proof of each theorem. Even when we use some theoretical results from other papers, we still need to write again those results, and explain clearly with rigorous mathematical formulas on how we can apply those results here. \n\n+ In the experiments, in the first sentence of Page 9, it says that for the two real-world datasets, \nevery domain dimension is normalized between, will this make all the search spaces of all the datasets to be the same? If yes, then these datasets might not be appropriate to be used to evaluate the proposed method as the method claims to work for different search spaces?\n+ The colors of the plots make it quite difficult to distinguish between the methods. Besides, in Figure 7 (left plot – default setting), it seems the proposed method does not perform well?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper develops a new transfer learning method, namely MPHD (model pre-training on heterogeneous domains) for Bayesian optimization on heterogeneous search space. The main idea is to pre-train a model that map from domain-specific context to the specifications of hierarchical GPs. The paper derives theoretical analysis to show that under some assumptions, the GP estimates (length scales) converge to the true value. Experimental results are conducted on various synthetic and real-world datasets to evaluate the effectiveness of the proposed method.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "+ [Major] The concept of “context” needs to be formulated rigorously and described in detail.\n+ [Major] The assumptions used in all the lemmas and theorems in the paper need to be elaborated more to see if they’re common assumptions and whether they could occur in practice. All the assumptions need to be listed in the main paper.\n+ [Major] The proofs of Theorems 1 and 2 need to be rigorously proven.\n+ [Minor] The colours of the plots could be modified so as to help readers to see the performance of the proposed method and the baselines easier.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YAxTZOSCHb", "reviewer": "Reviewer_YACK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed method makes intuitive sense and is a natural method to handle the problem of heterogeneous domains in transfer learning for Bayesian optimization.\n- Section 2 and Figure 1 gave nice and intuitive introduction to the proposed problem formulation and algorithm.\n- I find the connection between the analyzed setting with increasing number of sub-datasets and the increasing domain setting (formalized by Lemma 1) very interesting.\n- The experiments in Section 3.2 provide nice illustrations with regard to the behavior of the proposed method, which match well with expectations.\n- The experimental results look good, the proposed methods not only work well in the novel setting of heterogeneous domains but are also competitive in the conventional problem setting with homogeneous domains.\n\nWeaknesses:\n- I think the paper lacks explanation on how to design the contexts for different domains. The contexts are important for the proposed method because if I understand correctly, they contain information about how the heterogeneous domains are related to each other, and hence they can be seen as the reason why information transfer across domains is possible. So, the design of the contexts is important for the paper and hence should be elaborated on.\n- First paragraph of Section 3.1, it says here that the contexts for different domains are assumed to be the same in the theoretical analysis, does it mean that the theoretical analysis does not account for domain heterogeneity? If this is the case, then the significance of the theoretical contributions may be restricted because being able to handle heterogeneous domains is the main focus and novelty of this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method to perform transfer learning for Bayesian optimization when the search spaces of different BO tasks are potentially heterogeneous. The proposed method uses a neural network to learn the mapping from the contexts of different domains to the corresponding hyperparameters for the prior distribution from which the GP hyperparameters for this domain are sampled from. The proposed method is theoretically shown to be asymptotically consistent (i.e., the method is guaranteed to be able to find the true hyperparameters as the available datasets increase), and is empirically shown to perform competitively in experiments with both homogeneous and heterogeneous domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In addition to the few weaknesses above, I have a couple of questions below:\n- About the assumption of Asymptotic Identifiability, I wonder what's the intuition behind this assumption and how realistic is it? What's the requirement on the input locations in the sub-datasets in order to satisfy this assumption?\n- The neural network $\\phi$ outputs the hyperparameters $\\alpha_{i,h}$ of the prior distribution, but different domains may have different number of such hyperparameters $\\alpha_{i,h}$, so I wonder how do you decide the number of outputs from this neural network $\\phi$?\n- Figure 9, right figure, it looks like you didn't compare with MPHD Standard in this experiment?", "broader_impact_concerns": "No ethical concerns that I can think of.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers value the studied problem and the contributions made, as well as the empirical results. The majority agrees that the paper would be suitable for publication at TMLR.

However, all reviewers flagged in their reviews that the notion of context was not particularly clear. While the majority of the reviewers agree that the revision has alleviated that issue, and resolved many other, they do remain sceptical about the amount of information in the context features used in the simulations.

   In the minor revision, hence please
   
   1. add good intuitive arguments to the paper on why and how the simple context information provides the information needed for the method to work as intended.

   When reading the paper, I was confused by the following two additional points:

   2. Fig 1, right column (test domain T): why does the green box indicate that the length-scales / noise variances of domain T are inputs for the learned model \hat{phi} alongside the context? My understanding from Section 2.2 and Eq 6 is that the model phi only takes the context as input, and that parameters such as the length scales are thought to be samples from the prior with hyperparams phi(context), with their MAP estimates being used for BO on a test function (eq 6). Please clarify or amend Fig 1.

   3. Sec 3.2 (synthetic super-dataset L): the hyperparameters are said to depend linearly on the dimensions of each domain. Please brielfy explain the motivation for this, e.g. what functional properties are you modelling?
Decision: Accept (poster)