Efficient Bayesian Optimization with Deep Kernel Learning and Transformer Pre-trained on Muliple Heterogeneous Datasets
Authors: Wenlong Lyu
Shoubo Hu
Jie Chuai
Zhitang Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Pre-training
Bayesian optimization
Transformer
Transfer learning

Abstract: Bayesian optimization (BO) is widely adopted in black-box optimization problems and it relies on a surrogate model to approximate the black-box response function. With the increasing number of black-box optimization tasks solved and even more to solve, the ability to learn from multiple prior tasks to jointly pre-train a surrogate model is long-awaited to further boost optimization efficiency. In this paper, we propose a simple approach to pre-train a surrogate, which is a Gaussian process (GP) with a kernel defined on deep features learned from a Transformer-based encoder, using datasets from prior tasks with possibly heterogeneous input spaces. In addition, we provide a simple yet effective mix-up initialization strategy for input tokens corresponding to unseen input variables and therefore accelerate new tasks' convergence. Experiments on both synthetic and real benchmark problems demonstrate the effectiveness of our proposed pre-training and transfer BO strategy over existing methods.

Under review as a conference paper at ICLR 2023
EFFICIENT BAYESIAN OPTIMIZATION WITH DEEP KERNEL LEARNING AND TRANSFORMER PRE- TRAINED ON MULIPLE HETEROGENEOUS DATASETS
Anonymous authors Paper under double-bling review
ABSTRACT
Bayesian optimization (BO) is widely adopted in black-box optimization prob- lems and it relies on surrogate model to approximate the black-bo response function. With the increasing number of black-box optimization tasks solved and even more to solve, the ability to learn from multiple prior tasks to jointly pre-train a surrogate model is long-awaites to further boost optimization efficiency. In this paper, we propose a simple approach to pre-train a surrogate, which is a Gaussian process (GP) with kernel defined deep features learned from Transformer- based encoder, using datasets from prior tasks with possibly heterogeneous input spaces. In addition, provide simple yet effective mix-up initialization strategy for input tokens corresponding to unseen input variables and therefore accelerate new tasks' convergence. Experiments on both synthetic and real benchmark prob- lems demonstrate the effectiveness of our proposed pre-training and transfer BO strategy over existing methods.
INTRODUCTION
In black-box optimization problems, one could only observe outputs of the function being optimized based on some given imputs, and can hardly access the explicit form of the function. These kinds of optimization problems are ubiquitous in practice (e.g., (Mahapatra et al., 2015] Korovina et al.) 2020 Griffths & Lobato, 2020).. Among black-box optimization problems, some are particularly challenging since their function evaluations are expensive, in sense that the evaluation either takes substantial amount of time or requires considerable monetary cost. To this end, Bayesian Optimization (BO; Shahriari et al.] (2016)) was proposed as sample-efficient and derivative-free solution for finding optimal input value of black-box functions.
BO algorithms are typically equipped with two core components: a surrogate and an acquisition function. The surrogate is model the objective function from historical interactions, and the acquisition function measures the utility of gathering new input points by trading off exploration and exploitation. Traditional BO algorithms adopt Gaussian process (GP: Rasmussen & Williams )2009)) as the surrogates, and different tasks are usually optimized respectively in a cold-start man- ner. In recent years, model pre-training showed significant improvements in both convergence speed and prediction accuracy (Szegedy et al.) 2016 Devlin et al. 2019), pre-training surrogate(s) BO becomes a promising research direction to boost optimization efficiency.
Most existing work on surrogate mre-training (Bardenet et al. 2013 Swersky et al. 2013 Yogatama
& Mann 2014, Springenberg et al. 2016 Wistuba et al.) [017] Perrone al., 2018 Feurer al.,
2018al Sa. Wistuba & Grabocka [202]) assumes that the target task shares the same input search space with prior tasks generating historical datasets. this assumption violated, the pre-trained surro- gate cannot be directly applied and one has conduct cold-start BO. Such assumption largely restricts the scope of application of a pre-trained surrogate, and also prevents from learning useful information by training based method was proposed recently. It formulates the optimization task as a sequence modeling problem and pre-trains single surrogate using various optimization trajectories (Chen al. 2022).
large number of similar datasets. To overcome these limitations, text-
In this work, we focus on surrogate pre- trainiiinng that transfers knowledge from prior tasks to new ones with possibly different input search spaces, for further improving the optimization efficiency

---

Under review as a conference paper at ICLR 2023
of BO. We adopt combination of Transformer (Vaswani et al. 2017 and deep kernel Gaussian process (Wilson et al. 2016b) for the surrogate, which enables joint training on prior datasets with variable input dimensions. For target task, only the feature tokenizer of the pre-trained model needs to be modularized and reconstructed according to its input space. Other modules of the pre- trained model remain unchanged when applied to tasks, which allows the new task make the most of prior knowledge. Our contributions can be summarized as follows:
To the best of our knowledge, this is the first transfer BO method that is able to jointly pre-train on tabular data from tasks with heterogeneous input spaces.
We provide simple yet effective strategy of transferring the pre-trained model to new tasks with previously unseen input variables to improve optimization efficiency.
Our transfer BO method shows clear advantage on both synthetic and real problems from different domains, and also achieves the new state-of-the-ar results on the HPO-B (Pineda-Arange et al. 2021) public datasets.
2
BACKGROUND
Gaussian process A Gaussian process collection of random variables, any finite number of which have joint Gaussian distribution (Rasmussen & Williams) 2009). Formally, a GP is rep- resented as f(x GP(m(2x) k( x')). where m(x) and k(x,, x denotes mean and covariance function, respectively. Given dataset
{(x(1) y))) ) with n examples, any collection of function values has a joint Gaussian distribution f = [ f(x(())] NNu,, Kx,x), where the mean vector μ4; = m(x(()), [Kx.]e = (1) 20)) nice property of GP that its distributions of various derived quantities can be obtained explicitly. Specifically, under the additive Gaussian noise assumption, the predictive distribution of the GP evaluated at a new test example a+**) can be derived as
((f())/a)()) D) ((E(f()))) cov(((()))
where Elf m(x(+)) + xx)_x Kxxx foxx cov(f(*)) k(x('),, (())) ) K ( xx x a² - xx,'')); K ai)) denotes the vector of covariances between the test example ai()) and the n training examples, and is the vector consisting of all response values.
Bayesian Optimization Bayesian optimization (Shahriari et al. 2016) uses a probabilistic sur- rogate model for data-efficient black-box optimization. It is suited for expensive black-box opti- mization, where objective evaluation can be consumonn or of high cost. Given the previously gathered dataset D, BO uses surrogate models like GP to fit the dataset. For new input (*) the sur- rogate model gives predictive distribution in equation | then acquisition function is constructed with both the prediction and uncertainty information to balance exploitation and exploration. The acquisition is optimized by third-party optimizer like evolutionary algorithm to generate BO recom- mendation. Throughout this paper, we use the lower confidence bound (LCB) Srinivas et al. (2009) as the acquisition function. LCB(x) m(x) - x a (x), where o (x) denotes the standard deviation and x (set to 3 in experiments) is a constant for tuning the exploitation and exploration trade-off.
Transsssfrrrrr (Gorishniy et al. 2021) is a recently proposed attention- model for tabular data modeling. It consists of Feature- Tokkeize layer, multiple Transformer lay- ers, and a prediction layer. The Feature-Tokenizer layer enables its ability of handling tabular data. For d numerical input features x = -.i...., xa]] the dependent embedding table W E Rd and a oolumn-dependent embedding table B E Rdxds where d, is the dimension of embedding vector. During forward-par pass of the Tokuunree-T layer, the i-th feature x; would be transformed to x; x W. + b,, where w, and b, are the i-th row in W and B. In this way, an x d matrix is transformed into a x d x de tensor. Then, [CLSS] token embedding is appended to the tensor and the tensor is passed to the stacked transformer layers to extract output embedding vectors. The output embedding vector corresponding to the [ [CLS] token is used as the output representation. The output representation is then passed into the prediction layer for final model prediction. The tokenization process for categorical data is implemented by a look-up table, in which each categorical variable corresponds to a b, and each unique value of a variable corresponds to a wt;
TT-Trrasformer
layer initializes a value-
2

---

Under review as a conference paper at ICLR 2023
We use FT-Transformer the backbone of method. Throughout this paper. we only consider numerical features, however as FT-Transformer can also handle categorical features, our method can be easily extended to mixed search space with both numerical and categorical features.
3 METHODOLOGY
3.1
PROBLEM SETTING
Given target function fo (x) Rdr + where dr the dimension of fr. we would like apply Bayesian optimization to find its minimizer:
x. argmin
Assume we have N source dataset . D where D ". X E RNxidd and vs E RN, we want to pre-train a surrogate model on all these historical datasets to accelerate the BO convergence on target task.
If dj = d2 ... dy = dr, and the feature names are aligned, pre-training methods like (Wistubal & Grabockal 2021 Wistuba al., 2016 Feurer al. (0188) can be applied. However, if either the dimension or parameter names are unaligned, pre-training and fine-tuning become non-trivial.
Taking hyper-pan pppee-paraetter optimization (HPO) for AutoML as an example. Suppose we have the following two historical HPO records:
HPO for Random forest dataset A, where max.features, max.depth tuned,
HPO for LightGBM on dataset B, where learning_rate and reg_al reg.alpha are tuned,
now we want to perform HPO for XGBoost model on new dataset C, the hyper-parae syper-parameters to be tuned are learning innate,,,,,,, ma max_depth and col_ wesssplleby_lle.. two source datasets pre-train the surrogate model for BO.
and we want to use the
3.2 MULTI-SOURCE DEEP KERNEL LEARNING WITH FT- FI-TRANSFORMER
The basic idea of our method to use FT-Transformer (Gorishniy et al. 2021) as the feature extrac- tor of deep kernel Gaussian processes (FT-DKL) and pre-train the FT-DKL similar source tasks. Given that transformer can handle variable-length input, we can use the FT-DKL to jointly pre-train on multiple heterogeneous datasets with unaligned parameter spaces, making the FT-Transformer multi-source FT- PT-Trrasformmr.
The first step of algorithm is data normalization. When there multiple unaligned source datasets, we independently normalize the objective of each source dataset. As for the normalization of input features, if two source datasets share common features, the shared common features merged and jointly normalized.
After the source datasets are normalized, we initialize a multi-source FT-Transformer, where the embedding table in the Feature-tokenizer layer corresponds to the union of features of all source datasets. Following the common procedures of training DKL models, we firstly pre-train the multi- source FT-Transe TT-arssffrrmer with linear output layer and MSE loss, and then replace the linear output layer with a sparse variational Gaussian process (SVGP) layer and pre-train the model with ELBO loss, in this stage, the weight of FT-Transformers, the GP hyper-parameters and variational parame- ters are jointly updated. Details about SVGP and sparse varational deep kernel learning can be seen in (Hensman et al., 2015; Wilson et al., 2016a).
After pre-training, we can transfer the model to downstream target optimization task, where the transformer layers, sparse Gaussian process layer and the [CLS] embedding of Feature-Tokenizer layer are copied to initialize the target FT-DKL model.
For the target task parameters already
in source tasks, the corresponding source embedding vectors are also copied to the target FT-DKL model; for the unseen target task parameters, we use mix-up initialization: randomly select two embedding vectors and e2 from source embeddings and a random number a ~ 2(00 1), the new embedding vector is initialized as
e +
a)) x e2
3

---

Under review as a conference paper at ICLR 2023
After the target FT-DKL is initialized, directly fine-tune the target FT-DKL model with target data using ELBO loss at the start each BО iteration. The fine-tuned FT-DKL model can be used regular Gaussian process to construct acquisition functions and give recommendations. We summarize our algorithm in Algorithm [ demonstrate the proposed model architecture.
In Section A.1 we provide an illustrative example to
Algorithm/ Pre-training and the-tuning of multi-source FT-DKL
Require: source datasets { D
Jointly normalize the source datasets
Construct multi-source FT-Transformer with the union of all source features for the Feature- Tokenizer layer
Joint training of multi-source datasets with MSE loss Joint training of multi-source with GP layer and ELBO loss
while target task not finished do
Transfer the source FT-DKL to target FT-DKL with embedding mix-up Fine-tune the target FT-KL with ELBO loss Use the predictive distribution of FT-DKL to construct acquisition function Optimize the acquision function for BO recommendation
end while
RELATED WORK
Warm-starting a new BO task by knowledge transfer from prior tasks can significantly boost opti- mization efficiency. One of the most common transfer approaches is to pre-train surrogate model on the entire prior data. Earlier methods considered learning a joint GP-based surrogate model on combined prior data (Bardenet et al. 2013 Swersky et al. 2013 Yogatama & Mann 2014), which may easily suffer from high computational complexity when applied to large data sets. As a result, some later works tried to alleviate this issue by using multi-task learning (Perrone et al. 2018) or ensembles of GP, where a GP was learned for each task (Wistuba et al. Feurer et al.) [20 8a). Neural network models have also been adopted either learn task embedding for each task side Bayesian neural network (Springenberg et al. 2016), or pre-train deep kernel model and fine-tune on target tasks (Wistuba & Grabockal 2021).
Besides the surrogate model, learning new acquisition functions is another valid approach for knowl- edge transfer (Volpp et al., 2020). However, all of these existing works considered a fixed search space. In other words, new task with different search space would fail in directly borrowing strength from prior data sets. Most recently, a text-based pre-trained model called OptFormer (Chen) al. 2022) was introduced to address this issue by adopting Transformer model as the surrogate. Although both OptFormer and our work are able to pre-train using heterogeneous datasets, our work directly conducts learning on tabular data whereas OptFormer requires an additional tokenization step for the optimization trajectories.
Transformer (Vaswani et al. 2017) was initially proposed for machine translation. It was later adopted, beyond natural language processing, in computer vision (Dosovitskiy et al. 2021 Parmar al., 2018), reinforcement learning (Chen al., 2021 ZZheng al.] 2022), etc. Because of its success in various fields, Transformer was also restructured for tabular data (Huang et al. 2020) Gorishniy al. 202T> another strong (neural network) baseline besides multi-layer perceptron.
5 EXPERIMENTS
In this section, we demonstrate the high sample efficiency of our pre-trained FT-DKL with three experiments, including high-dimensional synthetic function, HPO on the HPO-B (Pineda-Arango jet al. 2021) benchmark, and a real-world wireless network optimization (WNO) problem. For the synthetic function and wireless network optimization problem, transferred knowledge from a single www-dimensional source problem to a high-dimensional target task, while for the HPO-B benchmarks, 758 source tasks with dimensions ranging from 2 to 18 were merged and pre-trained,
4

---

Under review as a conference paper at ICLR 2023
the pre-trained model was fine-tuned for 86 different target HPO problems. The statistics of these experiments can be seen in Table I
Table I: Statistics of experimental datasets
Task
Source tasks
Source dimension
Source evaluations
Target tasks
Target dimension
Synthetic
20
800
30
HPO-B
758
2-18
3,279,050
86
2-18
WNO
48
1800
69
All experiments were conducted server with eight-core Intel (R). CPU and one Quadro RTX 5000 GPU.
Xeon (R) Gold 6134
5.1 SCALED AND SHIFTED HIGH DIMENSIONAL ACKLEY FUNCTION OPTIMIZATION
In this section, we use the Ackley function with input scaling and offset demonstration. Ackley function is popular benchmark function for black-box optimization algorithms, To make the func- tion harder to optimize and transfer, we introduced random offset and scaling to each dimension. To do that, firstly scaled the original search space 1-1, where D was the input dimension; then we modified the original Ackley function as shown Eq 2 for the target optimization task.
In our experiment, we set = 30
fD(
(s) x (œ1 (z1--1).... -.)....77 3D (ID - (ID-OD))
8;
u(0.01, 2). i= = 1,2....,
~ u(-0.8, 0.8), = 1,2,
Firstly, we compared GP-UCB on the original Ackley function and the modified Ackley function. As shown in Figure la we that although GP-UCB able to reach near-optimal solution for the original Ackley function, it failed to optimize the Ackley function with input scaling and offset.
We then compared different surrogate models the modified Ackley function. For pre-trained FT-DKL, we used the same modified Ackley function but with D = 20 as the source task, leaving 10 target task parameters unseen in the source task. We ran evolutionary algorithm on the 20- dimensional source task, and sampled 800 data points from the optimization trajectory to pre-train the FT-DKL model.
The following models were compared. For all surrogate models, lower confidence bound (LCB) was used the acquisition function. We randomly sampled points initialize BO, except for GP-50, where 50 initial random points were used. For each model, BO was repeated five times to average random fluctuations.
GP. where Gaussian process with Matern3/2 kernel was used for BO,
FT-DKL, where the FT-DKL model used the surrogate model, without any pre-training and model transfer,
GP-50, Gaussian process as surrogate model, but with 50 points as random initialization, Tretraine-FFT-DKC FT-DKL pre-trained on 20-D data used as surrogate model.
As shown in Figure we can see that with only five points as random initialization, both GP and FT-DKL. failed optimize the offleyOffsseScalee function. With 50 points random initialization, GP was able to find some low-regret solutions, but the result was far from optimal. On the other hand, when were able to pre-train the FT-DKL on the 20-D source function and fine-tune the model on the 30-D target function, the pre- trained FT-DKL outperformed other models significantly.
In this paper, only consider how model performance is improved by multi-source pre-training. However, from a practical point of view. a more effective approach for transferring from a 20-D function to 30-D function would be directly copy the optimal values of the 20 dimensions, and only optimize the rest 10 dimensions. In Appendix A.3] we show that our proposed approach still outperformed Gaussian processes and FT-DKL. when fixing the first 20 dimensions.

---

Under review as a conference paper at ICLR 2023
225
275
354
352
338
75-
35.
25-
(b)
Figure I: (a) GP-UCB on Ackley and the Ackley with scale and offset, it can bee seen that the Ackley with input scaling and offset is much harder to optimize; (b) Comparison of different algorithms on the Ackley with scale and offset.
5.2 HYPPERPARAMETER OPTIMIZATION USING HPO-B BENCHMARK
In this section, we demonstrate our method on HPO problems with the HPO-B benchmark (Pineda- Arango et al. 2021). HPO-B is the largest public benchmark for HPO, containing HPO-B tasks, with 176 different search spaces. We used the HPO-B-v3 version our experi- ment, where tasks related the most frequent 16 search spaces were extracted and splitted into meta-train dataset, meta vatta-vaaidat ion daaddaaasseeeee and meta meta-test-tot -dataset. Af- ter the rrain-validation-ee splitting, there were 758 HPO tasks with 3,279,050 hyper- nyper-parameter configuration evaluations with 16 search spaces in the meta-train-dataser and 86 tasks in meta-teet test dataset.
than 1900
We jointly pre-trained the FT-DKI. model on all the 758 meta-train tasks and then transferred the model to the target task during the optimization of meta-test tasks. We didn't 't use the meta vali- dation dataset in this experiment. We pre-trained the model for 300 epochs with MSE loss and 50 epochs with ELBO loss. Unlike FSBO (Wistuba & Grabockal 2021) where a distinct model was pre-trained for each search space, we used one common model with shared transformer layers for all the 758 source tasks. During pre-training the meta-features like space-ID and dataset-ID were treated categorical features augment the dataset.
We followed the evaluation protocol HPO-B, where five different sets initialization points provided by the benchmark for repeated experiments. We compared algorithm against the results provided by the HPO-B benchmark, the following transfer and non-transfer BO methods were compared:
Random: Random search
GP: Gaussian processes
DNGO (Snoek et al.
2015):
Bayesian linear regression with feature extracted by neural networks,
DGP (Wilson et al.
2016b
Gaussian process with deep kernel,
BOHAMIANN (Springenberg et al. 2016): BO with BNN surrogate, trained with adaptive
SGDHMC
TST WWitubba et al. 2016), RGPE (Feurer al. 2018b) TAF (Wistuba al. 2017): Transfer
learning by weighted combination of Gaussian processes trained on source tasks with same design spaces,
FSBO (Wistuba & Grabockal 2021): pre-trained on source tasks that share the same design space with target task.
Few-shot Bayesian optimization where deep kernel GP was
The result of normalized regret and average rank is shown in Figure and Table As can seen, are-trained model outperformed all other reported results by large margin, both with regard to average rank and average regret, throughout all the iterations, the immediate regret remained 34%- 65% of the second best algorithm FSBO. Among all algorithms, method was the only that

---

Under review as a conference paper at ICLR 2023
Rank
ECHNMANS
000
10-1
Average Regred
Average
BOHGMAN
DNCO
TST
10-2
Renkg
15 30 45 60 75 105 Number of trials
15 30 45 60 90 105
Koshis
Pacaared FTDC
Number
trials
DONKMANN
DGP
BOHAMAANN
Figure 2: Comparisons of normalized regret and average ranks across all search spaces.
Table 2: Normalized regret comparison at different iterations
Algorithms
Regret@1 1
Regret@5 5
Regret@@ 15
Regret@ 30
Regret 50
Regret@ 100
Random
0.189
0.142
0.102
0.082
0.072
0.0540
GP
0.181
0.106
0.064
0,044
0.035
0.0258
IST
0.164
0.104
0.064
0.047
0.039
0,0278
DNGO
0.165
0.099
0.057
0.041
0.028
0.0224
TAF
0.164
0.101
0.060
0.043
0.036
0.0254
DGP
0.183
0.099
0.061
0.041
0.028
0.0173
BOHAMIANN
0.173
0.118
0.061
0.041
0.028
0.0158
RGPE
0.140
0.104
0.064
0.053
0.046
0.0278
FSBO
0.152
0.087
0.049
0.032
0.021
0.0105
Pretrained-FT-DKLL
0.066
0.030
0.017
0.011
0.009
0.0065
achieved regret less than 0.1 after only one iteration and less than 0.01 after 100 iterations. The per-space comparison of normalized regret and average rank is put in Appendix A.4 where pre-trained FT-DKL showed the best performance on most of the design spaces.
Does big model generalize better? We have witnessed lot of news reporting large language models showing better zero/few-shot performance, then one interesting question to ask is how trans- former model size affects the sample efficiency of pre-trained model. To answer that question, we increased the model size to the level of BERT (Devlin et al. 2019)) with 768-dimensional em- bedding, 12-heads attention, and 12 transformer layers, the model now has more than 30 million trainable parameters.
Given only GPU to use, the large model is very slow train. took about 50 minutes train for one epoch, so we didn't t follow the standard BO procedure as in previous sections. Firstly, GP was used, we only pre-trained the model with MSE loss for 250 epochs, the model with linear output layer was directly used as the surrogate model; secondly, there's 's no fine-tuning during target task optimization, instead, we only used the pre-trained multi-source PT-Transformer for zero-shot prediction on the meta-test dataset. We sorted the zero-shot prediction result and the hyper- parameter configurations with the top 100 predictions was recommended one by one for 100 target task iterations. With this batched zero-shot optimization, the target task can be done much faster than BO with smaller FT-DKL.
We compared the batched zero-shot optimization of previously reported pre-trained FT-DKL and FSBO, and we also performed batched zero-shot optimization for the smaller FT-Transformer used for FT-DKL. The result of average rank and regret can be in Figure
As can be seen in Figure D. with equal-sized FT-Transformer, running BO with pre-trained FD- DKL showed better performance than only running batched zero-shot optimization; however, with a much larger multi-source FT- TT-nrsssfrrm,r, the performance zero-shot optimization would be comparable the result of BO.

---

Under review as a conference paper at ICLR 2023
3,00
Average Regret
10-4
Ramigg25
2.75
beth pwo sie
Phinetred DNd
2.50
Raning
< 2.25
10
1.75
Rankgg 100
15 30
60 75 105 Number of trials
30 45 60 75 105 Number of trials
FS80
batch zero shot
Figure 3: Comparisons of normalized regret and average ranks across all search spaces.
5.3 WIRELESS NETWORK OPTIMIZATION
In this section, we evaluate our pre-training strategy in real-world applications. The task to op- timize the parameters of cellular network. A cellular network consists of many wireless cells and provides the infrastructure for modern mobile communications. Each wireless cell has many parameters (e.g., the antenna angle, the transmission power) that need be optimized to adapt to its surrounding wireless environment. By optimizing the parameters, the performance (e.g., data throughput) of the network can be improved provide better user experience.
Due to the heterogeneous wireless environment of a cellular network, different cells have different optimal parameter configurations. Moreover, tuning parameters of cell affects the performance of its neighbors due to the inter-cell interference. Therefore, parameters of different cells within network should be jointly optimized.
Source and Target Task. In our experiment, the network consisted of 23 cells, where 69 parameters need be optimized jointly to increase the overall network throughput. the source task, part of the network (i.e., 16 cells with 48 parameters) had been optimized historically, and we used the data from the source task to pre-train the surrogate model, and transferred the 69 69-parameter optimiza- tion task to boost the optimization efficiency. The experiments were conducted on an industrial-level simulator that imitated the behavior real cellular network.
Search Space. The search range of each parameter was an ordered set of integers with step size of 1. Among the 69 parameters, 46 parameters had a search range of size 41, and 23 parameters had a search range of size 26.
Surrogate Models. We compared the optimization performance the following strategy/surrogate models on the 69- parameter optimization task.
Random: A uniformly random sampling was performed within the 69 parameter search space.
Random Forest: A random forest regressor with 100 estimators was used as the surrogate model without pre-training, and initialized by 70 random samples.
Gaussian Process: A Gaussian process model with combined linear and Matern kernel was used withowt pre-training, and initialized by 70 random samples.
Pretrained-FT-DKLL. The proposed model was pre-trained with 1800 samples collected from previous 48- parameeet BO experiments. Among the 1800 samples, 1000 samples were obtained by uniformly random sampling from the 48-parameter search space, and 800 samples were the BO traces with RF and GP surrogate models and 400 samples The optimization on the target task was initialized by 5 random samples.
collected for each model.
Setup. The exploration budget of each run was 400 iterations, and runs were performed for each surrogate model. The mean and variance over different runs were recorded.
The performance of the surrogate models over different runs are presented in Figure and the numerical values reported in Table The regret is the negative of the 23 23-cell-network throughput and the value should be minimized as small as possible. is clear from the plot that our pre-training strategy boosted the optimization efficiency dramatically compared the other baselines. After
8

---

Under review as a conference paper at ICLR 2023
Regrat Iterationn
Random
-23.0
-335
Pretrained FT. DKL
Algorithms
Regret @ 80
Regret 400
24.0
Random
-23.853
-23.945
24.5
RF
-23.838
-24.466
GP
-23.840
-24.601
25.0
0
50
100
150
200
250
300
350
400
Pretrained-FT-DLL
-24.633
-24.867
Figure 4: Regret vs. Number of iterations
Figure Regret by iteration 80 and 400
only few initial random samples, the Pretrained-FF-DKIL. model quickly located its search region with low regret. The achieved performance of Pretrained-FT-DKKL. by 80 iterations already surpassed RF and GP's final performance with 400 iterations, which indicated that our strategy boosted the optimization efficiency by around five times.
6 LIMITATION AND DISCUSSION
Overall, we believe that our proposed method represents a new paradigm of transfer learning in BO, however, there are still several limitations of our work to overcome.
The main issue is about the training overhead of Transformers. Even with GPUs, transformer train- ing is still time-consuming compared to GP and traditional deep kernel GP. Although the excel- lent zero/few-sho performance alleviates the requirement for long-time BO iterations, we still need hardware-friendee Transformers and haansfommr-friendde hardware for more widely application of our method.
Secondly, as shown (Ober et al. 202T), it's even more easier to over-fit a deep kernel model than to witr-fit& traditional neural network. The over-fitting be addressed by introducing Lipschitz constraint in neural works (Liu et al. 2020). However it's still unclear how Lipschitz constraints can be efficiently introduced to Transformers.
Finally, we believe that the quality of the source data is very important. Currently, we use shared embedding vector for common features, and determine whether not two source datasets have common features by matching their parameter names, so it's possible that our system can be mis- guided if malicious uploads a dataset with the same parameter names but completely irrelevant or adversarial parameter values.
7 CONCLUSION
In this paper, we introduced simple deep kernel model with multi-source FT-Traasformer, The model can be used jointly pre-train multiple heterogeneous source datasets and can be efficiently fine-tuned for downstream target task with unaligned parameter space. We tested the proposed FT- DKL model on three synthetic and real-world benchmarks and found the model to be highly sample- efficient. We also found that bigger surrogate model that matched the size of BERT showed even better zero-shot optimization performance. We believe that our research paves the way toward a more unified surrogate model for Bayesian optimization.
REFERENCES
Rémi Bardenet, Mátyás Brendel, Balázs Kégl, and Michèle Sebag. Collaborative hyperparameter tuning. In ICML, 2013.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, P. Abbeel, A. Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. ArXiv, 0bs/2106.01345 2021.

---

Under review as a conference paper at ICLR 2023
Yutian Chen, Xingyou Song, Chansoo Lee, Z. Wang, Qiuyi Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, A. Doucet, Mare' Aurelio Ranzato, Sagi Perel, and Nando de Freitas. Towards learning universal hyperparameter optimizers with transformers. ArXiv, abs/2205.13320 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition scale. ArXiv, abs/2010. 19999, 2021.
Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for bayesian opti- mization. ArXiv, abs/ http/8802.0219, 2018a.
Matthias Feurer, Benjamin Letham, Frank Hutter, and Eytan Bakshy. Practical transfer learning for bayesian optimization. arXiv preprint arXiv: /002.82022222 02, 2018b.
Yu. V. Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. ArXiv, 1tt/2106.111552 2021.
Ryan-Rhys Griffiths and José Miguel Hernández Lobato. Constrained bayesian optimization for automatic chemical design using variational autoencoders. Chemical Science, 11:577 586, 2020.
James Hensman, Alexander Matthews, and Zoubin Ghahramani. Scalable variational gaussian pro-
cess classification. In Artificial Intelligence and Statistics, pp. 351-360. PMLR, 2015.
Xin Huang, Ashish Khetan, Milan W. Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. ArXiv, abs/2012.06678, 2020.
Ksenia Korovina, Sailun Xu, Kirthevasan Kandasamy, Willie Neiswanger, Barnabás Póczos, Jeff G. Schneider, and Eric P. Xing. Chembo: Bayesian optimization of small organic molecules with synthesizable recommendations. In AISTATS, 2020.
Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshmi- narayaman. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33:7498-7512, 2020.
Prasanta Kumar Mahapatra, Susmita Ganguli, and Amod Kumar. A hybrid particle swarm opti- mization and artificial immune system algorithm for image enhancement. Soft Computing, 19: 2101-2109, 2015.
Sebastian W. Ober, Carl H. Rasmussen, and Mark van der Wilk. The promises and pitfalls of deep kernel learning. In Cassio de Campos and Marloes H. Maathuis (eds.), Proceedings of the Thirty- Seventh Conference Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Ma- chine Learning Research, pp. 1216- PMLR, 27-30 Jul 2021.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 4055 4064. PMLR, 10- 15 Jul 2018.
Valerio Perrone, Rodolphe Jenatton, Matthias W Seeger, and Cedric Archambeau. Scalable hyperpa- rameter transfer learning. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Desa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Sebastian Pineda-Arange Hadi S. Jomaa, Martin Wistuba, and Josif Grabocka. HPO-B: A large- scale reproducible benchmark for boack-box HPO based on openml. Neural Information Process- ing Systems (NeurIPS) Track Datasets and Benchmarks, 2021.
10

---

Under review as a conference paper at ICLR 2023
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. In Adaptive computation and machine learning, 2009.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan Adams, and Nando de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1): 148-175, 2016. doi: 10. 1111://0O...21..244444444.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat Prabhat, and Ryan P. Adams. Scalable bayesian optimization using deep neural networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37, ICML' 15, pp. 2171-2180. JMLR.org, 2015.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In NIPS, 2016.
Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. cess optimization in the bandit setting: No regret and experimental design. arXiv:09/2.3999 2009.
Gaussian pro- arXiv preprint
Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-tast bayesian optimization. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.). Advances in Neural Infor- mation Processing Systems, volume 26. Curran Associates, Inc., 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pultern Recognition (CVPR), pp. 2818-2826, 2016.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
Michael Volpp, Lukas P. Fröhlich, Kirsten Fischer, Andreas Doerr Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning in bayesian opti- mization. In International Conference on Learning Representations, 2020.
Andrew G Wilson, Zhiting Hu, Russ R Salakhutdinov, and Eric P Xing. Stochastic variational deep kernel learning. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016a.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. ArXiv, abs//511..022/1 2016b.
Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates. In International Conference on Learning Representations, 2021.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thiemas Two-stage transfer surrogate model for automatic hyperparameter optimization. In Paolo Frasconi, Niels Landwehr, Giuseppe Manco, and Jilles Vreeken (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 199- 214, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46128-11
Martin Wistuba, Nicolas Schilling, and Lars Schmidt--hiemee Scalable gaussian process-based transfer surrogates for hyperparameter optimization. Machine Learning, 107:43-78, 2017.
Dani Yogatama and Gideon S. Mann. Efficient transfer learning method for automatic hyperparam- eter tuning. In AISTATS. 2014.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In ICML, 2022.
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 AN ILLUSTAAVVVE EXAMPLE OF THE MULTI-SOURCE FT-TRANS TE--TRANSFRRMER
Assume there are overall five parameters be optimized: x X1 X2, X3, X4, X5 where in the source datasets, only the first four parameters have been used and parameter X5 only occurs in the target optimization task.
Assume we now have two tasks. Task I has parameters x' = X2] and task 2 has parameters x² = X2, X3. X3.xx1 x1)". Therefore, each example of parameters (+)) E IR² in task 1, whereas x(i)) R in task 2. Examples of response y(u E R are the same in both tasks.
Inputs
Embedding parameters
Embeddings
Inputs
Embedding parametens
Embeddings
CLS)
Wi W;
Task
Ww
W2
D1
X
Target
Task
W:
D1
8
Source
Ii
W; by
My X
Tesk W,
X
(a)
Figure 6: Illustration of feature tokenizer in our joint training framework. (a) In pre-training, each variable in examples is encoded using its corresponding embedding parameters; (b) For a target example, embedding parameters of input variable that cxists in the training tasks would be directly adopted. Those new input variables would be constructed a mixed-up manner: we first ran- domly sample two existing embedding parameters w, and wy, then their convex combination aww + (1 - az) Wy, where a ~ 4(00, 1), as their embedding parameters.
During pre- pre-trainining, we construct Feature Tokenizer with three trainable parameters:
Classification token embedding [CLS] E RRd
Weight embedding W [w,, u2, W3, wa]" E Bixde
Bias embedding B = [bi, b2, by, b4] E IR 4xd.
The forward-pan -aassdd-asssaaa of the Feature tokenizer would
Task 1: FT((())) stack( [CLS], w1 x b, w2 x x (6) bu);
Task 2: (T()()) stack( [CLS] 1 W2 x2 (i) b2 W3 x 23 by. Wa X x + ba).
(4)
(6)
The tokenized features of (i) would be fed into the shared transformer layer and the [CLS] embedding of the output embedding (+)) would be used as the output embedding
(6) = (ransformer(FT((<())) E TRd.
In this way, datasets with heterogeneous design space can be jointly trained, the output embedding can be fed into linear layer for MSE training, and Gaussian process layer for ELBO training.
A.2 HYPER-PARAMERR PARAMETERS AND TRAINING DETAILS THE MULTI-SOURCE FT-DKL MODEL.
Unless specified, the following dyper-parameee were used.
128 dimension embedding
three transformer layers
512 dimension feed-forward ML.P
12

---

Under review as a conference paper at ICLR 2023
Freeze the first dimensions AckleyOffsetScale
GP
20.0
FT-DKL
Pretrained FT-DKL
17.5
15.0
10.0
7.5
5.0
2.5
25
50
75
100
125
150
175
200
Figure 7: Compare the Pretrained FT-DKL vs Gaussian process on scaled and offsetted 30D-Ackley function, where the first 20 dimensions were fixed to the optimal values. We can see that it is easier to optimize this 10-dimensional function compared to the 30-D modified Ackley function; however, FT-DKL with pre-trae pre-training still outperformed Gaussian process and un pretraiined FT-DKL.
learning rate for Transformer and embedding layers: le-5
learning rate for variational parameters with natural gradient: 0.1
learning rate for Gaussian process kernels and inducing point: le-3
We used 128 inducing points for the Ackley and the wireless network optimization problems, and 512 inducing points for HPO-B optimization. We used Adam optimizer to optimize all parameters except for the variational parameters where natural gradient descent was used.
As the model was firstly pre-trained with linear output layer and MSE loss, we found that com- monly used RBF-like kernels generally have worse ELBO loss, so used additive kernel with a mix of linear and matern3/2 kernel. The variance tialized as ceil();), where o the variance of the weight of linear output layer optimized with MSE loss. Also, during the training of Gaussian process with ELBO, we removed the dropout and layer-normalization of FT-Transformer to improve training stability.
of the linear kernel was ini-
We used StandardScala from sklearn to normalize input for the experiment of modified Ackley and wireless network optimization, and the SbantileTrane fo rmer for the HPO-B benchmark. Generally, we found that Quant ieaatileeT wettiieeerrasrrrreeee used with FT-Transformer lead to better prediction accuracy; however, when used with Gaussian process, Quant Tranass f ormer might cause very bad in-between uncertainty prediction. That's 's not a big issue for the HPO-B bench- mark, as there are more than three million source data points for pre- pre-training.
A.3 ADDITIONAL. EXPERIMENTAL. RESULT FOR THE ACKLEY FUNCTION OPTIMIZATION
To optimize the 30-dimensional Ackley function with input scaling and offset, we firstly performed Evolutionary optimization on the 20-D OcaleSsca function, and then during optimization of the 30-D function, fixed the first 20 dimensions the optimal value of the 20-D function and only optimized the rest 10 dimensions.
We compared Gaussian process and FT-DKL against pretrained FT-DKL model. As shown in Figure 7 we can see that although GP and FT-DKL was able to converge to near-optimal solution when only optimizing 10 dimensions, it was still outperformed by the pre-trained FT-DKI.
13

---

Under review as a conference paper at ICLR 2023
Search space No. 4796
Search space No. 5527
Search space No. 5636
Search space No. 5859
Average Rank
Average Rank
Average Rank
Average Rank
7.5
5.0
5.0
2.5
2.5
15 45 60 75 9D105
15 30 60 75 90105
15 30 45 60 75 90105
15 60 75 90105
Number trials
Number of tralls
Number trials
Number trials
Search space No.
Search space
Search space 5906
Search
Average Rank
Averape Rank
Average Rank
Average Rank
7.5
5.0
S
15 45 60 75 90105
15 75 90105 Number of trials
15 30 45 60 90105
45 60 75 90105
Number trials
Number trials
Number of trials
Search space No. 5970
Search space No. 5971
Search space No. 6766
Search space No. 6767
Average Rank
Average Rank
Average Rank
7.5
Average
5.0
Number
Search space No. 6794
Search space No. 7607
Search space No. 7609
Search ace No. 5889
Average Rtnk
Average Rank
Auerage Rank
Average Rank m 59
3D 45 60 75 90105 Number trials
15 30 60 75 90105 Number of trials
15 30 45 60 75 90105
15 45 60 75 90105
Number of trials
Number of trials
DNGO
Figure 8: Per-space average rank comparison HPO-B
A.4 PER-SPACE RESULT OF HPO-B EXPERIMENTS
14

---

Under review as a conference paper at ICLR 2023
Search space No. 4796
Search space No. 5527
Search space No. 5636
Search space No. 5859
10-4
Regret
Average Begret
Avcroge Rogret
20
60 75 90 105
15 30 60 75 90 105
15 30 45
105
90 105
Number trials
Number of trill
Number of trias
Number of trials
Search space No. 5860
Search space No. 5891
Search space No. 5906
Search space No. 5965
10-1
Avorage Repres
Avorage Rearet
1.0"
10
10
10
10-2
60
305
00 75
30 65
105
Number of trials
Number of trials
Number of trals
Number of tials
Search space No. 3970
Search space No. 5971
Search space No. 6766
Search space No. 6767
10
nugret
10-1
Average Begret
Avsrnee Regret
10-
10-3
10-4
1.0
AverAge
I0-4
10
10
15 30 45 60 75 90 105
15 30 45 50 75 90
105
15 30 45
105
15
45 60 75 TO 105
Number trais
Number of brals
Number of trais
Number of trais
Search space No. 6794
Search space No. 7607
10:1
Search space No. 7609
Search space No. 5889
Average Raggut
Regret
Average Rogret
Ave agg Rooret
10-2
2.0
10-3
10
50 45 60 75 90 105
15 50
50 75 90
105
15 30 43
75 105
68 75 90 105 Number of trals
Number traal
Number of brals
Number of trals
Sancom
7ST
SGFE
DNGO
+580
D07
FAP
Figure 9: Per-space normalized regret comparison on HPO-B
15


Reviews:
[{"review_id": "_7aMeGakcmR", "reviewer": "Reviewer_cAwj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose the use of sparse-variational GPs with a deep kernel and a transformer architecture to conduct hyperparameter optimization across different search spaces. They conduct an empirical comparison on three different tasks against most of the state-of-the-art.", "summary_of_the_review": "This work presents an interesting work that attempts to leverage knowledge from observations made on similar search spaces. There are several concerns regarding the experimental protocol: missing related work, possible test leakage, lack of clarity of the impact of using the additional data and what is causing it, and whether the improvements hold when considering wall clock time.", "strength_and_weaknesses": "**Strengths**\n- Interesting idea\n- Many experiments\n\n**Weaknesses**\n- Experimental setup (details below)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "V5gkYTQHtI_", "reviewer": "Reviewer_DKdC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work suggests a Bayesian optimization strategy with deep kernel learning and a feature-tokenizer Transformer, where we are given multiple heterogeneous datasets. By utilizing deep kernel learning and the feature-tokenizer Transformer, it initializes a set of parameters for the feature-tokenizer Transformer in order to adopt to a target task. Finally, the authors demonstrate the experimental results to show the effectiveness of the proposed method.", "summary_of_the_review": "Therefore, the current version is not enough to be accepted in ICLR 2023; please see the aforementioned comments.", "strength_and_weaknesses": "From now, I will describe the strengths of this work first.\n\n## Strengths\n\nI think the most important strength of this work is that the proposed method is capable of handling diverse datasets defined on different search spaces.\n\nAlso, a simple method based on a mix-up initialization technique is quite impressive. I think that the simplicity of the algorithm is quite important in terms of reproducibility and versatility.\n\nHowever, it has some limitations regarding presentation and reasoning. Moreover, the novelty of this work is also my concern.\n\n## Weaknesses\n\nFirst of all, I think the presentation of this work can be improved by revising sentences, elaborating explanations, and removing redundant contents. Although an introduction section is clear, a background section needs to be improved. For example, the explanation of Gaussian process regression is quite redundant. I think that the writing about deep kernel learning for Gaussian process regression should be described. I am curious how deep kernel learning is used in this work and which part of deep kernel learning is significant in proposing the algorithm. The description on Bayesian optimization is similar. Since I am quite familiar with Gaussian processes and Bayesian optimization, I can understand Section 3 even though the background section is not kind, but I think most readers who are not familiar with these topics are difficult to understand this work. Also, the description of feature-tokenizer Transformer is not enough. I am curious about the architecture of the Transformer and its details, e.g., layers, layer normalization, and inputs/outputs. Although the feature-tokenizer Transformer is directly adopted in your work, it should be carefully discussed. Also, Algorithm 1 seems like the plain sentence rather than a pseudo code. I think that Figure 6 is more helpful to understand this work. This figure should be located on the main manuscript.\n\nMore importantly, reasoning on why your algorithm works well is not sufficient. I carefully read Section 3.2, but it does not explain why it works well. As describe above, the mix-up initialization strategy is very clear and it seems effective. However, the reasoning is missing. I would like to hear why it is effective.\n\nI would not like to mention the novelty of this work directly. However, the proposed method is a combination of existing work. I emphasize that this fact is not important, but you need to describe why you choose such a combination and which part of the algorithm shows a certain circumstance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DGHA5tQruQ", "reviewer": "Reviewer_xRh6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the problem of black-box optimization of expensive black-box functions when data from multiple related optimization tasks is available. The key contribution is to employ a Feature-Tokenizer (FT) transformer to learn from multiple source datasets where each dataset might be defined over different input space. A deep kernel learning based gaussian process surrogate model is trained on top of the transformer representation to be plugged in with standard Bayesian optimization techniques on the target task. Experiments are performed on 3 benchmarks including hyper-parameter optimization and wireless network optimization. \n", "summary_of_the_review": "Overall, I think the paper considers an important problem and provides a simple solution based on existing ideas. However, some of the design choices and the experimental analysis require more elaborate discussion to improve the quality of the work. ", "strength_and_weaknesses": "- The paper considers an important problem since multiple related optimization tasks are being solved quite commonly nowadays (for e.g. hyper-parameter optimization of ML models).\n\n- The proposed approach performs well on the benchmarks employed for experimentation and analysis. \n\n- The illustrative example in the appendix is really nice and useful to understand the transformer architecture employed in the paper.\n\n\nHowever, I have few questions and suggestions to understand some of the details better and that will hopefully improve the paper's contributions:\n\n\n- Although there is a passing remark about generalization issues with deep kernel learning (especially in the small data setting) in limitations section, this should be discussed in the beginning with more elaborate discussion. One way to test the surrogate model independently of the BO loop is to construct a standard train-test scenario and evaluate the model log-likelihood or mean squared error on the test set. \n\n- It is mentioned that most of the existing meta-learning/transfer learning approaches in Bayesian optimization are not applicable for varying input spaces. However, one straight-forward way to employ them is to pick a large enough search space that contains a union of input spaces from all the source tasks. What would be the advantages of the proposed approach over this simple baseline?\n\n- The mix-up initialization for the unseen task parameters is defined without proper justification. Please add more details about this choice including some principles that make it pertinent to the problem setting. \n\n- It is surprising that only one benchmark in the experimental evaluation has more than one source tasks (i.e. only HPO-B has 758 source tasks while other has only 1 source task). This seems to limit the analysis of the proposed approach which would be presumably most effective with multiple sources. It is absolutely not necessary to do this in the rebuttal period but if possible, please consider adding another benchmark with multiple source tasks to show the effectiveness of the proposed approach in a more robust way.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject