Towards Learning Universal Hyperparameter Optimizers with Transformers
Authors: Yutian Chen
Xingyou Song
Chansoo Lee
Zi Wang
Qiuyi Zhang
David Dohan
Kazuya Kawakami
Greg Kochanski
Arnaud Doucet
MarcAurelio Ranzato
Sagi Perel
Nando de Freitas
Venue: NeurIPS 2022 Accept (2022)
Keywords: OptFormer
Transformer
hyperparameter
optimization
offline
tuning
meta
learning
meta-learning
bayesian
optimization
blackbox

Abstract: Meta-learning hyperparameter optimization (HPO) algorithms from prior experiments is a promising approach to improve optimization efficiency over objective functions from a similar distribution. However, existing methods are restricted to learning from experiments sharing the same set of hyperparameters. In this paper, we introduce the OptFormer, the first text-based Transformer HPO framework that provides a universal end-to-end interface for jointly learning policy and function prediction when trained on vast tuning data from the wild, such as Google’s Vizier database, one of the world’s largest HPO datasets. Our extensive experiments demonstrate that the OptFormer can simultaneously imitate at least 7 different HPO algorithms, which can be further improved via its function uncertainty estimates. Compared to a Gaussian Process, the OptFormer also learns a robust prior distribution for hyperparameter response functions, and can thereby provide more accurate and better calibrated predictions. This work paves the path to future extensions for training a Transformer-based model as a general HPO optimizer.

Towards Learning Universal Hyperparameter Optimizers with Transformers
Yutian Chen', Xingyou Song", Chansoo Lee", Zi Wang", Qiuyi Zhang", David Dohan", Kazuya Kawakami', Greg Kochanski', Arnaud Doucet Marc'aurelio Ranzato', Sagi Perel?, Nando de Freitas' Deepmind, "Google Research, Brain Team
Abstract
Meta-learning hyperparameter optimization (HPO) algorithms from prior experi- ments is promising approach to improve optimization efficiency over objective functions from similar distribution. However, existing methods are restricted learning from experiments sharing the same set of hyperparameters. In this paper, we introduce the OPTFORMER, the first text-based Transformer HPO framework that provides universal end-to-end interface for jointly learning policy and func- tion prediction when trained on vast tuning data from the wild, such as Google's s Vizier database, one of the world's 's largest HPO datasets. Our extensive experiments demonstrate that the OPTFORMER can simultaneously imitate at least 7 different HPO algorithmss, which timates. Compared to a Gaussian Process, the OPTFORMER also Iearns a robust prior distribution for hyperparameter response functions, and thereby provide more accurate and better calibrated predictions. This work paves the path to future extensions for training Transformer bbasoedd model general HPO optimizer.
be further improved via its function uncertainty
Introduction
The emergence of public machine learning data platforms such as OpenML. [1] and hyperparameter optimization (HPO) services such Google Vizier [2], Amazon SageMaker [3] and Microsoft Azure [4] have made large-scale datasets containing hyperparameter evaluations accessible. For our use-case in this paper, Google Vizier is the de-facte HPO service across Google, having optimized some of Google's ' largest products and research efforts, and contains a collection of valuable tuning data within the last 5 years. While there is growing interest in leveraging such data meta-learn hyperparameter optimization algorithms [5-8], dealing with large datasets consisting of experimental trials in the wild can be challenging, due large variations in HPO problems and their associated text metadata (e.g. shown later in Table 1).
Thus, most
and transfer-learingg HPO methods [7-16] consider restrictive setting where all tasks must share the same set of hyperparameters so that the input data can be represented as fixed-sized vectors. Consequently, such methods only exploit small portion of the available data to learn priors. This drawback more severe for large datasets which contain significant amounts of useful information.
To overcome these limitations, we introduce the OPTFORMER, general hyperparameter optimization framework based on Transformers [17]. Transformers have demonstrated excellent performance in many data tasks, ranging from natural language [18]. images [19, 20], biological data [21, 22]. code [23, 24], and control [25, 26]. Here, we investigate how to use Transformer
universal interface for modelling experimental data and learn HPO algorithms, as given a sufficient amount of data, Transformer can potentially learn
complex prior distribution than standard Bayesian Optimization (BO) with Gaussian Processes (GPs), especially as the Transformer possesses certain computational advantages over GPs for large datasets.
Code: https: bittps://githuas cowp://hhuu.cm//oogggggee wwwprooeeeeeeeerreetttooooaem Google AI Blog: https:// ai. woggoollellogoooo com 2ttomm/22///////// fome: htt.:///.......tttoorre-e---aaae www.rtw--niieeeeeeeeeeeeeee hetml...llllll..................
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

1e-3
op "SGD"
Ir 1-4
opr opr="Adam "Adam"
acc=0.7
Ir=3e-4
Opps opr="Adam" Adom"
OptFormer
convenl cdarit. accureey
GSt Husle deuble (1a-d. 1a-3) opt typa catagoical Fi900r, "Acteor)
1o-3
opt "SGD"
0.c
e-4
optn Adam"
0,7
Ir-3e-
Metadata
Trial
Figure I:
Illustration of the OPTFORMER model over hyperparameter optimization trajectory. is trained to predict both hyperparameter suggestions (in green) and response function values (in red).
We introduce a serialization scheme to convert combination of any metadata and an optimization trajectory into text, represented as a sequence tokens, and formulate the HPO task as a sequence modeling problem. We adopt supervised learning approach, by learning to predict parameters and hyperparameter response functions from offline tuning data (See Fig. 1). In order further improve optimization performance, we augment the model by utilizing its own function prediction during inference (Section 4.3). Extensive experiments both public and private datasets demonstrate the OPTFORMER competitive tuning and generalization abilities.
In summary, contributions are follows:
We formulate, to the best of our knowledge, the first meta-learning HPO framework learn both policy and function priors from data across different search spaces.
The OPTFORMER capable of learning the behaviors of diverse blackbox optimization algo- rithms relying broad class of methods (non-adaptive, evolutionary, and Bayesian).
Furthermore, the OPTFORMER learns the prior over objective functions and provides both accurate and well calibrated predictions, in many cases significantly surpassing GPs in log-predictive likelihood and expected calibration error (ECE) [27].
Lastly, OPTFORMER policies augmented with model-based optimization, such as the use of Expected Improvement acquisition functions, are competitive HPO algorithms. To the best of our knowledge, this the first time Transformers are augmented with acquisition functions for online adaptation.
Preliminaries
2.1
Meta-learning for hyperparameter optimization
HPO aims to find a set of hyperparameters x from search space X' to maximize a model performance metric, y fz),, often referred to response function. Table shows example of HPO experimental data. Following the HPO nomenclature [2, 28), an experimental study consists of metadata (m) and history of trials (h). The metadata contains arbitrary unstructured information, including but not limited to descriptions of the problem, optimization algorithm, names, types and value ranges of hyperparameters. The history after trials, h; (I1, ./...., Zt, Mt), contains sequence of trials, each of which consists of parameter suggestion æ and function value y.
The goal of the meta-learning approach for HPO is to learn the shared knowledge among the objective functions from a dataset of multiple tuning experiments represented as studies and to obtain an optimal HPO algorithm for new hyperparameter tuning tasks from similar distribution to those in the dataset.
An HPO algorithm maps the metadata and history to a distribution over hyperparameter suggestions, i.e. x (a m, h4)) Using the terminology of offline RI. [29], we refer the algorithm used to generate the trajectories in a datasct the behavior policy Rb.

---

We primarily consider search spaces A' with a fixed number D of hyperparameters per task, and hence æ = (x""),.,,x")) I(D)), with each hyperparameter x(()) being of type DOUBLE, INTEGER, DISCRETE, or CATEGORICAL (see Appendix A.I for details). More complex search spaces can be supported as discussed in Section 7.
Table 1:
Example of a study (m, h) with two parameters and two trials. Metadata m appears in blue and history h in purple.
2.2 Transformer model
The Transformer model is an efficient iicention-bbsedd neural network architecture for sequence modeling [17]. We adopt the T5 Transformer encoder-decoder architecture [30]. The encoder and decoder each consist of a stack of multi-head self- attention layers which construct pairwise interactions between positions, followed by position-wise feed-forward networks. The encoder converts a sequence of input token representations m, to a sequence of continuous embeddings, which is fed to the decoder to generate sequence of output tokens h one element at a time (see Fig. 1).
"name": "convnet cifar10".
"metric"; "accuracy",
"goal": "MAXIMIZE""
"algorithm": "random search",
"parameter":
*name": "opt_kx.1r",
"type"; "DOUBLE",
"min_value": 1e-6,
*max_ max__aluer: 1e-2,
3 Related work
"scale_type" "LoG"
"parameter": {
There has been a rich of works in meta-learning and trans- fer learning by modifying specific core components of the BO pipeline, such as the acquisition function or the GP, in order tackle BO's myopic behavior, or obtaining more information from similar tasks. For instance, approaches include learning new acquisition functions [31], multi-task BO [7-13] and BO for transfer learning using contextual GPs [14-16]. [32] also studies the use of meta-BO BO for hyperparameter tuning tasks in machine learning. However, all of these works consider a fixed search space.
"name": "apt_type"
"type": "CATECOKICAL",
cazegories"
""Sc"", "Adam"),
"trial"
"parameten";
*opt_kw 1nt D.CD21237573,
type_type": "SGD*
"metric":
more radical meta-learning approach non-differentiable optimization trains recurrent neural networks (RNNs) as neural optimizers from scratch. [33] first proposed training an RNN with gradient descent to optimize blackbox functions and hy- perparameters while [34, 35] train RNNS using reinforcement learning (RL) to solve RL tasks. Unfortunately, prior works are limited fixed search spaces and only use online generated data,
"accuracy": 0.69482429, })
"trial"
"parametes7:
hop__kw.1975
D. D.C0038292334,
"opc_type": "Adam"
constraining the training objectives to be cheaply computable.
messic":
"acurracy:: 0.71642583
In this work, we wish to overcome the limitations of previous works by exploiting the Transformer architecture. Numerous works have demonstrated Transformers' strong capabilities in flexible symbolic and numerical ma- nipulation. On the symbolic side, Transformers have been shown able to manipulate symbolic mathematical expressions [36-38] and generate code [23, 24]. Furthermore, on the numerical side, Transformers have also been shown able to perform linear algebra computations [39], Bayesian Inference [40], and offline RL. [25, 26, 41]. For AutoML specifically, [42] has demonstrated Trans- formers' and analogous graph neural networks' abilities to use dataset descriptions and metadata to generate classification and data preprocessing pipelines. However, to date, there has been little effort in attacking the full problem of hyperparameter tuning in the blackbox optimization setting. In this paper, the challenging task of learning algorithms from blackbox optimization trajectories can be seen as significant extension of both symbolic and numerical manipulation. Since the underlying algorithm can composed of multiple symbolic and mathematical operations with unbounded complexity, the model must infer potentially very complex behavior over long horizons.
4 Universal interface and model for hyperparameter optimization
In this section, we provide a universal interface for modeling HPO studies with mixed textual and numerical information sequence of discrete tokens. We train our OPTFORMER generative
3

---

Table Serialized study after preprocessing and tokenization. Metadata m appears blue, normal- ized and quantized values of æ; in green, and in red.
<namex:" conneet on cifar10", cifario",cmetricx""" cemetrie:"accuuraa wewpaanysssoabb:MMMMMMMMAA.
<algorithm>;" "random search"
After preprocessing
"wp..ooprrr""r""o <scale bpe>:<LGG>
&-name>:"opt "opp kwdr"
&<names:"" "oppt htpp_tppp""""eppeet
http://wppeeeeceoooooooca/oaaore "Adam")
<831><0>> <t>i <645>cl>. <999>
name " con v net on ci far 10 metris: ucc "
MAXIMIZE, algorithm: random. search
Subwords after tokenization
& name : " op kw. ",type: Sope:DOBBLE, DOUBLE, min value: e-6, value
sewls__tapeeeeeeeeeeeeeesepp type LOG
& name : op type ", .type: CATEGORICAL,
categories
831 0 0 645 999
model given dataset and explain how to the OPTFORMER's parameter and function prediction abilitics to implement an HPO policy.
4.1 Study tokenization
To generalize over HPO problems of different parameter sizes, types, and metadata, we propose to serialize the study as one-dimensional textual sequence, also advocated in [26]. Unfortunately, naive serialization approach, e.g. via ISON [43], will produce unnecessarily long sequences.
To improve scalability, we compress the textual representation metadata m by removing redun- dant phrases and punctuation (e.g., "parameter", quotes) and encoding keywords (e.g.. "name", "algorithm") and enumerating types (e.g. "DOUBLE") into single tokens.
For the historical sequence h, we convert every DOUBLE and INTEGER parameter along with every function value into a single token, by normalizing and discretizing them into integers, with an quantization level of Q 1000: c.g.
int nnorm
Q). where Fnorm (z Imina (Fmax Frinn )-
The range of is defined by the search space and the range of y is obtained from observed values in h. For other types, we use the index in their value set.
The shortened text string is then converted to sequence of tokens via the SentencePiece tokenizer [44] (see Table 2 for an example). Every trial represented by text, which represented as a sequence of normalized and quantized tokens, x ,..., ((()) *,, Mr, "I" where the token separates parameter and function values and "I" separates trials. See Appendix A.2 for further details on tokenization.
4.2 Model and training loss
After tokenization, the converted historical sequence is as follows:
h 1
(2)
(D) *,
11. "I"
(D)
(2)
We can now apply a Transformer model to learn the conditional distribution tokens in using the chain rule, given the metadata m, depicted in Fig. 1. The joint distribution is presented in Appendix D.I.
Given dataset D of hyperparameter optimization studies, we train the OPTFORMER by maximizing the weighted log- likelihood for each study (m, h) D:
L(O; m. h) Σn wn log Po(T()) m. (1m--)))
(3)
with w. = if im)) E (*, "I"} and w, = 1 otherwise. That is, we mask out the separator tokens (*, "I") and predict parameter & and function tokens ÿ only. Note that ((()) denotes the n-th token, that the zi-th element of the list in Equation (2), and (11:n-1) denotes all tokens up to the (n - 1)-th token. Further details and data augmentations are provided in Appendix D.2.

---

4.3 Inference and decoding
Parameter prediction: To decode the predicted parameter token 2t back to its original parameter range, we truncate the output distribution to the vocabulary range corresponding to valid parameter values (0, Q) and reverse tokenization procedure in Section 4.1. For DOUBLE INTEGER parameter 2, we use a piecewise constant distribution:
(=
Q Pa(T) ....) (T mmin), if x E [Tmin, Imax], otherwise 0.
(4)
For all other parameter types, ε corresponds to the index of the set of feasible values. Putting these together, we may now sample parameter æ, from the model's prior distribution and thus define an HPO policy:
Peiior (2t 11-1) = II po ( (d) (mm, 1 Zi (2:d-1)). d=1
(5)
As we use a supervised learning loss, we expect Tprior to approximate the behavior policy mi.
Note that traditional BO algorithms require running Bayesian inference and then conducting a global search in the hyperparameter space with an acquisition function. Thus the runtime complexity of making one hyperparameter suggestion is cubic in for a typical GP- Gased BO method that performs ARD each iteration [45]. In contrast, generating one suggestion by the OPTFORMER consists of decoding D parameter tokens with an input sequence of (D + 3) tokens, which are then parsed into the D parameter values, producing runtime of ( linear in i, with proper caching.
Function prediction: To decode the real-valued function y/ from the discrete distribution Po 1,, dt). we construct the same piecewise constant distribution as in Eq. (4) with the range [ymin Vmax used in tokenization. Note that the limited support of will not be a concern for HPO when either the range is known values. For more general use as a few-shot function prediction model, one could consider adopting the Riemann Distribution in [40], which supports an unbounded range.
set the range large enough compared to observed
Augmented HPO policies with function prediction: At best, the learned policy Tprior can only perform well as the original policy n. when using behavioral cloning. However, can take advantage the model's simultaneous function prediction ability to improve the policy with model- based planning or offline RL techniques. While comprehensive study of policy improvements for Transformers is out of the scope of this work, we consider here a simple yet effective policy improvement operator: sampling M 100 candidate suggestions from Tprior and choosing the suggestion with the highest score defined by an acquisition function u(-) as follows:
(aue 1a-1) argmax u(po(-Im. h, ae (f)), with x (i) Tprior (ar) m, h1- {
(6)
Common acquisition functions include Expected Improvement (EI), Probability of Improvement
(PI), Upper Confidence Bound (UCB), and Thompson Sampling, see for example [46]. At a high level, this approach to combining imitated policies with function prediction is reminiscent of the idea behind the offline RL approach of BCQ [47].
Because apply linear mapping from the original y value to the quantized value before discretiza-
tion, we can simply deñin the acquisition functions on the discrete distribution Po m. h4-1, Z.:) as follows:
Wen (20g") = Era [max( - ["]]
Ucun (x](a) = (Dantile(Po((M)) hom,hh-11 h, a):
(8)
Upi (x=l²)) ((7")= = Σ Po m, 2)-
(9)
(rs(x) = T. with T Po m, h1-1,
(10)
where 3" = max+gi-1 Dr in El and PI is the threshold to measure improvement. We define the UCB acquisition function with quantile parameter a. Our TS acquisition is defined as a sampled function value at given location from the marginal predictive distribution. It is inspired by the traditional Thompson Sampling method [45] but different in that the correlation between different locations ignored.
5

---

5 Data
Training the OPTFORMER requires HPO studies with optimization trajectories. The most natural dataset we possess is the entire Google Vizier [2] database, one of the world's largest collections of real world hyperparameter tuning studies, which we denote as RealWorldData. There are around 750K studies, each with on average 300 trials, covering a vast class of production and machine learning applications at Google, ranging from vision, speech, NLP and robotics, and representing one of the most representative distributions of HPO tasks for machine learning models in practice. These studies were generated with a mixture of non-adaptive, evolutionary, and BO algorithms. However, as the dataset does not contain sufficient algorithm information, we have to treat the corresponding behavior policy as a randomly mixed algorithm Tb.
In addition, we create two new datasets based on public benchmarks. HPO-B is the largest public benchmark for HPO containing about 1.9K tuning tasks, most of which use one of 16 shared search spaces. In the continuous evaluation setting, it fits an XGBoost model to the trial data of every tuning task as the objective function. For further control over specific function dimensions and properties, we use the blackbox optimization benchmark BBOB [48], consisting of 24 types of synthetic functions with customizable properties (dimension sizes, rotations, shifts, discretizations, noise types) we randomize over.
For each of the two public benchmarks (HPO-B and BBOB). we apply a fixed set of 7 HPO algorithms to generate a dataset of optimization trajectories. In contrast to RealWorldData, we specify the algorithm name in the metadata m as part of the conditioning input for our model. The controlled algorithms used are: (1) Grid Search, (2) Shuffled Grid Search, (3) Random Search, (4) Regularized Evolution [49]. (5) Hill-Climbing, (6) Eagle Strategy [50], and (7) Vizier's GP-UCB [2]. Appendix B contains detailed explanations of the algorithms.
Table 3: Offline training datasets considered in this study. More details are given in Appendix C
along with examples of studies in Table
("R") Real
("H") HPO-B
("B") BBOB
#Studies
750K
10M
10M
#Trials/ / study
300 (on average)
120
300
Study source
Google's 's database
Generated
Generated
T
Mixed
Controlled
Controlled
Obj. Functions
HPO tasks
HPO tasks
Synthetic
Search space
Different per task
16 shared search spaces
Randomized
Experiments
We train single Transformer model with 250M parameters the union of the three datasets described above, RealWoridData, HPO-B, and BBOB (hyperparameter details in Appendix D.2).
Each dataset contains a corresponding "test" set of functions, either using synthetic functions (BBOB) or fitting a machine learning model to obtain the objective (Real WorldDaa, HPO-B). We evaluate mainly on the two natural HPO benchmarks. RealWorldData and HPO-B. The train/test subsets of Real Darl are split temporally to avoid information leak (see Appendix C for details).
To aggregate results across functions with different output scaling, we normalize all the test functions. This is standard practice in the literature [2, 5, 51-54). We define our performance metric at trial t as the hest-so-far normalized function value max;g(1:x) (y: /rand)/ (Jumu - frand), where Vrand the median of function values randomly sampled in the search space to be robust to outliers, and 3max is the maximum, if known, or best value found by any algorithm. For the HPO-B benchmark. we use the recommended bounds provided in [5]. We also consider other metrics when comparing different algorithms in Appendix E.3, including the performance profile and average ranking. We find our results are consistent over different metrics.
Because the OPTFORMER is trained to predict the conditional distributions of parameter and function values, we would like to answer the following questions when evaluating on unseen test problems:

---

Regularized Evolution
Regularized Evolution
1.0
1.0
OptFormer
Best normalizzed function
0.8
Target policy
Grid Search
Density
0.6
Shuffled Grid Search
Random Search
0.4
Eagle Strategy
0.4
Target policy
Regularized Evolution
OptFormer
Hill-Climbing
0.0
0.2
Trial
1.00
Vizier
Hill-Climbing
Hill-Climbing
1.5
OptFormer Target policy
ner function
1.0
OgtFormer policy
0.95
0.8
0.90
Density
1.0-
0.85
0.5
0.4
Target policy
OptFormer
0.80
0.80
0.85
0.90
0.95
1.00
0.0
0.2
Target pollcy
X
Trial
(c) Average best function at trial 100 with standard deviation.
(a) Policy distribution
(b) Optimization trajectory
Figure 2: Comparing the performance of different algorithms outputted by the OPTFORMER condi-
tioned on the corresponding algorithm's 's name.
1. Can the OPTFORMER learn to imitate multiple HPO algorithms with one model? (Section 6.1)
2. Can the OPTFORMER learn good prior over hyperparameter response functions? (Section 6.2)
Is the OPTFORMER competitive approach for HPO? (Section 6.3)
6.1 Imitating HPO policies
We first evaluate how well the OPTFORMER can learn the conditional distribution of parameter suggestions given by the behavior policies in the dataset, and how well can imitate multiple algorithmss. As the algorithm's name is contained in the metadata m, the policy prio ( m, f2z) simply by altering this variable. Fig. 2a compares two different policies the OPTFORMER, when it is conditioned the corresponding policy name. We observe good match between the imitated algorithms and the OPTFORMER (additional algorithms are shown in Appendix 1)
modify the behaviour of
Fig. 2b we run target policies on the BBOB dataset's test functions and compare the optimization trajectories of the algorithms and the OPTFORMER. In Fig. 2c compare the average and standard deviation of the best normalized function values at trial 100. Our model imitates most algorithms very accurately in both the mean and variance except for the most complicated algorithm, Vizier, where prio is slightly worse the LUNACEK benchmark. We expand on this Appendix E. Because Vizier is the best performing HPO algorithm among all considered, the OPTFORMER will imitate Vizier faithfully, although not perfectly, in the following experiments.
6.2 Learning priors for hyperparameter response functions
In this section, assess the OPTFORMER's ability to learn the conditional distribution of the function value as a few-shot function regressor. Specifically, for every function in each test dataset, repeatedly sample up 200 random trials (I1, V1... Zt. yt), 200, and predict the conditional distribution p(ye 1 ./...., Tt). We compare with a GP model with output warping - details provided in Appendix B. We report the log- ongprrddicive likelihood log p(ys (z....) in Table 4.
As uncertainty estimation is important for HPO, we also evaluate how well the function predictive distribution calibrated. When predictive distribution (o( matches the true distribution, the estimated CDF F(y) = f- pe (y'l..... )dy will be uniformly distributed. In Fig. 3, we plot the cumulative histogram F(y) on RealWorldData test set and check the deviation from the diagonal line to assess goodness-of-fit proposed by Rosenblatt [55]. The OPTFORMER has smaller
7

---

Percentage of data with
1.0
GP
Table 4: Log-pue pred-ricieeive likelihood (with 1-std. stan- dard error, higher better (f)) and ECE (percentage of error, lower is better ())) on WarlddDaaaa and HPO-B test sets.
0.8
OptFormer
Vi
0.6
CDF(y)
Log-predictive likelihood ata HPO-B
0.4
Model GP
0.2
(0.06)
4.03(0.04)
OPTFORMER
2.12 (0.05)
6.16 (0.04)
0.0
ECE (percent %)
0.0
0.2
0.4
0.6
0.8
1.0
Model
RealWorldData
HPO-B
CDF level F
GP
5.34 (0.06)
2.39 (0.05)
Figure 3: Cumulative histogram of pre- dicted CDF(y) on RealWorldData test set.
OPTFORMER
1.11 (0.02)
1.89 (0.01)
RealWorldData
HPO-B
0.98
Random Search
Best normalized function
0.9
Best normalized Lizmallieed. function
GP-UCB
Vizier
0.8
GP
DGP
0.94
ABLR
0.7
FSBO
HyperBO
OptFormer
0.6
OptFormer (EI)
0.90
20
40
60
60
80
100
Trial
Trial
Figure 4: Higher is better. Best normalized function value averaged over 16 RealWorldData test functions (left) and over 86 HPO-B test functions (right) with 1-std confidence interval from runs. GP* and DGP* results are provided by [5]. The transfer learning methods ABLR, FSBO and HyperBO cannot be applied to RealWorldData.
deviation than the almost across entire range. We also compare calibration performance using the expected calibration error (ECE) [27]. Readers are referred to [27] and Appendix E.2 for a detailed explanation of ECE. We observe from Table 4 that the OPTFORMER achieves better predictive likelihood and ECE than the GP on both datasets.
6.3 Augmenting prior policy with function prediction
We evaluate the OPTFORMER hyperparameter optimization algorithm two benchmarks, RealWorldData and HPO-B. We compare our prior policy, the OPTFORMER, and an augmented policy with Expected Improvement, the OPTFoRMER (EI), against standard HPO baselines, including Random Search, our implementation of GP-UCB, and the well-tuned Vizier service. For HPO-B, also include the GP (not to be confused with GP-UCB) and DGP (GP with deep kemel)) baseline results provided by the original paper [5]. Additionally, we include three recent transfer- learning methods based on multi-takk GP models: ABLR [12, 56), FSBO [7], and HyperBO [57, 58] (implementation details in Appendix B). Please note that all of these transfer learning methods require learning GPs multiple tasks sharing the same search space. Therefore, none of them apply the Real WorldDaaa benchmark where every study has its own search space.
We show the trajectory of the best normalized function value averaged over all functions from each benchmark in Fig. 4. While the prior policy returned by the OPTFORMER does not perform as well Vizier, it is comparable or slightly better than our GP-UCB baseline and ABLR.

---

Training data
HPO-B
0.98
0.98
Best normalized function
Best n function
0.96
0.96
0.94
HyperBO
OptFormer (EI)
OptFormer
0.92
Optformer-H (EI)
0.92
OptFormer (EI)
OptFormerRR (EI)
OptFormermmin
OptFormer88 (EI)
OptForme-miin (EI)
0.90
0.90
100
60
Trial
Trial
(a) Ablation training data.
(b) Ablation metadata.
Prior policy
Acquisition function
0.98
0.98
Best norma Cormalized lized function
Best norma lized function
0.96
HyperBO
0.94
0.94
Optmorerr
OptFormer
OptFormer (EI)
0.92
OptFormer
0.92
OptFormer
Random Search
OptForee (PI)
Random Search (EI)
OptFormer (UCB)
0.90
0.90
20
60
80
100
20
40
60
80
100
Trial
Trial
(c) Ablation on the prior policy.
(d) Ablation on the acquisition function.
Figure 5: Best normalized function values averaged over HPO-B test functions with 1-std confidence interval. Ablation curves are shown with O markers. (a) The more similar the training dataset, the better the transfer. Here, the suffix with "H", "R", "B" indicates training on HPO-B, RealWorldData, and BBOB respectively. (b) Removing the majority of metadata hurts function prediction. (c) The prior policy improves performance with or without the Expected Improvement acquisition function. (d) All acquisition functions provide significant improvement.
The most significant improvement is achieved when augment prior policy with the Expected Improvement acquisition function. The resulting OPTFORMER (EI) outperforms baselines across the board on both benchmarks. This illustrates that the OPTFORMER is able to learn the distribution of functions in the meta-training split and transfers to the meta-testing split.
It is worth noting that to run 100 trials for about half of the test functions, the required history token sequence is longer than the 1024-token length used in training, with the maximum length about twice the training horizon. The superior performance of the OPTFORMER (EI) thus demonstrates its good generalization performance beyond the optimization horizon it is trained for.
6.4 Ablations
We provide further ablations three important components for our policy:
Training dataset. To understand the impact of the training datasets on the OPTFORMER, we train three variants on individual datasets (OPTFORMER-THH) "H", "B" respectively for RaataoridData. HPO-B, BBOB) and study their transfer learning performances on HPO-B. Fig. 5a verifies that training with in-domain data ("H") gives better performance than training over the more diverse across-domain Real WorldDatt HPO dataset ("R"), which is better than training over the synthetic BBOB data ("B"). Nonetheless, training on RealWor enough to give comparable performance to the best transfer learning baseline at the end of 100 trials. Lastly, training on all of the datasets (OPTFORMER) gives further advantage over OPTFORMER-H. This suggests that more data does not hurt the model's performance but rather may improve it, if the extra data is oft-oi-domain.

---

Meta-data m. We have demonstrated how the OPTFORMER's behavior can be controlled by the algorithm name in metadata m in Section 6.1. Here we study whether the OPTFORMER learns to depend on other meta information. At inference time, we provide minimum information in m (OPTFORMER-min) by excluding all textual information and parameter value ranges. We only keep necessary information such as parameter types and algorithm names. Fig. 5b shows that the prior policy of OPTFORMER-min performs comparably with the OPTFORMER, partly due to the use of data augmentation (see Appendix D.2). The augmented policy OPTFORMER-min (EI) (dashed orange) improves upon the prior policy but significantly worse than the full model, suggesting that the missing metadata impacts the model's predictions on function values.
Prior policy. Section 6.3 demonstrated the benefit of adding an acquisition function to the prior policy. A natural question is whether good prior policy is needed at all. In Fig. 5c, we replace the prior policy in the OPTFORMER (EI) with random search (Random Search (EI), dashed blue line). While adding Expected Improvement still improves this random search policy's performance, the best method requires both a good prior policy and the acquisition function.
Choice of acquisition function. In Fig. 5d, we compare the Expected Improvement (EI) with Thompson Sampling (TS), Probability of Improvement (PI), and Upper Confidence Bound (UCB) with confidence level of 0.9. We observe that the prior policy is improved by all the acquisition functions. Particularly, OPTFORMER (EI) is the best among all the choices though the difference is relatively small compared to the advantage
other baselines and OPTFORMER prior policy. We provide additional analysis with results on both the WorldDaaa and HPO-B datasets, as well as other evaluation metrics in Appendix E.4.
7 Limitations and future extensions
We list few limitations of this work and discuss some potential extensions. (1) We did not consider parameters that do not always apply or are subject to dynamic constraints depending on other parameter values. Such parameters are common in AutoMI. [59] and NAS applications [60]. Our work can be extended to support these applications, by providing the conditional specifications as text in metadata m. (2) We also considered only sequential optimization with batch size of 1. To support parallel suggestions, one could apply random masking to input function value observations to simulate scenarios with parallel pending trials [33]. (3) While we trained the Transformer to clone the behavior policy offline, there are extensive literature on offline RL [29] that could be applied here [25, 47, (1-64]. One could also consider meta- meta-training acquisition functions in [31] within the same model and online fine-tuning as in 17, 41]. (4) We considered a single objective function, though multiple objectives can be easily included by outputting multiple function tokens in trial. (5) The maximum sequence length is limited by the quadratic memory size requirement of a Transformer, which could be mitigated with more scalable architecture variants such as Performer [65].
8 Conclusion
We presented first step to learning a universal Transformer model for hyperparameter optimization from large scale datasets containing tuning experiments with vastly different search spaces and experiment descriptions. By training on a diverse set of synthetic and real-world tuning trajectories, we demonstrated the capacity of a single Transformer model to imitate fundamentally different HPO policies, learn to make well calibrated few-shot function predictions, and provide competitive optimization performance on unseen test functions comparable with the existing, long-tried GP-based baselines. Many extensions are readily conceivable for future exploration.
Acknowledgments
We would like to thank Chris Dyer, Luke Metz, Kevin Murphy, Yannis Assael, and Esteban Real for providing valuable feedback during their reviews of this paper. We further thank Sebastian Pineda Arango for technical discussions on the HPO-B benchmark and Christof Angermueller on biological benchmarks. In addition, we thank Daniel Golovin, Daiyi Peng. Yingjie Miao, Jack Parker-Holder, Jie Tan, Lucio Dery, and Aleksandra Faust for multiple useful conversations.
10

---

References
[1] Joaquin Vanschoren, Jan N Van Rijn. Bernd Bischl, and Luis Torgo, OpenML: Networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49 60, 2014.
[2] Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. Google Vizier: A service for black-box optimization. ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 1487-1495, 2017.
[3] Valerio Perrone, Huibin Shen, Aida Zolic, laroslav Shcherbatyi, Amr Ahmed, Tanya Bansal, Michele Donini, Fela Winkelmolen, Rodolphe Jenatton, Jean Baptiste Faddoul, Barbara Pogorzelska, Miroslav Miladinovic, Krishnaram Kenthapadi, Matthias W. Seeger, and Cédric Archambeau. Amazon SageMaker automatic model tuning: Scalable black-box optimization. CoRR, abs/2012.08489 2020.
[4] Deepak Mukunthu, Parashar Shah, and Wee Hyong Tok. Practical Automated Machine Learning on Azure: Using Azure Machine Learning Quickly Build AI Solutions. O' O'Reilly Media, 2019.
[5] Sebastian Pineda-Arango, Hadi S. Jomaa, Martin Wistuba, and Josif Grabocka. HPO-B: A large- scale reproducible benchmark for black-box HPO based on openml. CoRR. abs/2106.0620 2021.
[6] Hyunghun Cho, Yongjin Kim, Eunjung Lee, Daeyoung Choi, Yongjae Lee, and Wonjong Rhee. Basic enhancement strategies when using bayesian optimization for hyperparameter tuning of deep neural networks. IEEE Access, 552588-526088 2020.
[7] Martin Wistuba and Josif Grabocka. Few-shot Bayesian optimization with deep kernel surro- gates. In International Conference on Learning Representations, 2021.
[8] Matthias Feurer, Benjamin Letham, and Fytan Bakshy. Scalable meta-learning for Bayesian optimization using ranking-weighh Gaussian process ensembles. In AutoML Workshop at ICML, volume 7, 2018.
[9] Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task Bayesian optimization. In Advances in Neural Information Processing Systems, 2013.
[10] Dani Yogatama and Gideon Mann. Efficient transfer learning method for automatic hyperpa- rameter tuning. Artificial Intelligence and Statistics, 2014.
[11] Matthias Poloczek, Jialei Wang, and Peter Frazier. Multi-information source optimization. In Advances in Neural Information Processing Systems, 2017.
[12] Valerio Perrone, Rodolphe Jenatton, Matthias Seeger, and Cédric Archambeau. Scalable hyperparameter transfer learning. In Advances in Neural Information Processing Systems, pages 2018.
[13] Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, and Andreas Krause, PACOH: Bayes-optimal meta-learning with PAC-guarantees. In International Conference Machine Learning, pages 9116-9126. PMLR, 2021.
[14] Andreas Krause and Cheng Ong. Contextual Gaussian process bandit optimization. In Advances in Neural Information Processing Systems, 2011.
[15] Rémi Bardenet, Mátyás Brendel, Balázs Kégl, and Michele Sebag. Collaborative hyperparameter tuning. In International Conference on Machine Learning. 2013.
[16] Matthias Poloczek, Jialei Wang, and Peter Frazier. Warm starting Bayesian optimization. In Winter Simulation Conference (WSC). IEEE, 2016.
[17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems, pages 5998-6098000 2017.
[18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Association for Computational Linguistics, pages 4171-4186, 2019.
[19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image
11

---

recognition at scale. In International Conference on Learning Representations. OpenReview.net, 2021.
[20] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, volume 119, pages 1691-1703.
[21] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 1 (18(15)c2011223911 2021.
[22]
Ananthan Nambiar, Maeve Heflin, Simon I. Liu, Sergei Maslov, Mark Hopkins, and Anna Ritz. Transforming the language of life: Transformer neural networks for protein prediction tasks. Association for Computing Machinery, 2020.
[23] Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. CoRR, abs/2203.07$ 00,814,7722 2022.
[24] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr. Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.0333 2021.
[25] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems, pages 15084-15097, 2021.
[26] Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Marth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. generalist agent. arXiv preprint arXiv:2205.061755 2022.
[27] Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In AAAI Conference on Artificial Intelligence, 2015.
[28] Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, and Ion Stoica. Tune: A research platform for distributed model selection and training. CoRR, abs//807.005118. 2018.
[29] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01644 2020.
[30] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
[31] Michael Volpp, Lukas P. Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning in Bayesian optimization. In International Conference on Learning Representations. OpenReview.net, 2020.
[32] Matthias Feurer, Jost Springenberg, and Frank Hutter. Initializing Bayesian hyperparameter optimization via meta-learning In AAAI Conference Artificial Intelligence, 2015.
12

---

[33] Yutian Chen, Matthew W Hoffman, Sergio Gómez Colmenarejo. Misha Denil, Timothy P Lillicrap. Matt Botvinick, and Nando Freitas. Learning learn without gradient descent by gradient descent. In International Conference on Machine Learning, pages 748-756. PMLR, 2017.
[34] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Picter Abbeel. RIS-2S: Fast reinforcement learning via slow reinforcement learning. CoRR, abs/161 1.02779, 2016.
[35] Jane X. Wang. Zeb Kurth-Nelson, Hubert Soyer, Joel Z. Leibo, Dhruva Tirumala, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matt M. Botvinick. Learning to reinforcement learn. In CogSci, 2017.
[36] Mojtaba Valipour, Bowen You, Maysum Panju, and Ali Ghodsi. SymbolicGPT: A generative transformer model for symbolic regression. CoRR, abs/2106. 14131, 2021.
[37] Guillaume Lample and François Charton. Deep learning for symbolic mathematics. In Interna- tional Conference on Learning Representations, OpenReview.net, 2020.
[38] Vishesh Agarwal, Somak Aditya, and Navin Goyal. Analyzing the nuances of transform- ers' polynomial simplification abilities. 1st Mathematical Reasoning in General Artificial Intelligncee Workshop at ICLR, 2021.
[39] François Charton. Linear algebra with transformers. CoRR,
2021.
[40] Samuel Müller, Noah Hollmann, Sebastian Pineda-Arango, Josif Grabocka, and Frank Hutter. Transformers can do Bayesian inference. 2022.
[41] Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. arXiv preprint arXiv:2202.0556 05007, 2022.
[42] Nikhil Singh, Brandon Kates, Jeff Mentch, Anant Kharkar, Madeleine Udell, and Iddo Drori. Privileged zero-shot automl. CoRR, abs/2106.13744 13743, 2021.
[43] Pierre Bourhis, Juan L Reutter, Fernando Suárez, and Domagoj Vrgoë. Json: data model, query languages and schema specification. In Proceedings of the 36th ACM symposium on principles of database systems, pages 123-135, 2017.
[44] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71,
2018.
[45] Roman Garnett, Bayesian Optimization. Cambridge University Press, 2022.
[46] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104 (1):148-175, 2015.
[47] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
[48] Ouassim Ait ElHara, Konstantinos Varelas, Due Manh Nguyen, Tea Tusar, Dimo Brockhoff, Nikolaus Hansen, and Anne Auger. COCO: The large scale black-box optimization benchmark- ing labob-laggescale test suite. ArXiv, abs/1903.06636, 2019.
[49] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages
2019.
[50] Xin-She Yang and Suash Deb. Eagle strategy using Lévy walk and firefly algorithms for stochastic optimization. In Nature inspired cooperative strategies for optimization (NICSO 2010), pages 101-111. Springer, 2010.
[51] Ryan Turner, David Eriksson, Michael McCourt, Juha Kiili, Eero Laaksonen, Zhen Xu, and Isabelle Guyon. Bayesian optimization superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. In NeurIPS 2020 Competition and Demonstration Track, pages 3-26, 2021.
[52] Alexander Cowen-Rivers Wenlong Lyu, Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan Rhys Griffiths, Alexandre Max Maraval, Hao Jianye, Jun Wang, Jan Peters, et al. Hebo:
13

---

Pushing the limits of sample-efficient hyper-paramete parameter optimisation. Journal of Artificial Intelligence Research, 74: 14:1126--333 1349, 2022.
[53] Luke Metz, Niru Maheswaranathan, Ruoxi Sun, C Daniel Freeman, Ben Poole, and Jascha Sohl-Dickstein. Taskset: A dataset of optimization tasks. 2020.
[54] Mikhail Evchenko, Joaquin Vanschoren, Holger H Hoos, Mare Schoenauer, and Michèle Sebag. Frugal machine learning. arXiv preprint arXiv:2111.03731, 2021.
[55] Murray Rosenblatt. Remarks on multivariate transformation. The Annals of Mathematical Statistics, 23(3):470- - 472, 1952.
[56] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning. volume 4. Springer, 2006.
[57] Zi Wang, Beomjoon Kim, and Leslie Pack Kaelbling. Regret bounds for meta Bayesian optimization with unknown gaussian process prior. In Advances in Neural Information Processing Systems, 2018.
[58] Zi Wang. George E Dahl, Kevin Swersky, Chansoo Lee, Zelda Mariet, Zachary Nado, Justin Gilmer, Jasper Snock, and Zoubin Ghahramani. Pre-trained Gaussian processes for Bayesian optimization. arXiv preprint arXiv:2109.00 08215, 2022.
[59] James Bergstra, Dan Yamins, David D Cox, et al. Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms. In Proceedings of the 12th Python in science conference, volume 13, page 20, 2013.
[60] Daiyi Peng. Xuanyi Dong, Esteban Real, Mingxing Tan, Yifeng Lu, Gabriel Bender, Hanxiao Liu, Adam Kraft, Chen Liang, and Quoe Le. PyGlove: Symbolic programming for automated machine learning. Advances in Neural Information Processing Systems, 33:96-108, 2020.
[61] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information P Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[62] Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, and Nando de Freitas. Regu- larized behavior value estimation. arXiv preprint vrXiv:2103.000 09575, 2021.
[63] Jürgen Schmidhuber. Reinforcement learning upside down: Don't 't predict rewards- - just map them to actions. CoRR, abs/1912.0295 2019.
[64] Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. Advances in Neural Information Processing Systems, 34, 2021.
[65] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
[66] Daniel Golovin, John Karro, Greg Kochanski, Chansoo Lee, Xingyou Song, and Qiuyi (Richard) Zhang. Gradientless descent: High dineeniimesio zeroth-order optimization. In International Conference on Learning Representations. OpenReview.net, 2020.
[67] James Kennedy and Russell Eberhart. Particle swarm optimization. In ICNN'95 international conference on neural networks, volume 4, pages 1942-1948, 1995.
[68] Alexander I Cowen-Rivers, Wenlong Lyu, Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan Rhys Griffiths, Alexandre Max Maraval, Hao Jianye, Jun Wang, Jan Peters, al. An empirical study of assumptions in Bayesian optimisation. arXiv preprint arXiv:2012.03826, 2020.
[69] Elizabeth D Dolan and Jorge J Moré. Benchmarking optimization software with performance profiles. Mathematical Programming, 91(2):201-213, 2002.
[70] Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL, https://
openreview net/ Eorum? Heweenidd-HHxyyy H.dyHHXxxXBBDD.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the pa- per's contributions and scope? [Yes]] Abstract and introduction point out the specific contributions in bullet points, which we have covered throughout or methodology and experiments.
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] The main potential negative societal impacts of our work come from the inherent associated risks of using large Transformer models, such as bias. However, currently this does not apply to work, using language models as of yet. Such a risk may grow however, if in future works, we c.g. pretrained language models for warm horm-starting in our OPTFORMER framework. (d) Have you read the ethies review guidelines and ensured that your paper conforms to them? [Yes] The most relevant potential ethics concern is our use of RealWoridData. However, we mention in Appendix C, we anonymized the data to prevent users from being identified in the model.
data only contains numerical optimization data, and
not
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Not applicable; no theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] Not applicable; no theoretical results.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? (Yes] We included the link to the primary T5x codebase in Appendix D.2, along with relevant hyperpa- rameters. But we do not include the training data. The datasets generated on public benchmarks, BBOB and HPO-B, can be reproduced by running publicly available HPO algorithms.
(b) Did you specify all the training details (e.g., data splits, hyperparameters. how they were chosen)? [Yes] These are specified in Appendix A, Appendix C, and Appendix D.2. (c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes] In experiments, we plotted error bars using standard deviation over 5 runs.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix D.2.
4. If you are using existing assets code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We have cited [48] for our use of BBOB dataset, [5] for the HPO-B dataset, and [30] for training T5 models.
(b) Did you mention the license of the assets? [Yes] For data, we mentioned the use of Apache 2.0 License when using the HPO-B dataset.
(c) Did you include any new assets either in the supplemental material as URL? [Yes] We cited and linked the T5x codebase (see Appendix D.2) throughout this paper. We
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes] The BBOB dataset synthetic, and the HPO-B dataset public. For the Real WorldDaaa dataset, users by default are agreeing to the condition that their tuning data may be used for research purposes. We have also ensured anonymity over the data by removing personally identifiable information.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] As discussed in Appendix C, we anonymized user details from studies in the Real WoridData dataset.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] Not applicable.
15

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] Not applicable.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] Not applicable.
16


Reviews:
[{"review_id": "wQVTpqnC1Rr", "reviewer": "Reviewer_MJPS", "summary": "The paper describes a new meta-learning approach for hyperparameter optimization (HPO) based on a transformer model. The model is trained on offline generated data that includes the metadata that characterizes the optimization problem, for example the search space and  and the history of observed trials, i.e function values and input configuration. During inference time, the model can be combined with HPO policies, such as Thompson sampling or upper confidence bounds to suggest new hyperparameter configurations.", "questions": "- Section 6.2: How do you compute the predictive distribution p(y|...)? My understanding is that the transformer only predicts discrete outputs with [0, Q)\n\n- Section 6.4 prior policy: How is Random Search combined with Thompson sampling (Random Search-TS)?\n\n- What was the computational budget to train the transformer model and how long did it train?\n\n- Do you plan to open-source the dataset and the code to reproduce the results?", "limitations": "While the method improves across a set of baselines, it does not improve yet over more sophisticated algorithms such as Vizier on real world datasets. I assume that it would also not outperform current state-of-the-art methods that early stop poorly performing configurations, such as Hyperband or BOHB. However, I think this is fine for a research paper, but it would not be sufficient for production.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n### Reason for overall rating\n\nCurrent transfer learning approaches for HPO are limited to a fixed search space and the same underlying machine learning model and only transfer knowledge across different datasets. This paper presents, to the best of my knowledge, the first approach that enables meta-learning across these different dimensions. While I don't think the method is ready for a practice yet, the paper marks a first important step towards more universal HPO methods.\n\n\n### Strengths\n\nThe paper aims to learn a more general meta-learning approach for HPO, that generalizes not only across datasets, but also machine learning methods and search spaces. This, in theory,  allows to access a much large amount of offline data and allows to generalize across different domains.\n\nOverall, I found the different parts of the paper, e.g tokenization, inference and decoding of the model, well motivated and clearly explained.\n\n`The empirical evaluation of the paper contains sensible set of baselines. Also the ablation study provides convincing insights in the proposed approach.\n\n\n\n### Weaknesses\n\nIt remains a bit unclear how well this model generalizes beyond the training data. For example, what would happen if the method is applied to other problem domains, such as neural architecture search or general gradient-free optimization problems. Similarly, how does the method scale with the dimensionality of the search space?\n\nThe dataset generation seems a bit ad-hoc. Is it really necessary to include trajectories of such a large variety of optimizers or would it be sufficient to limit to few state-of-the-art optimizers? This could potentially reduce the dataset size and would allow to us a smaller architecture.\n\nThe paper could elaborate on the pre-training of the model. For example, how did different design decision of the network architecture effect downstream performance? How difficult was the pre-training, e.g did you have to restart from previous checkpoints, etc ? \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "sB1LLeGtRb", "reviewer": "Reviewer_mbxf", "summary": "The authors train a single Transformer language model (LM) on a variety of tokenized HPO trajectories from a variety of tasks and HPO algorithms. For each HPO trajectory the LM is primed on the task/algorithm (name, search space, metric, algorithm) and is then trained to predict hyper-parameters as well as the optimized metric step-by-step for each trial in order.", "questions": "- Is the discretization of scalars for the input important? How would a model perform, where these numbers are normalized in some way and fed to the network directly?\n- Do you have an explanation for the strong performance of using RealWorldData for training (Fig. 5 a) compared to the larger HPO-B dataset, when comparing on HPO-B? (Transfer from a smaller dataset is better than in-domain training. This is unusual.)\n- What do you mean by \"temporal train/test splits\" in line 269?\n- How do you calculate the Thompson Sampling utility function?", "limitations": "The listed limitations are fair, even though one limitation, I would expect there to be, is missing: Handling much longer sequences, as the Transformer is trained with a maximum sequence length.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The results are strong: Reproducing the performance of most algorithms up to 100 trials as well as improving upon them. This could be the path forward for the HPO community.\n- The authors provide insightful ablations.\n- The method is clearly described and simple.\n\nWeaknesses\n- This paper does neither open-source its codebase (built open an open-source codebase), nor the trained model and not even the training data (built upon open datasets). Actually, even a performance number for optimization for follow-up work to compare with is missing, as scores are calculated using an undisclosed metric and the main results are only reported in the form of plots and not tables. \n- For RealWorldData (and HPO-B actually, even though less interesting there) a SOTA BO method would be an interesting baseline to add, like HEBO.\n- Is the conclusion of Meta-data ablation (line 287) based on a model trained with meta-data? In that case, I would guess the worse performance stems from a train/test distribution shift, rather than from missing metadata really.\n\n\nSummary:\nThe results are strong with some evaluation problems. It is not reproducible, though, which in this case is a particularly big problem, as this paper proposes a very new direction for a field of research in which the expertise to reproduce the results based only on descriptions (previous methods require a very different background) and the resources to reproduce the results without data (running HPO with different optimizers on millions of problems) are missing. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YvpCk0M4lId", "reviewer": "Reviewer_yjnm", "summary": "The paper studies hyperparameter transfer with metadata in an \"open-set\" setting -- allowing different configuration spaces across tasks. A transformer-based hyperparameter tuner, namely OptFromer, is proposed to predict policy and response function values (e.g., validation performance) in a sequence-to-sequence training style, where the learned policy maps text-based metadata to pre-discretized hyperparameter configurations. To my best knowledge, the proposed method is the first HPO method that learns prior knowledge from the collected text-based configurations. Experimental results on one collected real-world dataset and two public benchmarks were provided in terms of the policy behavior imitation, response function prediction, and HPO. ", "questions": "- Table 4 shows more calibrated results of OptFormer than the GP-based methods. It is well known that GP could provide well-calibrated uncertainty estimates. Yet, in accordance with Eq (3-5), it seems like the proposed method just follows the standard supervised training strategy. Any insights into why OptFormer could show a better calibration result? Is the reason due to using the transformer architectures [40]? I'm also curious about the ECE comparison between OptFormer and OptFormer (TS)\n- As shown in Fig. 4, the Vizier performs slightly better than OptFormer (TS) on the RealWoldData dataset. While the paper implies the reason as the GP surrogate-based test functions, it is unclear why OptFormer performs much better than GP-UCB. Also, the comparison results between RealWorldData (*mixed algorithms*) and HPO-B (*controlled algorithms*) cast a shadow on the biased issue of the proposed OptFormer. \n\n**Post after rebuttal**\nThanks for providing a detailed response to the questions. Due to my travel schedule, I haven't got a chance to further discuss with the authors. Yet, most of my previous concerns were well addressed. Particularly, it would be interesting to add an LSTM baseline in future work and explore more the calibration of HPO from a pre-training perspective. I would like to champion this work by upgrading my score. ", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- The proposed hyperparameter prior learning method is orthogonal to existing GP-based hyperparameter transfer methods in two folds: 1) it relaxes the limitation of sharing the same configuration space across different tasks, and 2) the data-driven approach given by training a transformer model significantly improves the efficiency. \n- It is interesting and novel to learn prior knowledge from text-based metadata. The seq-2-seq supervised learning framework is technically sounded with proper practical treatments. Moreover, the transformer structure is well-motivated to capture both symbolic and numerical manipulation.\n- While some necessary implementation details are missing in the manuscript, the augmented HPO policy with Thompson Sampling provides a good implementation similar to offline RL. \n- Extensive experimental results demonstrate the effectiveness of the learned HPO policy on its well-calibrated predictions and utility performance. \n\nCons:\n- The main concern of this work is its unclear **meta knowledge** learned from text-based metadata. How does the proposed OptFormer indeed imitate the other HPO algorithms? Does the transformer simply memorize the choices given by different HPO algorithms? Can the proposed method adapt to more complex algorithms (e.g., hypergradient-based) and large-scale hyperparameters? \n- The learned prior seems risky to be biased on closed-set model architectures and tasks (datasets). It remains unclear if the HPO policy can be generalized to **unseen** tasks. I may miss something in the appendix; yet, it will be helpful to give more details about the split of training/test set in the RealWorldData. A non-overlapping split over the tasks or algorithms will be more convincing. \n- One major technical contribution of this work is to introduce transformers for learning HPO priors. Hence, it is expected to give an ablation study regarding this architecture choice. One baseline based on RNN (e.g., LSTMs or GRUs) will be useful to validate this point empirically. \n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AqzpBG-8MR", "reviewer": "Reviewer_C6k8", "summary": "The authors propose using Transformer to imitate the hyper-parameter optimization. The authors claim this is the first work in the area, and claim the proposed method exceeds several classic HPO methods.", "questions": "1. how do you justify max_i\\in{1:t}(y_i-y_rand)/(y_max-y_rand) a good evaluation metric? Why don't you use the y_i and plot as a range? \n\n3. Fig.4 made a few strong claims, especially that OPT-Former performs better than GP-UCB. I'm trying to understand the underly causes. Here is my guess: opt-former is trained on datasets that potentially contain tasks with similar distributions tested in Fig.4. The advantage of GP-UCB is to start without any priors and gradually approximate the underlying function contours by sampling. Without any prior data, I'm surprised that OPT-Former can perform better than GP-UCB. I'd be happy to see if I'm wrong, and it will be compelling if the authors can provide an anonymous link to the code for a quick comparisons. (key factor for me to improve the score)\n\n4. The methods used in this paper does not fully capture the landscape of black box optimization solvers today. The authors may find the following repo to be useful. (feel free use at your discretion, it is just a suggestion)\n\na. https://botorch.org/\n\nb. https://github.com/facebookresearch/nevergrad\n\nc. https://github.com/facebookresearch/LaMCTS\n\nd. https://github.com/uber-research/TuRBO\nThese repos have encapsulate several exciting BBO algorithms today.\n", "limitations": "no limitation found", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. very comprehensive evaluations.\n2. seemly reasonable idea.\n\nWeakness:\n1. limited novelty: perhaps applying transformer to HPO could be counted as a novel point, but I feel this is not enough.\n2. The paper is difficult to read, please try to improve the readability. Many places in the paper tries to impress the readers with sophisticated terms even on very simple concepts.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept