Zero-shot Transfer Learning for Gray-box Hyper-parameter Optimization
Authors: Hadi Samer Jomaa
Lars Schmidt-Thieme
Josif Grabocka
Venue:  (2021)
Keywords: Hyper-parameter Optimization
Transfer Learning
Meta-learning

Abstract: Zero-shot hyper-parameter optimization refers to the process of selecting hyper- parameter configurations that are expected to perform well for a given dataset upfront, without access to any observations of the losses of the target response. Existing zero-shot approaches are posed as initialization strategies for Bayesian Optimization and they often rely on engineered meta-features to measure dataset similarity, operating under the assumption that the responses of similar datasets behaves similarly with respect to the same hyper-parameters. Solutions for zero- shot HPO are embarrassingly parallelizable and thus can reduce vastly the required wallclock time of learning a single model. We propose a very simple HPO model called Gray-box Zero(0)-Shot Initialization (GROSI) as a conditional parametric surrogate that learns a universal response model by exploiting the relationship between the hyper-parameters and the dataset meta-features directly. In contrast to existing HPO solutions, we achieve transfer of knowledge without engineered meta- features, but rather through a shared model that is trained simultaneously across all datasets. We design and optimize a novel loss function that allows us to regress from the dataset/hyper-parameter pair unto the response. Experiments on 120 datasets demonstrate the strong performance of GROSI, compared to conventional initialization strategies. We also show that by fine-tuning GROSI to the target dataset, we can outperform state-of-the-art sequential HPO algorithms.

Under review as a conference paper at ICLR 2021
ZERO-SHOT TRANSFER LEARNING FOR GRAY-BOX HYPER-PARAMETER OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Zero-shot hyper-parametts optimization refers to the process of selecting hyper- parameter configurations that are expected to perform well for given dataset upfront, without access to any observations of the losses of the target response. Existing zero-shot approaches are posed as initialization strategies for Bayesian Optimization and they often rely engineered meta-feettures to measure dataset similarity, operating under the assumption that the responses of similar datasets behaves similarly with respect to the same hyper-paraman pareerrrrrmmerss... Solutions for zero- shot HPO are embarrassingly parallelizable and thus can reduce vastly the required wallclock time of learning single model. We propose very simple HPO model called Gray-box Zero(O)-Shon Initialization (GROSI) as conditional parametric surrogate that learns a universal response model by exploiting the relationship between the existing HPO solutions, we achieve transfer of knowledge without engineered meta- features, but rather through a shared model that is trained simultaneously across all datasets. We design and optimize a novel loss function that allows us regress from the dataset/hyper-parcee datasets demonstrate the strong performance of GROSI, compared to conventional initialization strategies. We also show that by fine-tuning GROSI to the target dataset, we can outperform state-of ofr sequential HPO algorithms.
and the dataset meta-features directly. In contrast to
pair unto the response. Experiments on 120
INTRODUCTION
Within the research community, the concentration of efforts towards solving the problem of hyper- parameter optimization (HPO) has been mainly through sequential model-based optimization (SMBO), i.c. iteratively fitting probabilistic response model, typically a Gaussian process (Ras- mussen (2003), ), to a history of observations of losses of the target response, and suggesting the next prree-pprameters via a policy, acquisition function, that balances exploration and exploitation by leveraging the uncertainty in the posterior distribution (Jones et al. (1998); Wistuba et al. (2018); Snoek et al. (2122). However, even when solutions are defined in conjunction with transfer learning techniques (Bardenet et al. (2013); Wistuba al. (2016); Feurer et al. (2015)) the performance of SMBO is heavily affected by the choice of the initial hyper- hyper-paramen parameters. Furthermore. SMBO is sequential by design and additional acceleration by parallelization is not possible.
In this paper, we present the problem of zero-shot parammerrreeerr optimization as a meta-learning objective that exploits dataset information part of the surrogate model. Instead of treating HPO as a black-bor box function, operating blindly on the response of the it gray-box function (Whitley al. (016)). by capturing the relationship among the dataset meta-features and hyper-parameters to approximate the response model.
alone, we treat
In this paper, we propose novel formulation of HPO as a conditional gray-box function optimization problem, Section 4 that allows us to regress from the dataset/hyper-paraane parammmeterr pair directly onto the response. Driven by the assumption that similar datasets should have similar response approximations, we introduce additional data-driven similarity regularization objective to penalize the difference between the predicted response of similar datasets. In Section 5, we perform an extensive battery of experiments that highlight the capacity of our universal model to serve solution for: (1) zero-shot HPO as a sund-alone task, (2) zero-shot as an initialization strategy for Bayesian Optimization (BO), (3) transferable sequential model-basee optimization. A summary of our contributions is:

---

Under review as a conference paper at ICLR 2021
formulation of the zero-shot hhper- optimization problem in which response model predicts upfront the full set of to observations of losses of the target response;
configurations to try, without access
novel multi-task optimization objective that models the inherent similarity between datasets and their respective responses;
three new meta-datasets with different search spaces and cardinalities to facilitate the experiments and serve as a benchmark for future work;
an empirical demonstration of the performance of our approach through a battery of ex- periments that address the aforementioned research aspects, and comparison against state-of-the-arr HPO solutions for transfer-leamning.
RELATED WORK
The straightforward zero-shot approaches for HPO consist of random search (Bergstra & Bengio
(2012)) ) or simply selecting syper-parameters that perform well on general tasks (Brazdil al. (2003) ). Some recent work has also shown that simply selecting random hyper-parameee
from restricted search space significantly outperforms existing solutions, and improves the performance of conventional SMBO approaches (Perrone et al. (2019)). The restricted search space is created by eliminating regions that are further away from the best
of the training tasks.
Another prominent direction for zero-shot HPO depends heavily engineered meta-features, i... dataset characteristics (Vanschoren (2018), ) to measure the similarity of datasets. Following the assumption that the responses of similar datasets behave similarly the hyper-parameters, it has been shown that even the simplest of meta-features (Bardenet et al.] (2013) improve the performance of single task BO algorithms (Feurer al. (2014) 2015)). The target response is initialized with the top-performing hyper-parameters of the dataset nearest neighbor in the meta-feature space. The shortcomings of using engineered meta-feature, are that they are hard define (Leite Brazdil (2005)), and are often selected through rial-and-error or expert domain knowledge. As remedy, replacing engineered meta-feetures with learned features (Jomaa al. (2019)) compensates for such limitations, by producing expressive meta-features agnostic to any meta-task, such as HPO.
Zero-shot HPO also posed as an optimization problem that aims to minimize the meta-loss over a collection of datasets (Wistuba et al. (2015)) by replacing the discrete minimum function with a differentiable softmin function as an approximation. The initial configurations boost the single task BO without any meta-features. In (Wistuba et al. (2015b)), hyper-paraan appre-prrreeter combinations assigned a static ranking based on the cumulative average normalized error, and dataset similarity is estimated based the relative ranking of these combinations. Winkellmler et al. (2020) introduce a Bayesian Optimization solution for zero-shot HPO by iteratively fitting a surrogate model over the observed responses of different tasks, and selecting the next paprer--rraettrrs and datasets that minimize the aggregated observed loss.
Aside from zero-shot HPO, transfer learning is employed by learning better response models (Wistubal cet al. (2016)) based on the similarity of the response. Feurer et al. (2118) propose an ensemble model weighted sum of the predictions of base models
for BO by building the target response model as well as the target model. In addition to the transferable response models, Volpp et al. (2019) design transferable acquisition function as a policy for reinforcement learning framework. As a replacement to the standard Gaussian process. Perrone et al. (2018) train multi-task adaptive Bayesian linear regression model with a shared feature extractor that provides context information for each independent task.
optimization defind in
In contrast to the literature, we formulate the problem of zero-shot HPO as a gray-box function optimization problem, by designing universal response model defined over the combined domain of datasets and hyper-parameters. We rely on the embeddings to estimate the similarities across datasets and design novel multi-task optimization objective regress directly on the response. This allows us to delineate from the complexity paired with Bayesian uncertainty, as well as the trouble of engineering similarity measures.
2

---

Under review as a conference paper at ICLR 2021
3 HYPER-PARAMETER OPTIMIZATION
Consider dataset D (x((rrain) y (liiii) (I. _y (I( _f } for supervised learning task, with training, validation and test splits of predictors I E X and targets E V. We aim at training a parametric approximation of the target using 1 : f X) + where 0 θ denotes the parameters and À E A its aaper-parameters, by minimizing loss function L ) x y - R as:
( X = min L (y (z 0", ()) s.t. 0" = arg min & (y (s:(Trim), 0, 2)) (1) 0EG
AEA
We hereafter denote the validation
as the response E (() L (y) (z:Val); A)) Un- fortunately, direct optimization of the response (A) in terms of is not trivial, because 0" is the result of the minimization problem and its gradients with respect to À are not easy to com- pute. Instead, in order to learn the optimal E(>; B)
À train probabilistic surrogate B R parameterized by B, with B as the space of response model parameters, that minimizes the log- likeliioiiood of approximating the response {(A) over a set of К evaluations : (A1 e (A1)) ..... (AK, , l (Ak)). We denote given surrogate model. Given the surrogate, the next hyper-parameter to be evaluated ( is computed by maximizing an acquisition function A (e.g. El (Močkus 1975)) as:
the probability of estimating the response
K In (E(AE), ((Ai:))
(next) arg max A(((A;3*)) s.t. 8 arg min BEB
Σ
4 META-LEARNING OF CONDITIONAL GRAY-BOX SURROGATES
Let us define collection of T datasets {DW)..... ,..., D(2) and let g(t) (A) measure the response of the ayper-parameter on the t-th dataset D Furthermore, assume we have previously evaluated (())) many hyper-parameters ( h E (1...., k(O)) that particular dataset. We condition the surrogate to capture the characteristies of the t-th dataset, by taking as input the meta-features representation of the dataset d(()) Therefore, dataset-aware surrogate be trained using meta-learning over cumulative objective function O (B) as:
KCG
O (3)
ΣΣ
(00) (210) - 8 000; 8))²
(3)
4.1 THE
EXTRACTOR
Introducing engineered meta-features has had significant impact on hyper-parament optimization. However, learning meta-features across datasets of varying schema in task-agnostic setting provides more representative characteristics than to rely on hard-to-tune empirical estimates. The meta-feature extractor is a set-based function (Zaheer et al. (2017)) that presents itself as extended derivation of the Bolmogorov-Arnol representation theorem (Krková (1992)), which states that a multi-variate aggregation of univariate functions single variables, Appendix B
function á can be defined
Each supervised (tabular) dataset (()) = (z(1),)) (0) consists of instances (+() E X' E N and targets (())) e E NxC such that N, M and C represent the number of instances, predictors and targets respectively. The dataset can be further represented as set of smaller components, set of (1)
sets, (*) = a: ( v/.c) m (1....,M) M). E (1, (1,..., N), (I,...,C) A tabular dataset composed of columns (predictors, targets) and rows (instances) is reduced to single predictor-tait target pairs instead of an instance-target pairs. Based on this representation, a meta- meta-feature extractor parameterized neural network (Jomaa al. (2019)) is formulated in Equation 4 For simplicity
3

---

Under review as a conference paper at ICLR 2021
of notation, we drop the superscript (t) unless needed.
M Σ m=
C
N Σ N
(DD) h
[ Vi,e)
(4)
MC
im1
with f R22 -K, RK, - TRKK and h RKK RK represented by neural networks with y, Kg. and K output units, respectively. This set-based formulation captures the correlation between each variable (predictor) and its assigned target and permutation-invariant i.c. the output is unaffected by the ordering of the pairs in the set. Other set-based functions such (Edwards) Storkey 2016); Lee et al. (2019)) can also be used for meta-fe extraction, however, we focus on this deep-set formulation (Jomaa et al. (2019) because it is proven to work properly for hyper-parameter optimization.
4.2 THE AUXILIARY DATASET IDENTIFICATION TASK
The dataset identification task introduced previously as dataset similarity learning (Jomaa al. (019)), ) ensures that the meta-features of similar datasets are colocated in the meta-feature space, providing more expressive and distinct meta-features for every dataset.
Let pD a joint distribution over dataset pairs such that (D(t), D(9), 8) E Tx Tx (0,1) with being a binary dataset similarity indicator. We define a classification model $ x + R that provides an unnormalized probability estimate for being 1, follows:
((D("), D(9)) - ( (6tt) (g)))
(5) For
where : IR* I + Rt represents any distance metric, and a tuneable
simplicity, use the Euclidean distance to measure the similarity between the extracted meta- features, i.e. Z (o)()) ,(())) ||()()) ((9)]|, and set 7 = 1. The classification model is trained by
optimizing the negative log likelihood:
P(3) =
Σ
log (s(D((), ((o))
Σ
log (1 (I-)(D(),(()))
(6)
(1.))PP=
(1.))~PD-
with Po+ as the distribution of similar datasets, PD+ = ((D(t, D(1) ((),s) ~ Pp | s= 1}, and PD- as the distribution of dissimilar datasets, PD- = ((D(0), D(1), 8) ~ pn = 0}. Similar datasets are defined as multi-fidelity subsets (batches) of each dataset.
4.3 DATA-DRIVEN SIMILARITY REGULARIZATION
Our surrogate differs from prior practices, because we do not consider the response to be entirely black-box. Instead, since we know the features and the target values of dataset even before evaluating any hyper-parameter we model gray-box surrogate by exploiting the dataset characteristies & when approximating the response f. As result, the surrogate faces a new dataset that is similar to one of the 7 datasets from the collection it was optimized (i.e. similar meta- features & extracted directly from the dataset), it will estimate a similar response. Yet, we know apriori that two datasets are similar by means of the distance of their meta-features, we can explicitly regularize the surrogate to produce similar response estimations for such similar datasets, as:
T-1
T
(B) = Σ
Σ
(000 - -600) (2(21) ()- e2 .000 8))²
-)
1=1 g=1+1
Overall train the surrogate model to estimate the collection of response evaluations and explicitly capture the dataset similarity by solving the following problem, Equation R, end-to-end, where R controls the amount of similarity regularization, and 5 IR controls the impact of the dataset identification task:
8" arg min
R (8) (
(8)
Beb
4

---

Under review as a conference paper at ICLR 2021
NETWORK ARCHITECTURE
Our model architecture is divided into two modules, i= & o th the meta-feature extractor c, and the regression head vb. The meta-feature extractor á IR IR 1R composed of three functions, and h. The regression head is also composed of two functions, i.e. sb #1 00/2- We define by 1/1 : Kv x A - R kw as the function that takes as input the meta-feeture/hyper parameter pair, and by 2/2 R + E the function that approximates the response. Finally, let Dense(n) define one fully connected layer with n neurons, and ResidualBlock(n,n) be mx Dense(n) with residual connections (Zagoruyko & Komodakis (2016). We select the architecture presented in Table | based on the best observed average performance on the held-out validation sets across all meta-datasets, Appendix E.1
Equation[] namely f.
Table I: The network architecture optimized for every meta-dataset.
Functions
Architecture
f
Dense(16)66 Dense((6)xxxxReiiuuuBBbBll
g
Layout: t> with layers and 16 Neurons
h
(ense((1)):34) ResidualBlock(3. (adddduaBBocc((31)))Deeeee De):se 16)
w/
Layout: t- with layers and 4 Neurons
f/2
Layout: b- with layers and 16 Neurons
5 EXPERIMENTS
Our experiments are designed to answer three research questions]
Q1: Can we learn a universal response model that provides useful hyper-parameter initial- izations from unseen datasets without access previous observations of typer-parameters for the dataset itself?
Q2: Do the proposed suggestions serve as a good initialization strategy for existing SMBO algorithms?
Q3: Aside from zero-shot HPO, does the performance of our method improve by refitting the response model the observations of the syper-parameters for the target dataset and how well does approach compare to state-of- the-art methods in transfer learning for !PO
5.1 TRAINING PROTOCOL.
In Algorithm I we describe the pseudo-code for optimizing our response model via standard meta- learning optimization routines. We use stochastic gradient descent to optimize the internal model, and Adam optimizer (Kingma Ba (2015)) optimize the outer loop. We set the number of inner iterations to v = 5, and use a learning rate of 0.001 for both optimizers. We use batch size of 8 tasks sampled randomly with each iteration. The code implemented (2016)). The performance of the various optimizers is assessed by measuring the regret, which represents the distance between an observed response and the optimal response on a response surface. For hyper-paramete optimization, the meta-datasets optimal response is known. Since we normalize the response surfaces between (0, 1). we observe the normalized regret. The reported results represent the average over 5-fold cross-validation split for each meta-dataset, with 80 meta-train, 16 meta-valid, and 24 meta-test sets, and one unit standard deviation.
Tensoflow (Abadi et al.
provided beforehand, consequently, the
5.2
We create three meta-datasets by using 120 datasets chosen from the UCI repository (Asuncion) Newman 22077)) We then create the meta-instances by training a feedforward neural network
For a better understanding of the different problem settings, see Appendix A
The associated code and meta-dataset described will be available upon acceptance.
5

---

Under review as a conference paper at ICLR 2021
and report the validation accuracy. Each dataset is provided with predefined split 60% train, 15% validation, and 25% test instances. We train each configuration for 50 epochs with a learning rate of 0.001. The hyper-pes parameee search space is described in Table 2
Table 2:
Hyper- pprem-peremttrr search space for the meta-dataseti, The name of each the meta data-datasets is inspired by the most prominent hyper-parameter, highlighted in red.
Hyper parammerreeter
Layout Md
Regularization Md
Optimization Md
Activation
ReLU, SeLU 4, 8, 16.32
ReLU, SeLU, LeakyReLU
ReLU, SeLU, LeakyReLLU
Neurons
4,8, 16,32 1,3,5,7
4,8,
Layers
1,3,5,7
3,5,7
Layout
1,4.9,9,A
<,P,0,A
Dropout
0.0.5
0,0.2,
Normalization
False
False, True
False
Optimizer
ADAM
ADAM
ADAM, RMSProp, GD
The layout hyper-pane parem-rreetter (Jomaa et al. (2019)) corresponds the overall shape of the neural network, and provides information regarding the number of neurons in each layer. For example, all the layers in the neural network with a layout share the same number of neurons. We introduce an additional layout, A, where the number of neurons in each layer is successively halved until it reaches the corresponding number of neurons in the centeral layer, then doubles successively. We also use dropout (Srivastava et al. (2014)) and batch normalization (loffe & Szegedy (2015) as regularization strategies, and stochastic gradient descent (GD), ADAM (Kingma Ba 2015)) and
RMSProp (Tieleman & Hinton (2012) as optimizers. SeLU (Klambauer al. (2017) represents the self- self-normalizig activation unit. The search space consists of all possible combinations of the hyper-parameteryy After removing redundant configurations, the resulting meta-datasets have 256, 288 and 324 unique configurations respectively, For the purposes of our algorithm, we need access to the datasets used to generate the meta-features Further details are available in Appendix C
5.3 BASELINES
We introduce two sets of baselines to evaluate against the different aspects of our approach:
ZERO-SHOT SYPPER-PARAMETRR OPTIMIZATION
Random search (Bergstra & Bengio (2012) is the simplest approach where the hyper- parameters are selected randomly.
Average Rank represents the top hyper- parpme-rraeeters that had average the highest-ranking across the meta-train datasets.
IN-MMETAFEATURE) (Feurer al. (2015) refers to the process of selecting the top- performing hyper-parameteen of the nearest neighboring dataset based on their meta- features. We use two sets of well-established engineered meta-features, which we refer to as MF1 Feurer et al. (2155) and MF2 (Wistuba et al. (2016), as well as learned meta- features (Jomaa et 2019)), which we denote by D2V. The similarity measured by the Euclidean distance.
Ellipsoid (Perrone et al. 2019)) is also a random search approach, however the hyper- parameters are sampled from a hyper-ellipsoid search space that is restricted to encompass as many optimal hyper-parameter from the training dataset as possible.
SEQUEETIAL-MMDEEL BASED OPTIMIZATION FOR TRANSFER LEARNING
GP [Rasmussen 2003)) is standard Gaussian process response model with a Matern 3/2 and automatic relevance determination. This approach is trained independently each dataset.
UUnfortunttll. we could not evaluate our approach some of the published meta- meta-datasets (Schilling al. (2016)) due to the unavailability of the associated datasets (original predictors and target values) used for generation of the meta-instances

---

Under review as a conference paper at ICLR 2021
SMFO (Wistuba et al. (2015b)) is sequential model-free approach that provides collection of hyper-parameters by minimizing the ranking loss across all the tasks in the meta-train datasets.
TST-R (Wistuba et al. (2016) is two-stage approach where the parameters of the target response model are adjusted via kernel-weighted average based on the similarity of the syper-paramette response between the target dataset and the training datasets. We also evaluate the variant this approach that relies on meta- features, by replacing the engineered meta-features with learned meta-features, TST-D2V.
RGPE [Feurer et al. (0118) is ensemble model that estimates the target response model as a weighted combination the training datasets' response models and the target itself. The weights assigned based on ranking loss of the respective model.
ABLR (Perrone al. (2018) is multi-task ensemble of adaptive Bayesian linear regression models with all the tasks sharing a common feature extractor.
TAF-R (Wistuba et al. (2018) learns a transferable acquisition function, unlike the afore- mentioned algorithms that focus on a transferable response model, that selects the next haper-parameter based on a weighted combination the expected improvement of the target task, and predicted improvement the source tasks.
MetaBO (Volpp et al. (2019) is another transferable acquisition function, optimized as a policy reinforcement learning framework. This approach, however, demands pre-computed target response model as part of the state representation.
In our approach, we learn universal response model based on the underlying assumption that the response is not only dependent on the hyper- parameerrs,,s,, as is assumed in black-box optimization techniques, but also on the dataset itself, presenting the problem as a gray-box function optimization.
5.4 RESULTS AND DISCUSSION
Q1: ZERO-SHOT HPO AS A STAND-ALONE PROBLEM
In Table 3 we report the final normalized regret achieved by the different zero-shot approaches for the first hyper-parameters (Feurer et al. (2015)) ). Our method provides dataset-conditioned hyper- parameters that perform better than heuristies for small budget: The use of engineered meta-featee ffaturess to represent datasets for HPO solutions is not reliable, as the results achieved by NN-(MFI) and NN-(MF2) no better than random. On the other hand, using the meta-features extracted from the dataset directly, (N-(D2V)serve)) as a better approximation. Furthermore, random sampling from the restricted hyper-ellipsoid also outperforms the of initialization strategies based meta-features. We obtain the zero-shot hyper-parameters via Algorithm 2 Algorithm
The D2V meta-features are obtained via
Table 3: Results on several zero-shot HPO benchmarks. The numbers reported are the average normalized regret for 120 tasks on each meta-datasee evaluated as the average of 5-fold cross- validation scheme. We report the best results in bold and underline the second best.
Zero-shot 685
Zero-shot @ 20
Method
Layout Md
Regularization Md
Optimization
Layout Md
Regularization Md
Optimization
Random
13.988
2.629
15.105 :: 1.550
12,836 + 1.781
6.900 + 1.274
8.909 :: 0.440
7.185 +: 0.738
Average Rank
13.103
2.679
11.339 + 1.216
11.843 + 1.647
6.660 + 1.947
5.362 + 0.772
7.128 1.88
MN-MII
13.544
1.999
12.970
12.300
7,430
7.542
0.809
7.164 1.404
NN-MF2
2.181 14.350
12.137
1.008
12.717 2228
7.621 + 1.025
6.788 +: 1.122
8.055 4: 1.868
NN-D2V
12.278
2,400
13.831
+ 3.556
11.568 : 0.671
6.526 + 2,215
6.820
+ 1.992
6.695 + 0.942
Ellipsoid
11.114
10.713
5.713
1.776
6.665
GROSI
10.945
1.602
10.779
1.626
11.298
1.086
5.707 + 1.539
5.903
1.655
6.242 1.223
We depict the surrogate with the response model in P2 plausibility check.
7

---

Under review as a conference paper at ICLR 2021
Q2: ZHOO-SHOT HPO AS AN INITIALIZATION STRATEGY FOR SINGLE-TASK SEQUENTIAL HPO METHODS
We use the aforementioned initialization strategies to warm-start single task GP with Matern 3/2 kernel and automatic relevance determination as the response model. The quality of our suggested hyper-parameterd is reflected in the improved performance of the response model at the early stages The pseudo-code for
compared to metafeature-basse initialization and random search, Figure 1 sequential model-based optimization is provided by Algorithm
Layout
Regularization Md
Optimization Md
7
7
Normelized Regrett
4
4
3
2
2
2
0
40 Trials
60
0
20
40
60
20
40
60
Trials
Trials
GP-MF1
GP-MF2
GP-DZV
GP-GROSI
Figure 1: Average normalized regret for single-task sequential HPO methods, Gaussian process, with different initialization strategies.
Q3: SEQUENTIAL GRAY-BOX FUNCTION OPTIMIZATION
The proposed universal response model provides useful Hyper-parameters upfront without access to any observations of losses of the target responses. However, by iteratively refitting the model to the history observations, the response prediction is improved, as depicted Figure and summarized in Table 4 We refit our algorithm by optimizing Equation
the history of observed losses the after refitting. (1) greedily selecting the hyper-parametee with the highest predicted response, GROSI(+ I), and (2) selecting the next hyper-paramett
target dataset, Algorithm 4 We evaluate two policies for selecting the next
randomly from the top 5 hyper-parameter with the highest predicted response, GROSI(+10), which achieved the best regret average across the three meta-datasets, Appendix E.2 In contrast to the baselines that select acquisition function that capitalizes on the uncertainty of the posterior samples, incorporate uncertainty by selecting the next random and thus introduce small trade-off between exploration and exploitation.
through an
from the top-k hyper-parameter parameters uniformly at
Furthermore, our method outperforms the state-of-the-ar in transfer learning approaches for HPO in several cases while demonstrating in general competitive performance across all three meta-datasets. Table The baselines are warm-started with 20 randomly selected hyper-parameten (Feurer et al. (0015).
For better readability, the uncertainty quantification can found Figure
Layout
Regularization Md
Optimization Md
Normaizeed Rogres
4
3
Number of Trials
Number Trials
Number Trials
TST-R
TST-D2V
RGPE
TAF-R
ABLR
GROSI
GROSK(+10)
Figure 2: Average normalized regret for state-of-the-ae art transfer learning HPO methods.
For better visualization, some haselines are removed from Figure 2 but are still reported in Table
8

---

Under review as a conference paper at ICLR 2021
Table 4: Normalized regret for state-of-the-ar transfer learning HPO methods for up 80 trials after initialization with 20 configurations.
@ 50 after initialization
@80 after initialization
Method
Layout Md
Regularization Md
Optimization Md
Layout Md
Regularization Md
Optimization Md
SMFO
0.216
1.498
0.572
1.166
1.183 0.399
1.822 0,873
TST-R
1,982 +: 0.329
1,879 + 151
2.611 + 0.822
1.168 : 0.275
1.070 0.989
1.947 = 0.492
TST-D2V
1.921 +: 0.880
1.444 + 0.698
2.747 0.731
0.877 = 0.501
0.942 = 0.592
2.126 = 1.037
RGPE
4,069 +:
4,706 + 0.935
5.000 + 0.572
0.503
4,429 0.742
4.736+t 0.799
TAF-R
4.270 +: 0.805
3,969 +: 0.545
2.714 1.013
4.131 1,070
3.969 = 0.545
2,059:t 0.757
ABLR
2.770 +: 0.583
3.140 + 1.654
3.775 + 1.839
2.267 : 0.549
1.824 = 1.175
2.299 0.799
MetaBO
7.422 +:
7.145
1.222
6.468 0.921
7,422
7.127 1.247
1.036
GROSI
2.313 +: 0.828
1,621 +: 0.491
3.397 0.95
1.274
0,906
2,806 0.969
1
1,599 + 0.399
1,637 + 0.752
2.598 + 1,369
1,118 + 0.635
1,000 = 0.613
1.871 = 1.208
GROSI(+10)
1887 + 0.546
1,631 + 0.706
2.557 + 1.035
0.956 0.658
0.922 = 0.308
1,950 = 0.911
5.5 ABLATION STUDY
We perform several ablation experiments to analyze the contribution of each objective to the overall performance. The results are detailed in Table Treating zero-shot HPO as simple regression model by optimizing Equation 3 alone is suboptimal and does not scale across all meta-datasets.
We notice that adding the auxiliary dataset identification task, Equation 6 brings on significant improvement, similarly with the similarity driven regularization, Equation 7 This reinforces the notion that the responses of similar datasets behave similarly with regards to the hyper-parameters. Both losses help generate more expressive meta-features, the former more directly, by optimizing the inter- and intra-datase similarities, and the latter indirectly by penalizing the difference in the predicted response.
Table 5: Final results for zero-shot HPO for different variations of our model optimized with different objectives. The numbers reported are the average normalized regret after 20 trials.
Method
Layout Md
Regularization Md
Optimization Md
7,263 1 1.309
5.961 1. 1.841
7.625
0.546
RR
7.121 = 1.133
6.217
+ 2,369
6.488 = 0.609
6.139 = 1.654
6.105 1.765
6.332 1.516
Pretrained Meta-feature Extractor 6 o.
6.698 = 1.099
5.887
0.902
7.152 = 1,287
oR
6.516 0.911
5.650
1.678
9.925
7.130
GROSI
5.707 = 1.539
5.903 1.655
6.242 1.223
We also initialize the meta-feature extractor, o, by pretraining it independently, Algorithm However, we notice that this leads to generally poor performance as the model arrives quickly at local optimum. An artifact of the meta-dataset, we notice that pretraining GROST for Regularization Md provides small lift. A small sensitivity analysis can be found Appendix F.
6 CONCLUSION
In this paper, we formulate HPO as a gray-box function optimization problem that incorporates an important domain of the response function, the dataset itself. We design a novel universal response model for zero-shot HPO that provides good initial hyper-parameters for unseen datasets in the absence of associated observations of hyper-parameters. We propose and optimize novel multi-task objective to estimate the response while learning expressive dataset meta- featuree. We also reinforce the assumption that similar datasets behave similarly similarity driven regularization technique. As part future work, we will investigate the impact of approach within the reinforcement learning framework.
perammerrseeters by introducing novel

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX] symposium on operating systems design and implementation ({OSDI) 16), pp. 265-283, 2016.
Arthur Asuncion and David Newman. Uci machine learning repository. 2007.
Rémi Bardenet, Mátyás Brendel, Balázs Kégl, and Michele Sebag. Collaborative hyperparameter tuning. In International conference on machine learning, pp. 199-207, 2013.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(Feb):281 1-30, 2012.
Pavel B Brazdil, Carlos Soares, and Joaquim Pinto Da Costa. Ranking learning algorithms: Using ibl and meta-learning on accuracy and time results. Machine Leurning, 50(3):251-277 2003.
Harrison Edwards and Amos Storkey. Towards a neural statistician. arXiv preprint arXiv:1606.02185, 2016.
Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Using meta-learning to initialize Bayesian optimization of hyperparameters. Proceedings of the 2014 International Conference on Meta-learning and Algorithm Selection- 1201, pp. 3-10. Citeseer, 2014.
Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Initializing Bayesian hyperparameter optimization via meta-leaming. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for Bayesian opti- mization using ranking-weighted Gaussian process ensembles. In AutoML Workshop at ICML, 2018.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03117, 2015.
Hadi S. Jomaa, Lars Schmidt-Thieme, and Josif Grabocka. Dataset2vec: Learning dataset meta- features. arXiv preprint arXiv:/905.1 1063, 2019.
Donald R Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455 492, 1998.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Advances in neural information processing systems, pp. 971-980, 2017.
Věra Krková. Kolmogorov's theorem and multilayer neural networks. Neural networks, 5(3):501-506, 1992.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: A framework for attention based ietpuuaatioonnnvaiiin neural networks. In International Conference on Machine Learning, pp. 3744-3753 PMLR, 2019.
Rui Leite and Pavel Brazdil. Predicting relative performance of classifiers from samples. In Proceedings of the 22nd international conference on Machine learning, pp. 497-503, 2005.
Shusen Liu, Dan Maljovec, Bei Wang, Peer-Timo Bremer, and Valerio Pascucci. Visualizing high- dimensional data: Advances in the past decade. IEEE transactions on visualization and computer graphics, 1249-12224.. 2016.
Jonas Močkus. On Bayesian methods for seeking the extremum. In Optimization techniques IFIP technical conference, pp. 400-404. Springer, 1975.
Valerio Perrone, Rodolphe Jenatton, Matthias W Seeger, and Cédric Archambeau. Scalable hyperpa- rameter transfer learning. Advances in Neural Information Processing Systems, pp. 6845-6855, 2018.
10

---

Under review as a conference paper at ICLR 2021
Valerio Perrone, Huibin Shen, Matthias W Seeger, Cédric Archambeau, and Rodolphe Jenatton. Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning. In Advances in Neural Information Processing Systems, pp. 12771-12781, 2019.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School Machine Learning, pp. 63-71. Springer, 2003.
Nicolas Schilling, Martin Wistuba, and Lars SchmidttTThieme Scalable hyperparameter optimization with products of Gaussian process experts. In Joint European conference on machine learning and knowledge discovery in databases, pp. 33-48. Springer, 2016.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pp. 2951-2959, 2012.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958 2014.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
Joaquin Vanschoren. Meta-learning A survey. arXiv preprint arXiv:1810.03548. 2018.
Michael Volpp, Lukas Fröhlich, Andreas Doerr, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for Bayesian optimization. arXiv preprint arXiv:1904.02642, 2019.
L Darrell Whitley, Francisco Chicano, and Brian W Goldman. Gray box optimization for mk
landscapes (nk landscapes and max-ksa)) Evolutionary computation, 24(3)491-559, 2016.
Fela Winkelmolen, Nikita Ivkin, H Furkan Bozkurt, and Zohar Karnin. Practical and sample efficient zero-shot hpo. arXiv preprint 2rXiv:2007. 13382, 2020.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Learning hyperparameter optimization initializations. In 2015 IEEE international conference on data science and advanced analytics (DSAA), pp. 1-10. IEEE, 2015a.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Sequential model-free hyperparameter tuning. In 2015 IEEE international conference data mining, pp. 1033-1038. IEEE, 2015b.
Martin Wistuba, Nicolas Schilling, and Lars Shhmidt-Thiemme Two-stage transfer surrogate model for automatic hyperparameter optimization. In Joint European conference on machine learning and knowledge discovery in databases, pp. 199-214 214. Springer, 2016.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Scalable Gaussian process-based transfer surrogates for hyperparameter optimization. Machine Learning, 107(1):43-78, 2018.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 160.07160...7146 2016.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Advances in neural information processing systems, pp- 3391-3401, 2017.

---

Under review as a conference paper at ICLR 2021
A DETAILED PROBLEM SETTING
By a learning task we denote a pair (p. () of an unknown distribution of pairs (x, y) E M+L with M, L N, and loss l RL R- + R. A function ÿ R 12 is called model for task and
((::)) = E(..))=((((.
its (expected) loss.
Let be a learning algorithm that yields for every sample D of pairs (æ, y) from task (p, E) and pyper-parameters IR model I for the task. We call
((A) E(a(D,
the loss (or the response) of hyper-parameter X. We say validation loss for the loss estimated on fresh validation data.
Sequential single-task hyper-parameter optimization problem. Given an initial number K of pairs (Ag, (x) of syper-parameters and their (validation) losses and a budget E N of trials, find AK+1 AK+B. such that their smallest loss
sequentially B many
min kEI:K
8(Aa)
minimal among all such sequences. To compute the next guess λ the hyper-parameters ,, tried so far and their (validation) losses 4 := {(xx) can be used.
Zero-shot eross-task hyper-parameter optimization problem. Let plnkk be an unknown distribu- tion of supervised learning tasks. Given sample of triples ((p, f), 1) of learning tasks (p. E), hyper-parameters and their losses find for fresh task (p, E) and budget B observations of losses of such that their smallest loss minge1.B ((Ax) ) is minimal among all such sets.
without any on this task - set (A1,..., }B} } of hyper-parameters,
Sequential cross-task hyper-parameie optimization problem. Given both, (i) sample of triples ((p, E), X, 1) learning tasks (p, E). hyper-parameters À and their losses and (ii) a fresh task (p, E) and budget B N. find sequentially B many hyper-parameters Åι.... Åg, such that their smallest loss min {(AR) is minimal among all such sequences. To compute the next guess λg+11 the hyper-parameters X A tried far and their (validation) losses 4 = ((Ax)) ) as well as all data on other tasks can be used.
B THE META-FEATURE EXTRACTOR
The meta-feature extractor is a set-based function, and is represented as an extended derivation of the Kolmogorov-Arnol representation theorem (Krková (1992)) which states that a multi-variate function & can be defined as an aggregation of univariate functions over single variables:
2
M
Ha FM
hmm
9mg (m))
important note that permutation invariant, i.e. unaffected by any permutation on the input, which allows us to obtain the same output for the same order input. As simple variant of this formulation (Zaheer et al.) (2017)), we can replace the set of functions hi,. with single function h, and Gm.j with function y. In this paper, we incorporate the meta-feature extractor as part of the response model, effectively learning a conditional response on the dataset meta-features directly such that the approximation defined as ((x(t), ((9); 3), with ((0)_ = o(D())
variate data regardless of the
12

---

Under review as a conference paper at ICLR 2021
C META-DATASETS
C.I LAYOUT SYPPER-PARAEETE
Below are some examples of the number of neurons per layer for networks with different layout hyper-parameters given 4 neurons and 5 layers:
Layout C [4,4,4,4,4]
Layout <I: [4,8,16,32,64]
Layout [64,32,16.8,4]
Layout [4,8,16,8.4]
Layout A: [16,8,4,8,16]
The search space consists of all possible combinations of the hyper-parameters. After removing redundant configurations, e.g. A layout with layer is similar a layout with layer, the resulting meta-datasets have 256, 288, and 324 unique configurations respectively.
C.2
HYPER PAREER--RRMEER ENCODING
Below description of the encodings applied to our hyper-parameters. We also like note that the scalar values normalized between (0, 1).
Table 6: Encoding of the different hyper- parameters used in the meta-dataset.
Hyper-parametes
Encoding
Activation
One-hot encoding
Neurons
Scalar
Layers
Scalar
Layout
One-hot encoding
Dropout
Scalar
Normalization
Scalar
Optimizer
One-hot encoding
C.3 THE UCI DATASETS
Table 7 is an overview of the UCI datasets used to generate the meta-datasets.
13

---

Under review as a conference paper at ICLR 2021
Table 7: Summary of the 120 UCI datasets used to generate the meta-datasets.
UCI Dataset
# Instances
# Features
# Classes
UCI Dataset
Instances
# Features
# Classes
molec: biol-splice
31561
Iwomorm
5550
20
amnealing
798
31
5
plant-texture
malec-hiel molec-hiol-promeeer prommter
ringnorm
5550
20
contrac
1105
9
3
spect
stailog-landsat
energy-y2
576
8
cunn bench sonar mines rocks
156
60
2
stocl-plates
166
verteball-olumn--lless
233
6
balloons
12
2
chess-krvk
abalone
statlog-shurtlc
43500
statlog-vehiele
635
breast-cancer-wicc
524
page-hlacks
4105
somcion
1195
256
10
heart-hungarian
221
2
50668
ionosphere
263
monks.3
122
6
syuthetic-comter
450
wall- wall-following
4092
plant plant-shape
1200
100
vertebrul column 2clases
233
6
pittsburg bridges MATERIAL
7
planning
137
preast-cancers wisc-ding
427
cardiotocorraphyy-clae
spoctf
plant-margin
1200
100
bank
3391
nursery
9720
pendigits
7494
titanic
1651
3
tenching
113
5
3
energy-y
576
mushroom
monks-1
124
6
optical
3823
62
10
arrhythmia
primary tumor
breast-tissue
80
9
528
11
11
statlog austlg--usttalannnnom credit
snybean
tic-tac-tee
719
9
2
avcytes merluccius states 21
767
25
3
lymphography
chess krekp
monks-2
169
6
2
sudi
171
59
18
waveform
3750
image-aegmentation
fortility
75
9
lod-display
750
7
10
lenses
heart-va
150
wine-quality-red
1199
pittsburg-rridee-SSAM
69
7
parkinsons
146
priyptes Lww.leusssssmsss___... _2
684
wine-qualit--whitc
3674
stanlog-ecrman-crddi
750
24
2
pima
576
8
acute-infaammttion
pillsburg bridges T-OR-D
car
1296
low-res-spect
398
100
hosse-oolic
300
musk-1
357
166
heart switzerland
pitsburg-bridges-Hee
7
oocyles trisopterus states
684
becust cancer
215
congressional voting
326
spambase
3451
acute-nephriis
6
iris
113
crodit-approval
518
thyroid
3772
21
hill-valley
606
100
mammographic
721
oocytes meclucciss.ss wtt.:/ccciiiusoclus..u.. mncleus.4d
767
Hipd-indian-livee
437
9
seeds
158
7
3
blood
ozione
waveform-nooi
3750
40
magic
14265
10
2
statlog
statlog image
pittsbugg
bridges-TYEE
79
7
cylinder- bunds
384
35
2
echocardiogram
lung cancer
Bags
146
28
8
dermatology
275
6
letter
15000
cardiotocography-lUue
200
76
16
7
heart clevelcle
227
13
5
252
7
haberman-suue Surviiaa--urvival
yeast
1113
8
10
balance-scale
469
baye
132
3
134
libras
270
15
miniboone
97548
2
teesst-canzer wisc-prog prog
149
hepatitis
116
glass
161
9
post operative
8
3
14

---

Under review as a conference paper at ICLR 2021
D ALGORITHMS
We define pT as the task distribution that represents pairs of datasets and hyper-parameters, i.e. ((t.)) (D X T A, and pD be the distribution of the datasets defined in Section 4.2
Algorithm provides the overall optimization framework for GROSI, our approach.
Algorithm Learn, L GROSI(D)
1: Require:pp: distribution over datasets.pr: distribution over tasks
2: mequirel//mmer
learning rates
3: Randomly initialize /3 E B, the parameters of response model
4: while not done do
5: Set 3 8 B
6: Sample (D(1), x1 = T((,k) ~ PT
7: for steps
8: sample ( PD)((D))
9:
Evaluate gradients G +- V (O + R R+
10:
Compute adapted parameters with stochastic gradient descent: 3' + B' I Update 3 +
11:
Wouterr (3 B*)
12:
return /
After optimizing our objective via Algorithm D we apply Algorithm 2 to observe the results presented in Tables 3 and 5
Algorithm 2 Zero-shot HPO
1: Require: target dataset D(1), response model i; desired zero-shot hyper-parameten
2: H. +- arg min (D((),,)
3: return H
For sequential model-based optimization, surrogate i is fitted to the observed responses of the unknown function. Several initialization strategies exist to expedite the transfer of information across tasks, Section 5.4 In Algorithm we present the generic pseudo-code for SMBO, that requires an acquisition function, a, to sample the next iterate from the domain.
Algorithm 3 Sequential Model-based Optimization Warm-start
1: Require: target dataset (+)) response model Í; desired zero-shot
K, number
of trials 7, acquisition function a
2: Get initial hyper-parameters Ho <-Zero-shot HPO
3: A f- arg Xo (I(D(t),)) A))
4: for =
5:
fi to Hi-1
À + arg max a (I(D(),,))
6:
7:
H: + Hi-1 U{A}
8:
iD( >) < <l (D((), )ma)
9:
min t X
return min
In Section 5.4, we propose to initialize our response model on the target dataset, then iterativly tune it that particular dataset. Initially, select top K configurations based on Algorithm 2 zero-shot approach. Then, via Algorithm 4 we sample uniformly at random from the top ranking configurations. If X 1, then this represents the greedy policy.
Meta-feature learning from datasets with varying schema was initially proposed (Jomaa et al. (019 For our approach, introduce set-hased meta-feature extractor module to handle datasets
15

---

Under review as a conference paper at ICLR 2021
Algorithm 4 Learn_GROS((+X)
1: Require: target dataset D((), response model I; desired zero-shot hyper-parameters K, number of trials I, Number of top configurations choose from, X
2: Get initial hyper-parameters Ho -errro-shot HPO
3: A < arg min (I(D(),))
4: for =11
5:
fit 4 to H4-1 by optimizing Equation 3 N
Uniform arg minXEN,H.- (D(1).))
6:
7:
H4 f H4-1 U{}}
8:
if (D()) >) I (D(t), (min)) Amin)
9:
min
10:
return mii
of varying schema well, however, optimize Equation and use the dataset identification task as an auxiliary objective. However, to pre-train the meta-fcature extractor for the Ablation study, Section as well as in order extract meta-features for the NN-D2V, and TST-D2V,
follow
Algorithm 5 with pn+ as the distribution of similar datasets, PD+ {(D(t) D(9), s) ~ PD 1}.
and pn- as the distribution of dissimilar datasets, Pn- {(((t, D(a), s) ~ PD = 0}. Similar datasets defined as multi-fidelity subsets (batches) of each dataset.
Algorithm 5 Standalone Meta-feature Learning
1: Require:pD+ distribution over similar datasets, PD- distribution over dissimilar datasets
2: Require:lr learning rate
3: Randomly initialize 3 E B, the parameters of the meta- meta-feature extractor 6
4: while not done do
5: Sample (D(1) D(4)
ppt and (D(1), D(r) 0) PD (Both samples share (()) 6: Evaluate gradients G t V (P)
7: Compute adapted parameters with stochastic gradient descent: B 3 Ir
8: Update 8 + 3 Ir& (5 3)
9: refurn
16

---

Under review as a conference paper at ICLR 2021
E EXPERIMENTAL DETAILS
E.1 NETWORK ARCHITECTURE
Our model architecture is divided into two modules, i= & o th the meta-feature extractor c, and the regression head v.. The meta-feature extractor à IR IR composed of three functions, Equation 4 namely J. and h. The regression head is also composed of two functions, i.e./ d/ 0/122 We define by 1 : R Kv x A - IR Kw as the function that takes as input the neta-featurehhyper parameter pair, and by t22 RK + E the function that approximates the response. Finally, let Dense(n) define one fully connected layer with n neurons, and ResidualBlock(n,m) be mx Dense(n) with residual connections (Zagoruyko & Komodakis (2016))
To select a single universal response model, we evaluate the validation performance on the three network architectures described in Table 8 We select the architecture that has the best average performance between three across the three meta-datasets, Table 9 which turns out to be Architccture 3. The architectures assign different number of trainable variables for the meta-feature extractor and the coupled regression head.
Table 8: The network architecture optimized for every meta-dataset.
Functions
Architecture 1
f
g
Layout: E> with layers and 16 Neurons ( ResidualBlock(3.I 1e) (ense(16):31
h
Layout: 1- with layers and Neurons
d2
Layout: E> with 4 layers and 4 Neurons
Functions
Architecture 2
f
Dense(16);6x ResidualBlookk 1onne(11dXxxxXeeesdaaBBBnnnCCC Layout: -> with 3 layers and 16 Neurons Ressiuulloos (esiduulBBoock33332))deeeeeee Densee)))))))) Layout: 1> with layers and 4 Neurons
Densee)
h
1/2
Layout: 1> with layers and 16 Neurons
Functions
Architecture 3
f
Dense( (1nse(16)6x20) x (oo.auuBBBoccc..).)eeeeeeee Layout: i> with layers and 16 Neurons (ense(16);33 Residua/lBlock33... (enn Layout: & with 4 layers and 4 Neurons
g
h
d1
2/2
Layout: >> with layers and 16 Neurons
Table 9: Final results of each model optimized on the different meta-datasede The numbers reported are the average normalized regret after trials on held-out validation sets for the zero-shot task.
Architecture
Layout Md
Regularization Md
Optimization Md
Average Score
Architecture
0.942 + 0.2074
1.697 + 0.2897
2.935 +:: 1.4562
1.858 1 0.821 2.012 1 0.922
Architecture 2
0.973 = 0.5338
1.849 1 0.5397
3,215 + 1.3578
Architecture 3
1.041 = 0.4134
1.715 + 0.6354
2.579 + 0.9865
1.779 + 0.629
E.2 POLICY FOR SEQUENTIAL OPTIMIZATION
We propose GROSI zero-shot HPO solution. However, to emphasize the ability of our surrogate model to quickly adapt to new target datasets, we extend into a sequential optimization approach. Starting with the proposed zero-shot configurations, fine tune our model via Algorithm 4 We select the X = 10 based on the best average performance observed on the held-out validation sets, Table
17

---

Under review as a conference paper at ICLR 2021
Table 10: Final results for different variants of our sequential model optimization policy. The numbers reported are the average normalized regret after 80 trials on held-out validation after initialization with the exact same 20 configurations suggested by the our zero-shot approach.
Sequential Policy
Layout Md
Regularization Md
Optimization Md
Average Score
GROSI(+1)
0.543
1.793.11.057
2.835110.663
1.760110.91
GROSI(+3)
0.204
1.845=
2.738+11.033
1,596=1.004
GROSI(+5)
0,403=0
2.027=11171
2.480++09667
1.637=00892
GROSI(+7)
0.675
1.576:10.845
2.476:1
1.576:10.735
GROSI(+10)
0.550-10:498
1.727::1.075
1.996:11.111
1.424410.628
F ADDITIONAL. EXPERIMENTAL RESULTS
F.1 HYPER- PARAMETER SENSITIVITY ANALYSIS
We optimize our response model by minimizing Equation 8 which includes the dataset identification task, Equation and the similarity-driven regularization task, Equation 7 with auxiliary weights and assigned to both respectively. We report below the performance of our universal response model for different auxiliary weights. The results confirm the importance of emphasizing the auxiliary dataset identification task in conjunction with the similarity-driven regularization loss, which reinforces the intuition that similar datasets behave similarly to the hyper-parament response. The reported results throughout the paper are based 8 and 0.5.
Table 11: Final results of our universal response model optimized with different auxiliary weights. The numbers reported are the average normalized regret after 20 trials.
Method
Layout Md
Regularization Md
Optimization Md
8 0.5; a = 0.1
6.016 1.196
5.358 + 1.676
6.744 + 0.468
8 0.5: = 0.5
6.973 + 2.525
6,483 + 1.614
6.401 = 0.325
GROSI(OUrS): 1;a 0.5
5.707 = 1.539
5.903
1.655
6.242 = 1.223
F.2 ADDITIONAL RESULTS
Q1: ZERO-SHOT HPO AS A STAND-ALONE PROBLEM
As a plausibility argument for the usefulness of our zero-shot strategy, we depict in Figure the top 10 suggested hyper--arameters by our approach, well two initialization strategies on the actual response surface. Our picks can be seen colocated near the different optima in the search space whereas hyper-parameters other strategies are dispersed.
Q3: SEQUENTIAL GRAY-BOX FUNCTION OPTIMIZATION
We refit the universal response model the observations of the response on the target dataset by optimizing Equation 3 We depict the improvement achieved over the zero-shot approach in Figure 5
18

---

Under review as a conference paper at ICLR 2021
lenses
credit-approval
car
1.0
Respanse EMI
0.8
0.6
0.4
Surrogate itA.
0.2
0.0
Random
NN-D2V
GROSI
Figure 3: Top 10 ppper-prraettrss suggested by our approach for test datasets from the three different meta-datasets. We reduce the dimensionality of each search space into a representation via TSNE (Liu et al. (2016)). The first row represents the actual response surface. We notice that the true response and the predicted response are similar, and the location of the predicted minima, in green, overlaps with the minima of the actual response.
Layout Md
Regularization Md
Optimization Md
Normal red Regret
6
20
40
60
80
20
40
60
80
20
40
60
80
Number of Trials
Number of Trials
Number of Trials
TST-R
TST-DZV
RGPE
TAF-R
ABLR
GROSI
GROSI(+10)
Figure 4: Average normalized regret for tfute-of-the-at transfer learning HPO methods, with uncer- tainty quantification.
Layout
Regularization
Optimization
Normalized Regret
Number of Trials
Number
Number of Triais
GROSI
GROSI[+1]
GRO5
Figure 5: Average normalized regret for GROSI, our zero-shot approach, and the refitted response models GROSI(+1) and GROSI(+10). We shade in light blue the improvement over zero-shot performance.
19


Reviews:
[{"review_id": "p51-V4rAGc", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a novel approach to zero-shot transfer learning for hyper parameter optimisation (HPO). In contrast to standard HPO approaches, where HPs are found for each task by random/grid search or Bayesian Global Optimisation or sequential model based optimisation, the proposed approach learns a response model defined over the combined domain of datasets and hyper-parameters such that for a new task, for which training data and labels and some ranges of HPs are given, HPO can be done by querying the response model without having to evaluate the black-box function of the validation loss as a standard HPO would do. \n\nI think the manuscript is very well written, the motivation is sound, the related work section includes many relevant and well summarized examples of similar approaches, the notation and the presentation of the proposed approach are intuitiv and concise. \n\nOne thing that could probably be improved a bit is the coherence of the text, I mean, I had to jump a lot between the Appendix and the paper to understand some necessary details. For instance the core of the approach in section 4, and eqs 3-5 are very difficult to understand without the information referenced in the last paragraph of that section. I wonder if it would be possible to mention some of the relevant information, e.g. how eq. 5 is optimized or how the dataset2vec features are learned, right in section 4. \n\nThe experimental evaluation appears solid and comprehensive. I like how the authors investigated their approach in three different settings. Compared with a large number of alternative approaches the proposed approach achieves improvements in all HPO settings, ‘cold-start’, ‘warm-start’ and continuous learning of the response model. One question I had when thinking about the experiments is that the proposed approach relies on a number of parameters to be chosen well, in particular the HPs that determine the computation of the meta features and the response model. It appears that the response model used some basic settings that were not optimised for the task, but I still wonder how sensitive the learning of the response model is to the choice of hyper parameters. That includes the 'generalisation performance' of the dataset2vec model and all other ingredients of the response model, apart from the validation loss of the black/grey-box ML model.\n\nWhen comparing the proposed approach with existing approaches, it seems that at least some of the aspects that make this work so well should also be captured in other approaches that use some dataset features in a transfer learning HPO setting. Some of them are mentioned in the related work section (Perrone et al, Volpp et al, Feurer et al, …) but especially the experimental setting of SEQUENTIAL GRAY-BOX FUNCTION OPTIMIZATION looks very similar to reinforcement learning approaches to HPO which include dataset features in their policy (e.g. https://arxiv.org/abs/1906.11527). Given these similarities and the strong empirical results it would be great to get a better understanding of which elements that are different to existing work contribute most. The ablation study in Appendix F.2 is great, and could be discussed in the main text especially in relation to other work. \n\nTo summarise I think the experimental validation suggests that this approach has potential to improve the state of the art in HPO. Some relevant details of the method are not entirely clear, or at least could be better described in the main text. Also in the experimental evaluation it could made clearer that the results are not sensitive to the tuning of the, pretty complex, HPs of the response model itself.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BdVnMa60li_", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe authors of the paper propose a hyperparameter tuning algorithm that uses a simple non-probabilistic model to predict the performance of any new hyperparameter configuration on a target task. They use transfer learning from related tasks as well as learned meta-features for each task, and then simply pick the hyperparamater configuration predicted to perform best by this NN based model.\n\nThe authors provide results on 3 meta-datasets, each a slightly different hyperparamter optimization problem applied to the same 120 UCI datasets, thus providing relataed task for each of these three settings. The algorithm can be used in three modes Q1, as a zero-shot algorithm to immediately select the best performing hyperparameters Q2, as an initialization strategy used in the same way as Q1 but followed by a traditional sequential HPO algorithm, and Q3, as a standalone sequential tuning algorithm, by updating the model with any new evaluations, and then using it to suggest the configurations to evaluate next.\n\n\n#### Comments / Questions\nA crucial detail, that is only cursorly mentioned in the main test, is that the authors tune the architecture of their model to be adapted to the specific meta-dataset used (as described in table 5 in appendix D). I think this should be given more attention, having to tune the surrogate hyperparameters itself seems a fairly major limitation of the method and should be discussed. It is not clear exactly how many different architectures were tested, what the search space was, and how well any specific architecture would generalize to new meta-datasets, or even new datasets on the same meta-dataset.\n\nWithout more details on how the tuning of the surrogate architecture exactly happened, it does not seem possible to apply this method to other meta-datasets, making the paper incomplete a non-reproducable.\n\nIt is also not clear can the model can avoid getting stuck in local minima? I would be curious why this was not a problem with the Q3 experiments without the exploration usually given by a probabilistic model.\n\nAlgorithm 3 in the appendix mentions the use of an acquisition function, but the proposed model does not appear to be probabilistic, so it is not really clear what kind of acquisition function would be used here.\n\nNormalized regret used for all results is never defined.\n\n#### Pros\n* Important use case/setting (expeciall Q1 and Q2)\n* Simple idea, although not described so straighforwardly, I would avoid all the talk of a grey-box method and focus more on simplicity and the idea of providing a good and simple zero-shot NN model that is able to use meta-features in an effective way\n* Meta-datasets useful for research are being published with the paper (or at any rate promised to be published upon acceptance)\n* The experimental results look fairly competitive (although with some of the shortcomings and open questions I mention in this review)\n\n\n#### Cons\n* The authors tune their model architecture to be adapted to each specific meta-dataset, but this tuning process is not well described, making the method hard to reproduce.\n* The algorithm is not very principled from a probabilistic standpoint when used in setting Q3, the used exploitation and exploration trade off is just an heuristic, and not guaranteed to provide good exploration. I would expect it to plateau at a much worse value than bayesian optimization methods. Honestly I would think the paper would be stronger without this use case described, and more more focus given to the results and the zero-shot case.\n* The theory sections in the main text are fairly standard and could be shortened a lot, as well as not really needed to explain the method used, what is more relevant is the actual model used, in terms of architecture and tuning. In the current version the architecture is relegated to the appendix and the tuning absent.\n* In general the paper is not really self contained, instead a lot of the important details are contained in the appendix. I would also suggest to be clearer about what the main contribution is.\n* Any uncertainty quantification (confidence intervals or error bars) in the results is missing, thus it is difficult to assess how reliable the results are.\n\n\nI think in the current state the paper is not ready to be published. However the idea is simple and promising. I would suggest the authors improve the description of the experimental results, comparing against some more zero-shot baselines. Also either the tuning process of the model itself should be described in more detail in the main text, or not be required (in case a more general architecture can be provided).\n\n\n\n#### Typos/Other minor comments\nIt would be using always use the same term instead of simetimes probabilistic surrogate and other times response model.\n\nThe way l_hat and the selection of the surrogate is chosen by max likelihood is a bit convoluted, but as I said above, I don't think the theory sections in the main text add much insight into the method, just direcly describe the loss.\n\nSection 4:\ninputting -> taking as input\npage 14 first paragraph:\nto optimize outer loop -> to optimize *the* outer loop\nAppendix B1:  learningJomaa -> missing space\nAppendix Q1: typos in \"but the time training required time of underlying model.\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "A5kgZGbJUiU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this papar, the authors formulated a new objective function for HBO, which included an additional regularization term based on the dataset similarity. The authors used the distance between the meta-features of selected datasets to measure this dataset similarity and assumpted that similar datasets should have similar hyper-paprameters.  The experiments are complete and demonstrate the advantages of using this new optimization formulation.\n\nI have three major concerns as following.\n\n1. In Appendix B.1, it mentioned that using the similarity regularization defined in Equation 4. alone is not sufficient to measure the dataset similarity, so an additional similarity metric Equation 9. is used. What is the impact of directly replacing the distance metric in Equation 4. with Equation 9?\n\n2. Since the similarity regularization is computed using K selected datasets, and \"for each meta-dataset the architecture of the models are selected based on their performance on their validation datasets\", how to ensure that the proposed universal model trained on these K datasets can be generalized to other datasets.\n\n3. It is suggested to re-organize the description of the proposed method. In the current version of the manuscript, many essential details are put in the appendix, such as the optimization of Eq. 5. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "4EGXFoKfp3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe authors propose a new zero-shot hyper-parameter optimization method based-on the meta-learning framework. The proposed method incorporates two ideas from the meta-learning framework namely the task similarity based on the meta-features and the dataset identification. The former idea is used to achieve the requirement that responses to similar data sets should be similar.The latter idea has the role of preventing data of dissimilar tasks from being embedded in close proximity in the meta-feature space.\n\nIn this paper, the authors construct a new meta-dataset to evaluate the proposed method by compare it with various HPO methods (both zero-shot approach and sequential model based approach).\n\nWhile the performances of various HPO tasks certainly seems to have improved, there are some concerns.\n\n- How can you justify the proposed approach  which measures domain similarity in terms of distances between meta-features  (some kind of point estimation), rather than inter-distributional distances? My concern is that the information may be considerably dropped than measuring similarity by inter-distributional distance.\n\n- The algorithm in the appendix doesn't have a section on learning meta features, how (and when) are these updated in the algorithm?\n\n- Appendix B.1 defines a regularization term for dataset identification. \n\n  -- Although the range of \\hat{s} is defined as {0, 1}, in (8) \\hat{s} appears to be a continuous value. Are you using some kind of threshold to make it binary? \n\n  -- The authors argue that  \"Without loss of generality, we use the Euclidean distance to measure the similarity between the extracted meta features, ... and \\gamma = 1\". What do you mean by \"without loss of generality\" here? Does this mean that other metrics and \\gamma values will give same results?", "extra_scores": {}, "extra_text": {}}, {"review_id": "1oeEUqQ4Yu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n\nThis work proposes a solution for the zero-shot hyper-parameter optimization problem without access to observations of losses of the target response.\n\nThe proposed method is parallelizable and thus can reduce the required wall-clock time vastly.\n\nThree new meta-datasets with different search spaces are proposed and served as a benchmark.\n\nA battery of experiments and the completions comparison against state-of-the-art HPO solutions are conducted to demonstrate the performance of the proposed approach.\n\n\nCons:\n\nThis work may be treated as applying meta-learning to hyper-parameter optimization. That is, a model is introduced to learn how to learn a group of optimal parameters based on the given meta-features. The acquisition function, meta-features extraction, meta-learning optimization method, the dataset similarity loss, etc., are all borrowed from prior works.\n\nThe meta-features of datasets are directly computed by Dataset2Vec [1]. This paper points out that prior methods rely on engineered meta-features, but actually [1] can be applied to improve them as well. I suggest the authors discuss the situation where prior methods and [1] are combined to reveal the advantage of the proposed method.\n\n\n[1] Jomaa, Hadi S., Lars Schmidt-Thieme, and Josif Grabocka. \"Dataset2vec: Learning dataset meta-features.\" arXiv preprint arXiv:1905.11063 (2019).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a transfer learning approach to HPO. It builds and improves upon existing methods of zero-shot HPO where the high level idea is to use the outcomes of hyper-parameters on an offline collection of datasets in order to speed up HPO on a new dataset. On the plus side, the methods provided seem to be novel, and the results seem to be promising. The main issue is the writing and clarity of the paper, making it hard to be certain of the good qualities of the paper. Aggregating the reviews, the details are too spread out between the appendix and main body, the techniques require more motivation behind them, and important details of the experiment are somewhat vague. The authors provided a modified version which is definitely a step in the right direction, however, it does not seem to be enough. I think this is a solid paper based on a promising idea. However, given the almost unanimous agreement about that crucial gap in clarity even after the modified version was uploaded, I recommend rejecting the paper. 
Decision: Reject