A Copula approach for hyperparameter transfer learning
Authors: David Salinas
Huibin Shen
Valerio Perrone
Venue:  (2020)
Keywords: Hyperparameter optimization
Bayesian Optimization
Gaussian Process
Copula
Transfer-learning

Abstract: Bayesian optimization (BO) is a popular methodology to tune the hyperparameters of expensive black-box functions. Despite its success, standard BO focuses on a single task at a time and is not designed to leverage information from related functions, such as tuning performance metrics of the same algorithm across multiple datasets. In this work, we introduce a novel approach to achieve transfer learning across different datasets as well as different metrics. The main idea is to regress the mapping from hyperparameter to metric quantiles with a semi-parametric Gaussian Copula distribution, which provides robustness against different scales or outliers that can occur in different tasks. We introduce two methods to leverage this estimation: a Thompson sampling strategy as well as a Gaussian Copula process using such quantile estimate as a prior. We show that these strategies can combine the estimation of multiple metrics such as runtime and accuracy, steering the optimization toward cheaper hyperparameters for the same level of accuracy. Experiments on an extensive set of hyperparameter tuning tasks demonstrate significant improvements over state-of-the-art methods.

Under review as a conference paper at ICLR 2020
A COPULA APPROACH FOR HYPERPARAMETER TRANS- FER LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Bayesian optimization (BO) is a popular methodology to tune the hyperparame- ters of expensive black-box functions. Despite its success, standard BO focuses on a single task at time and is not designed to leverage information from related functions, such tuning performance objectives of the same algorithm across multiple datasets. In this work, we introduce a novel approach to achieve trans- fer learning across different datasets as well different objectives. The main idea is to regress the mapping from hyperparameter to objective quantiles with a semi-parametric Gaussian Copula distribution, which provides robustness against different scales outliers that can occur in different tasks. We introduce two methods to leverage this estimation: a Thompson sampling strategy as well as a Gaussian Copula process using such quantile estimate prior. We show that these strategies can combine the estimation of multiple objectives such as runtime and accuracy, steering the optimization toward cheaper hyperparameters for the same level of accuracy. Experiments on an extensive set of hyperparameter tuning tasks demonstrate significant improvements over state-of-the-art methods.
INTRODUCTION
Tuning complex machine learning models such as deep neural networks can be a daunting task. Object detection or language understanding models often rely on deep neural networks with many tunable hyperparameters, and automatic hyperparameter optimization (HPO) techniques such as Bayesian optimization (BO) are critical to find the good hyperparameters in short time. BO ad- dresses the black-box optimization problem by placing probabilistic model on the function to minimize (e.g., the mapping of neural network hyperparameters to a validation loss), and determine which hyperparameters to evaluate next by trading off exploration and exploitation through an ac- quisition function. While traditional BO focuses on each problem in isolation, recent years have seen a surge of interest in transfer learning for HPO. The key idea is to exploit evaluations from previous, related tasks (e.g., the same neural network tuned on multiple datasets) to further speed up the hyperparameter search.
central challenge of hyperparameter transfer learning is that different tasks typically have different scales, varying noise levels, and possibly contain outliers, making hard to learn joint model. In this work, we show how semi-parametric Gaussian Copula can be leveraged to learn a joint prior across datasets in such a way that scale issues vanish. We then demonstrate how such prior estimate can be used to transfer information across tasks and objectives. We propose two HPO strategies: Copula Thompson Sampling and Gaussian Copula Process. We show that these approaches can jointly model several objectives with potentially different scales, such as validation error and compute time, without requiring processing. We demonstrate significant speed-ups over a number of basclines in extensive experiments.
The paper is organized as follows. Section 2 reviews related work on transfer learning for HPO. Section 3 introduces Copula regression, the building block for the HPO strategies propose in Section Specifically, we show how Copula regression can be applied to design two HPO strategies, one based on Thompson sampling and an alternative Ga-based approach. Experimental results given in Section | where we evaluate both approaches against state-of-the-ar methods on three algorithms. Finally, 6 conclusions and further developments.

---

Under review as a conference paper at ICLR 2020
2 RELATED WORK
A variety of methods have been developed to induce transfer learning in HPO. The most com- mon approach is to model tasks jointly or via conditional independence structure, which has been been explored through multi-output GPs (Swersky et al., 2013), weighted combination of GPs (Schilling et al., 2016; Wistuba et al,, 2018: Feurer et al., 2018), and neural networks, either fully Bayesian (Springenberg et al. (016) or hybrid (Snoek et al. 2015: Perrone et al. 2018 Law jet al., 2018). A different line of research has focused on the setting where tasks come over time as a sequence and models need to be updated online new problems accrue. A way to achieve this is to fit a sequence of surrogate models to the residuals relative to predictions of the previously fitted model (Golovin al. 2017 Poloczek al. 2016). Specifically, the GP over the new task is centered on the predictive mean of the previously learned GP. Finally, rather than fittingg a surrogate model to all past data, some transfer can be achieved by warm-starting BO with the solutions to the previous BO problems (Feurer et al. 2015 Wistuba et al. 2015b).
A key challenge for joint models is that different black-boxes can exhibit heterogeneous scale and noise levels (Bardenet et al., 2013; Feurer et al. 2018). To address this, some methods have instead focused on search-space level, aiming to prune to focus on regions of the hyperparameter space where good configurations are likely to lic. An example is Wistuba et al.] (2015a), where related tasks are used to learn a promising search space during HPO, defining task similarity in terms of the distance of the respective data set meta- meta-features. A more recent alternative that does not require meta-features was introduced in Perrone et al. (2019), where a restricted search space in the form of low-volume hyper-rectangle or hyper-ellipsoid is learned from the optimal hyperparameters of related tasks. Rank estimation can be used to alleviate scales issues however the difficulty of feeding back rank information to GP leads to restricting assumptions, for instance (Bardenet et al. 2013) does not model the rank estimation uncertainty while (Feurer et al. 2018 uses independent GPs removing the adaptivity of the GP to the current task. Gaussian Copula Process (GCP) (Wilson & Ghahramani] 2010) can also be used to alleviate scale issues on a single task at the extra cost of estimating the CDF of the data. Using GCP for HPO was proposed in [Anderson et al. (2017) to handle potentially non-Gaussian data, albeit only considering non-parametric homoskedastic priors for the single-tast -asle and single objective case.
3 GAUSSIAN COPULA REGRESSION
For each task denote with f : RP - Ih the error function one wishes to minimize, and with D {IA, M.N the evaluations available for arbitrary task. Given the evaluations on M tasks DM = UIsigar (x), ve )= we are interested speeding up the optimization of an arbitrary new task f. namely in finding arg min fxx in the least number of evaluations. One possible approach to speed-up the optimization of ʃ is to build surrogate model f(x). While using Gaussian process is possible, scaling such an approach to the large number of evaluations available
in a transfer learning setting is challenging. Instead, we propose fitting a parametric estimate of (a) distribution which can be later used in HPO strategies as prior of a Gaussian Copula Process. A key requirement here is to learn a joint model, e.g., we would like to find 0 which fits well on all observed tasks f We show how this can be achieved with a semi-parametric Gaussian Copula in two steps: first we map all evaluations to quantiles with the empirical CDF, and then we fit a parametric Gaussian distribution on quantiles mapped through the Gaussian inverse CDF.
First, observe that as every yi comes from the same distribution for a given task, the probability integral transform results in u, Fly.), where F the cumulative distribution function of y. We then model the CDF of (ui...., uN) with a Gaussian Copula:
Clui,..., uN) $ (FF(F))())))....)))))))))))))))) T (F(xx)))
where @ is the standard normal CDF and ميرر is the CDF of normal distribution parametrized by and Σ. Assuming F to be invertible, we define the change of variable = Φ-1 a Fly) = (v)) and g = da | We regress the marginal distribution of P(2[z) with Gaussian distribution whose
Note that if is regressed perfectly, then finding the minimum of f is solved as parameter minimizing Of (x)) also minimizes fz) since o monotonically increasing.
2

---

Under review as a conference paper at ICLR 2020
mean and variance two deterministic parametric functions given by
P(z|0) ~ (((p(()). (N (x) bus V (w (x) bo)),
where hww (x) E IR is the output of a multi-layer perceptron (MLP) where wh. W/ E s by E R, Wa E IR" by E R are projection parameters and (t) = log + exp () is an activation mapping positive numbers. The parameters 0 {wn, Wyy by: Wa ba) together with the parameters in MLP are learned by minimizing the Gaussian negative log-likelihood on the available evaluations DMM UssigM La/, 4/ e.g., by minimizing
Σ (r.:)EDM
1 log (
1
(x)) 22)
1/ (0-M() + ('((0
with SGD. Here, the term v/ (=)) accounts for the change of variable = = ( Due to the
term & ((6²(()) eπrоrs committed where the quantile function changes rapidly have larger gradient than when the quantile function is flat. Note that while we weight evaluations of each tasks equally, one may alternatively normalize gradient contributions per number of task evaluations.
The transformation & requires F, which needs to be estimated. Rather than using a parametric or density estimation approach, we use the empirical CDF F(!) = / Σ 1 1: While this estimator has the advantage of being non-parametrie, it leads infinite value when evaluating & at the minimum of maximum of To avoid this issue, we use the Winsorized cut-off estimator
NN F(I) 1- 8N
if (()) < ÔN in F(1) 51-60 ÖN if ((1) > 8N
F(t)=
where N is the number of observations of and choosing 8N log strikes a bias-variance trade-off (Liu al. 2009). This approach is semi-parametric that the CDF is estimated with non-parametric estimator and the Gaussian Copula is estimated with a parametric approach.
The benefit of using non- con-parametric estimator for the CDF is that allows us to map the obser- vations of each task to comparable distributions as x ~ (O, 1) for all tasks j. This is critical to allow the joint learning the parametric estimates 14g and ds which share their parameter 0 across all tasks. As our experiments will show, one can regress a parametric estimate that has a standard error lower than This means that information can be leveraged from the evaluations obtained on related tasks, whereas a trivial predictor for would predict O and yield standard error of 1. In the next section we show how this estimator can be leveraged to design two novel HPO strategies.
4 COPULA BASED HPO
4.1 COPULA THOMPSON SAMPLING
Given the predictive distribution P(z[r) (z)o(z)) it is straightforward to derive Thompson sampling strategy (Thompson 1933) exploiting knowledge from previous tasks. Given N candidate hyperparameter configurations "i,....;''. we sample from each predictive distribu- tion & ~ (Nu (z;), (Ix))) and then evaluate f(I;) where i = arg min; ž, Pseudo-code is given in the appendix.
While this approach can re-use information from previous tasks, it does not exploit the evaluations from the current task as each draw is independent of the observed evaluations. This can become an issue if the new black-box significantly differs from previous tasks. We now show that Gaussian Copula regression can be combined with a GP to both learn from previous tasks while adapting to the current task.
4.2 GAUSSIAN COPULA PROCESS
Instead of modeling observations with a GP, we model them as a Gaussian Copula Process (GCP) Wilson & Ghahramani 2010). Observations mapped through the bijection v Φ-1 o F, where
3

---

Under review as a conference paper at ICLR 2020
recall that & is the standard normal CDF and that F is the CDF of y. As á is monotonically increasing and mapping into the line, we can alternatvelly view this modeling as a warped GP (Snelson et al. 2004) with non-parametric warping. One advantage of this transformation is that == (v() follows normal distribution, which may not be the case for = f(x). In the specific case of HPO, y may represent accuracy scores in [0, 1] of classifier where Gaussian cannot be used. Furthermore, we can use the information gained on other tasks with μg and de by using them as prior mean and variance. To do so, the following residual modeled with GP:
r(x) g(x) (o(
(o(x)
GP(m(()) <(x,,')),
where 9 f. We use a Matérn-5/2 covariance kernel and automatic relevance determination hyperparameters, optimized by type II maximum likelihood to determine GP hyperparameters (Ras-) mussen & Williams 2006). Fitting the GP gives the predictive distribution of the residual surrogate
f(x) ~ (N,, or (z)).
Because μlg and dg are deterministic functions, the predictive distribution the surrogate ğ is then given by
0(x) f(x))o(()) + no(x) ((((())),,(z))) ~ (2(
(r)))(())))
Using this predictive distribution, we can select the hyperparameter configuration maximizing the Expected Improvement (EI) (Mockus et al. 1978) of g(z). The EI can then be defined in closed form as
(I(r) El max g0. [(x))]
(((())(()))) + (((()))) where v(x) Mg(x)) g(3mmn)
with O and @ being the CDF and PDF the standard normal, respectively. When no observations are available, the empirical CDF F is not defined. Therefore, we warm-start the optimization on the new task by sampling set of No 5 hyperparameter configurations via Thompson sampling, as described above. Pseudo-code is given in Algorithm
Algorithm Gaussian Copula process (CGP)
Learn the parameters 0 of (x(x)) and (1())) on hold-out evaluations D-M by minimizing equation Sample an initial set of evaluations D = {(xx, (I No via Thompson sampling 2 while Has budget do
Fit the GP surrogate f to the observations {(x, )(v)-po(=) (z, y) D}
Sample N candidate hyperparameters . NN from the search space Compute the hyperparameter 2; where i arg max, El(x,,)
Evaluate y: = f(;;) and update = U {(x;, y:)))
end while
4.3 OPTIMIZING MULTIPLE OBJECTIVES
In addition optimizing the accuracy of black-box function, it is often desirable optimize its runtime or memory consumption. For instance, given two hyperparameters with the same ex- pected error, the one requiring fewer resources preferable. For tasks where runtime is avail- able, we use both runtime and error objectives by averaging in the transformed space, e.g., we set x(x) = (x=))(()) + z ziiee (x)), where (x) = (()foor)))) and zlime (x)me()) = vb( ((mm(())) denote the transformed error and time observations, respectively. This allows us to seamlessly optimize for time and error when rumning HPO, so that the cheaper hyperparameter is favored when two hy- perparameters lead to similar expected error. Notice many existing multi-objective methods can potentially be combined with our Copula transformation as an extension, which we believe is an interesting venue for future work.
4

---

Under review as a conference paper at ICLR 2020
tasks
# datasets
hyperparameterss
# evaluations per dataset
available objectives
DeepAR
11
6
220
quantile loss, time
FCNET
9
62208
MSE, time
XGBoost
1-AUC
Table 1: A summary of the three HPO problems we consider.
5 EXPERIMENTS
We considered the problem of tuning three algorithms on multiple datasets: XGBoost (Chen & Guestrin 2016), 2-layer feed-forward neural network (FCNET) (Klein & Hutter, 2019) and the RNN-based time series prediction model proposed in Salinas et al. (017) (DeepAR). We tuned XGBoost on bsvv datasets (Chang & Lin 2011) to minimize 1 AUC, and FCNET on datasets from Klein & Hutter (2019) to minimize the test mean squared error. As for DeepAR, the evaluations were collected on the data provided by Gluon'TSS (Alexandrov et al. 2019), consisting of 6 datasets from the M4 M4-competition (Makridakis et al. 2018) and 5 datasets used in Lai et al. 2017), and the goal is to minimize the quantile loss. Additionaly, for DeepAR and FCNET the runtime to evaluate each hyperparameter configuration was available, and exploiting this objective. More details on the HPO setup are in Table and the search spaces of the three problems is in Table 4 of the appendix. Lookup tables used advocated in Eggensperger et al. 2012), more details and statistics can be found in the appendix.
additional experiments
We compare against number of baselines. We consider random search and GP-based BO as two of the most popular HPO methods. As transfer learning baseline, we consider warm-start GP (Feurer al. 2015). using the best-performing evaluations from all the tasks to warm start the GP on the larget task (WS GP best). As an extension of WS GP best, we apply standardization on the objectives of the evaluations for every task and then use all of them to warm start the GP on the target task (WS GP all). We also compare against two recently published transfer learning methods for HPO: ABLR (Perrone et al. 2018) and a search space-based transfer learning method (Perrone et al.) 2019). ABLR is transfer learning approach consisting of shared neural network across tasks on top of which lies a Bayesian linear regression layer per task. Finally, Perrone et al. (2019) transfers information by fitting a bounding box to contain the best hyperparameters. from each previous task, and applies random search (Box RS) or GP-based BO (Box GP) in the learned search space.
We assess the transfer learning capabilities of these methods in leave-one-taske task-out setting: sequentially leave out one dataset and then aggregate the results for each algorithm. The perfor- mance of each method is first averaged over 30 replicates for one dataset in a task, and the relative improvements over random search are computed on every iteration for that dataset. The relative im- provement for an optimizer (opt) is defined by (Urendom Yopp)) /rrnndomm which is upper bounded by 100%. Notice that all the objectives are in Rt. By computing the relative improvements, we can aggregate results across all datasets for each algorithm. Finally, for all copula-based methods, we learn the mapping to copulas via a 3-layer MLP with 50 units per layer, optimized by ADAM with early-stopping.
5.1 ABLATION STUDY
To give more insight into the components of our method, perform detailed ablation study to investigate the choice of the MLP and compare the copula estimation to simple standardization.
Choice of copula estimators For copula-based methods, we use an MLP to estimate the output. We first compare to other possible options, including a linear model and a k-nearest neighbor esti- mator in a leave-one-out setting: we sequentially take the hyperparameter evaluations of one dataset as test set and use all evaluations from the other datasets as training set. We report the RMSE in Table 5) of the appendix when predicting the error of the blackbox. From this table, it is clear that MLP tends to be the best performing estimator among the three. In addition, low RMSE indicates that the task is close to the prior that we learned on all the other tasks, and we should thus expect transfer learning methods to perform well. As shown later by the BO experiments, FCNET has the lowest RMSE among the three algorithms, and all transfer learning methods indeed perform much better than single-task approaches.
5

---

Under review as a conference paper at ICLR 2020
Homoskedastic and Heteroskedastic noise The proposed Copula estimator (MLP) uses het- eroskedastic noise for the prior. We now compare to a homoskedastic version where we only estimate the mean. The results summarized in Table where average relative improvements over random search across all the iterations and replicates are shown. It is clear that heteroskedas- ticity tends to help most datasets.
Copula transformation and standardization In our method, we map objectives to be normally distributed two steps: first we apply the probability integral transform. followed by Copula transform using the inverse CDF of a Gaussian. To demonstrate the usefulness of such transforma- tion, we compare on each datasets separately. Results forms significantly worse than the Copula transformation, indicating that is not able to address the problem of varying scale and noise levels across tasks. Note that the relative improvement objec- tive is not lower bounded, so that when random search finds very small values the scale of relative improvement be arbitrary large (such for the Protein dataset in FCNET).
simple standardization the objectives where mean and std are computed reported in Table 2 It is clear that standardization per-
task
dataset
TS and
CTB
Bo
CTB
Be
32
std
CGP
No
CGP
lle
DocpAR
electricity
-13.2
0.3
0.8
-15.3
0.4
0.8
exchange-ran rate
-127.2
1.8
2.9
-130.6
3.0
m4-Daily
-58.0
-107.3
m--Hourly
-98.6
-0.8
-0.8
-94.7
0.7
3.0
Momthl
m4 Quarterty
15.6
0.5
0.8
-11.8
0.8
1.0
Weekly
-96.1
-81
m4- Yearly
-14.0
0.4
0.7
13..6
0.8
1.1
solar
-14.1
-8.7
0.8
traffic
-17.3
wiki-rolling
-4.5
0.3
0.3
-466
0.4
0.5
FCNet
naval
20602.7
72.0
78.9
-436844
81.7
82.3
parkinsons
-78.6
27.8
29.6
-96.1
42,1
38.8
protein
-18.8
-9.6
slice
870.6
46.3
53.5
14.2
58.5
$8.9
XGBoost
aba
-0.7
0.0
0.1
-0.3
0.2
0.2
australian
-50.8
-3.44
11.22
0.6
0.5
-12.7
0.6
0.7
heart
70.11
0..1
129.0
39
ijenn
-38.8
-11.7
madelon
-37.9
-0.4
-0.4
-16.2
49
42
spambuse
svmguidel
-28.3
0.8
-0.3
117.0
1.2
1.2
w6a
Table 2: Relative improvements over random search. TS std and GP std respectively using a simple standardization instead of the copula transformation. Ho and He stand for Homoskedastic and Heteroskedastic noise.
5.2 RESULTS
We now compare the proposed methods to other HPO baselines. The results on using only the error information are shown first followed by the results using both time and error information.
Results using only error information We start by studying the setting where only error objectives used learn the copula transformation. Within each task, we first aggregate 30 replicates for each method to compute the relative improvement over random search at every iteration, and then average the results across all iterations. The results are reported in Table 3 showing that CGP is the best method for almost every task except XGBoost. In XGBoost, there are several tasks on which methods without transfer learning perform quite well. This not surprising as we observe in an ablation study on copula estimators (see Sa the appendix) that some tasks in XCBoost have relatively high test errors, implying that the transferred prior will not help. In those tasks, CGP usually the runner-up method after standard GP. We also report the results at iteration 10, 50 and 100 in the Tables and in the appendix where we observe CGP and Box RS are the most competitive methods 10th iteration but at 100 iteration, CGP is clearly the best transfer learning method. This highlights the advantage of being adaptive to the target task our method while making effective transfer in the beginning.

---

Under review as a conference paper at ICLR 2020
task
dataset
ABLR
Box
Box
CGP
CTS
GP
WS
all
RB
beat
DeepAR
electricity
-2.7
-2.1
0.8
0.8
0.2
-2.4
-2.3
eachange-rate
0.5
-0.2
2.1
3,3
2.9
0.6
0.6
0.3
m4-Daily
m4-Hourly
-18.0
-12.1
-2.8
3.0
-0.8
-1.3
-9.8
-11.7
m+4MMonthyy
-0.7
1,0
0.6
0,2
0.2
0.6
m4- rur-Quarterly
-0.4
-0.1
0.6
0.8
0.0
m4 Weekly
338
-3.2
0.5
0.6
0.4
0.1
229
3.2
m4-Yearly
-0.1
0.7
0.5
0.3
solar
-0.3
0.3
0.8
1.1
0.5
0.3
-0,5
-0.1
traffic
-0.3
wiki-rolling
0.1
0.1
0.2
0,5
0.3
0.1
0.3
0.0
PCNet
naval
57.1
70.6
80.6
82.3
78.9
-64.3
61.2
65.9
parkinsons
14.1
27.1
27.3
38.5
29.4
20.2
27.6
proicin
0.3
6.8
5.8
8,3
6.6
3.3
6.3
5,9
XGBoost
afu
0.1
0.1
0.1
0.1
0.2
0.1
australian
-1.2
3.1
3.7
0.4
1.7
1.5
0.8
german.oumer
-1.3
0.2
heart
1.5
1.5
2.3
5.1
0.9
288
2.11
4.4
ijennl
madelon
2.4
3.5
1.7
4.2
-0.4
4.4
0.3
1.4
spambase
-299
1.0
-0.9
18
0.3
0..5
symguidel
-3.2
-0.3
1.1
0.4
0.4
w6a
0.8
-25
3,4
0.5
388
-0.8
0.9
Table 3: Relative improvements over random search averaged over all the iterations. methods are highlighted bold.
The best
-1.0
electricity, leokup
wt- lonkup
sptiniter
-1.7
-1.4
10.8
-1.6
-1.6
best
best
-1.8
18
19
18
16'
16
iteration
iteration
(a) Comparison on electricity for DeepAR,
(b) Comparison on m4 -4-meaaa -eerrrl for DeepAR,
prett_parkinsors_Poe lootup
cmet_prsteinlleek
op LiniiasT
0.25
-0.75
-1.00
Coadta R5
Coada 85
-1.25
-1.50
-1.75
Dox
Box
2.000
-2.00
iteration
iteration
(c) Comparison on park nsons for FCNET.
(d) Comparison on protein for FCNET.
xgboost resource
xpboost aax resource Loskup
-1.5
-1.6
Coada
-1.0
Cosuta
-177
Cocule
ADUS
-1.9
Doc 15
-1.6
Soc
Soc
-2.0
10'
18'
18a
161
(e) Comparison w for XGBoost.
(f) Comparison heart for XGBoost.
Figure 1: Results using only error information, with the current optimum plotted against the number of used evaluations.
We also show results two example datasets from each algorithm in Figure reporting confidence intervals obtained via bootstrap. Note that the performance of the methods in the examples for DeepAR and XGBoost exhibit quite high variations, especially in the beginning of the BO. We

---

Under review as a conference paper at ICLR 2020
-0.75
8.80
-0.45
4.05
A0.1
best
4..00
44.55
40.00
-0.95
Evellatiion time
breluatiion
(a) Comparison on dataset m4 1an4-Years Yearly over time.
(b) Comparison on dataset protein over time.
Figure 2: Two example tasks using both error and time information, with the current optimum plotted against time.
conjecture this is due to an insufficient number of evaluations in the lookup tables. Nevertheless, the general trend is that CTS and CGP outperform all baselines, especially in the beginning of the BO. can also be observed that CCP performs at least on par with the best method at the end of the BO. Box RS is also competitive at the beginning, but as expected fails to keep its advantage toward the end of the BO.
Results using both error and time information We then studied the ability of the copula-based approaches to transfer information from multiple objectives. Notice it is possible to combine Copula transformation with other multi-objective BO methods and we will leave this direction as future work. We show two example tasks in DeepAR and FCNET in Figure 2 where we fix the total number of iterations and plot performance against time with standard error. To obtain distributions over seeds for method, only consider the time range where 20 seeds are available ,whcch explains why methods can start and end at different times. With the ability to leverage training time information, the copula-basee approaches have clear advantage over all baselines, especially the beginning of the optimization.
We also report aggregate performance over all the tasks in Table 6 in the appendix. Due to the different methods finishing the optimization at different times, we only compare them up to the time taken by the fastest method. For each method we first compute an average over 30 replicates, then compute the relative improvement over random search, and finally average the results across all time points. The copula based methods converge to a good hyperparameter configuration significantly faster than all the considered basclines. Note that we obtain similar results as for Hyperband-str methods (Li et al. 2016), where the optimization can start much earlier than standard HPO, with the key difference that we only require a single machine.
6 CONCLUSIONS
We introduced new class of methods to accelerate hyperparameter optimization by exploiting eval- uations from previous tasks. The key idea was to leverage a semi-parametric Gaussian Copula prior, using it to account for the different scale and noise levels across tasks. Experiments showed that we considerably outperform standard approaches to BO, and deal with heterogeneous tasks more robustly compared to number of transfer learning approaches recently proposed in the literature. Finally, we showed that our approach can scamlessly combine multiple objectives, such as accuracy and runtime, further speeding up the search of good hyperparameter configurations.
A number of directions for future work are open. First, we could combine our Copula- HPO strategies with Hyperband-sty optimizers (Li et al. 2016). In addition, could generalize our approach to deal with settings in which related problems are not limited to the same algorithm run over different datasets. This would allow for different hyperparameter dimensions across tasks, or perform transfer learning across different black-boxes.
REFERENCES
A. Alexandrov, K. Benidis, M. Cohlke-Schnnide V. Flunkert, J. Gasthaus, T. Januschowski, D, C. Maddix, S. Rangapuram, D. Salinas, J. Schulz, L.. Stella, A. C. Türkmen, and Y. Wang. Gluon'TS:
8

---

Under review as a conference paper at ICLR 2020
Probabilistic Time Series Modeling in Python. arXiv preprint arXi:: 2019.
Alec Anderson, Sebastien Dubois, Alfredo Cuesta-Infanc and Kalyan Veeramachaneni. Sample, estimate, tune: Scaling bayesian auto-tuning of data science pipelines. In 2017 IEEE International Conference Data Science and Advanced Analytics (DSAA), pp. 361-372. IEEE, 2017.
Rémi Bardenet, Mátyás Brendel, Balázs Kégl, and Michele Sebag. Collaborative hyperparameter tuning. In Proceedings of the International Conference on Machine Learning (ICML), pp. 199-
207, 2013.
Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1-27:27, 2011.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. 2016.
Eggensperger, Hutter, HH Hoos, and Leyton-brown. Efficient benchmarking of hyperparam- eter optimizers via surrogates background: hyperparameter optimization. In Proceedings of the 29th AAAI Conference on Artificial Intelligence, pp. 1114-1120, 2012.
Matthias Feurer, Springenberg, and Frank Hutter. Initializing Bayesian hyperparameter opti- mization via meta-learning. In Proceedings of the Twenty-Ninth AAAI Conference Artificial Intelligence, 2015.
Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for Bayesian opti- mization using wanking-weighted Gaussian process ensembles. In ICML 2018 AutoML Workshop, July 2018.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D Sculley. Google Vizier: A service for black- optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1487-1495, 2017.
Aaron Klein and Frank Hutter. Tabular benchmarks for joint architecture and hyperparameter opti- mization. arXiv preprint 0005:v:9050004999 2019.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. CoRR, abs/1703.07015, 2017.
Ho Chung Leon Law, Peilin Zhao, Junzhou Huang, and Dino Sejdinovic. Hyperparameter learning via distributional transfer. Technical report, preprint
2018.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyper- band: A novel bandit-based approach to hyperparameter optimization. Technical report, preprint 10ti:16033066600, 2016.
Han Liu, John Lafferty, and Larry Wasserman. The Nonparanormal: Semiparametric Estimation of High Dimensional Undirected Graphs. 10:2295-233 2328, 2009. ISSN 1532-4435, doi: 10.1016/
0006-291X(91)91266)
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The M4 Competition: Re- sults, findings, conclusion and way forward. International Journal of Forecasting, 34(4):802-80) 2018. doi: 10. 10006//.ifforecast20011
Jonas Mockus, Vytautas Tiesis, and Antanas Zilinskas. The application of bayesian methods for seeking the extremum. Towards global optimization, 2(1 129):22 1978.
Valerio Perrone, Rodolphe Jenatton, Matthias Seeger, and Cédric Archambeau. Scalable hyper- parameter transfer learning. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Valerio Perrone, Huibin Shen, Matthias Seeger, Cédric Archambeau, and Rodolphe Jenatton. Learn- ing search spaces for bayesian optimization: Another view of hyperparameter transfer learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Matthias Poloczek, Jialei Wang, and Peter Frazier. Warm starting Bayesian optimization, In Winter Simulation Conference (WSC), 2016, pp. 770-781. IEEE, 2016.

---

Under review as a conference paper at ICLR 2020
Carl Rasmussen and Chris Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
David Salinas, Valentin Flunkert, and Jan Gasthaus. Deepar: Probabilistic forecasting with autore-
gressive recurrent networks. CoRR, abs/ 100.04410,, 2017.
Nicolas Schilling, Martin Wistuba, and Lars Schmidt-hieme. Scalable hyperparameter optimiza- tion with products of Gaussian process experts. In Joint European Conference on Machine Learn- ing and Knowledge Discovery in Databases, pp. 33 48. Springer, 2016.
Edward Snelson, Zoubin Ghahramani, and Carl E. Rasmussen. Warped gaussian processes. In S. Thrun, K. Saul, and B. Schölkopf (eds.), Advances in Neural Information Processing Systems (NeurIPS), pp. 337-344, MIT Press, 2004.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable Bayesian optimization using deep neural networks. In Proceedings of the International Conference on Machine Learning (ICML), 2171-2180, 2015.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust Bayesian neural networks. In Advances Neural Information Processing Systems (NeurIPS), pp. 4134-4142, 2016.
Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task Bayesian optimization. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2004-2012, 2013.
William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of samples. Biometrika, 25(3/4):285225)) 294,, 1933. ISSN 00063444.
Andrew Gordon Wilson and Zoubin Ghahramani. Copula processes. Proceedings of the 23rd International Conference Neural Information Processing Systems Volume 2, NeurlPS 2010, pp. 2460-2468, USA, 2010. Curran Associates Inc.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Hyperparameter search space pruning-a a new component for sequential model-based hyperparameter optimization. In Machine Learning and Knowledge Discovery in Databases, pp. 104-119. Springer, 2015a.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thiemee Learning hyperparameter optimiza- tion initializations. In Data Science and Advanced Analytics (DSAA), 2015. 36678 2015. IEEE International Conference on, pp. 1-10. IEEE, 2015b.
Martin Wistuba, Nicolas Schilling, and Lars Samiddt-Thieme. Scalable gaussian process process-based transfer surrogates for hyperparameter optimization. Machine Learning, 107(1):43-78, Jan 2018. ISSN 1573-0565. doi: 10. 10.10077/119944-07--56644 URL 0ttp:////000...1111111007777
610994-017-5684-9
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
Algorithm 2 Copula Thompson sampling (CTS)
Learn the parameters of (a( and (x(2) on hold-out evaluations D by minimizing equation while Has budget do
Sample N candidate hyperparameters EN from the search space
Draw & (p)(())) oh (zx)) for I....,N
Evaluate fæz; where i = arg min, 24
end while
A.1 LOOKUP TABLES
To speed experiments we used lookup table approach advocated in Eggensperger et al. (2012) which proposed use an extrapolation model built on pre-generated evaluations to limit the number of blackbox evaluations, thus saving significant amount of computational time. However, the ex- trapolation model can introduce noise and lead inconsistencies compared to using real blackbox evaluations. As result, in this work we reduced BO to the problem of selecting the next hyperpa- rameter configurations from a fixed set that has been evaluated advance, so that no extrapolation error is introduced.
All evaluations were obtained by querying each algorithm at hyperparameters sampled (log) uni- formly at random from their search space described in Table The CDF on the error objectives given in Figure
DecpAR
FONET
XcBoost
1.3
electricity
1.3
1.8
exchange-rate
parkcnsors
0.4
0.4
0.4
Tratfic
0.2
(a)
(b)
(c)
Figure 3: CDF of error metrics for the three tasks. Every line represent one dataset. The metrics are scaled first by min-max scaling and the x-axis is log scaled.
Results different iterations. We plot the improvement random research for all the meth- ods at iteration 10, 50 and 100 at Table 7 andd respectively. In short, at 10th iteration, transfer learning methods, especially our CGP and Box RS, performed much better than GP. But, when look- ing at results at 50 and 100 iterations, CGP outperforms clearly all other transfer methods because of improved adaptivity.
More details on prior MLP architecture. The MI. used to regress 44 and ag consists of 3 layers with 50 nodes, each with a dropout layer set to 0.5. The learning rate is set to 0.01, batch size to 64 and we optimize over 100 gradient updates times, lowering the learning rate by 10 each time.
11

---

Under review as a conference paper at ICLR 2020
tasks
hyperperameceer
search space
type
scale
layers
[1. 3]
integer
linear
# colls
[10, 120]
integer
lincar
DeepAR
learning
dropout
[10-
110- 0.11
continucos
continuous
log10
log10
context.ennth..oe lenghh_ihhttttii @ bathes per epoch
[10-1_ 1.4]
continucus
log10
110. 10
inleger
log10
round
[2
integer
log2
continueus
lincar
12 20 20
continuous
log2
min child weigh
[2 12-$,2% 2
continuous
log2
XGBooat
max Jlepth
[2,
integer
log2
subsample
[0.5,
continucos
lincar
colsample. bytree
[0.3, 1]
continuous
lincar
lambdda
[2 10, 2*]
continucus
Ing2
alpha
20 2*)
continuous
log2
initial_lr Ir
{0.005, 0.001, 0.05, 0.01, 0.05, 0.1
calegorical
batch_size
(8, 16, 32, 64
categorical
Ir schodule
{cosinc, fix
categorical
activation layer
(relit, tanh
categorical
FONET
activation layer
(relu, tanh
categorical
layer
{16. 32, 64, 128. 256, 512)
calegorical
size layer
[16, 32, 64, 128, 256. 512)
calegorical
dropout layer
(0.0.0.3, 0.6)
categorical
dropout layer 2
(0.0. 0.0)
categorical
Table 4: A summary of the search spaces for the three algorithms.
task
dataset
KNN
KNN 20
KNN.5
Linear
MI.P
DeepAR
electricity
0.801
0.740
exchange-rate rate
0.842
0,842
0.842
0.783
0.780
Deily
0.804
0.804
0.792
m4-Hourly
0.960
0,960
0.960
0.948
0.884
nal-Monthly
0.783
0.783
0.783
0.762
0.750
m4 Quanterly
0.868
0,868
0.868
0.792
0.773
m4- ne--Weekly
0.776
0.776
0.776
0.754
0.733
Yearly
0.759
solar
0.963
0.963
0.963
0.875
0.812
traffic
0.885
0.885
0.885
0.850
0.829
wiki-rolling
0.904
0.904
0.904
0.868
0.826
FCNet
naval
0.509
0.509
0.309
0.602
0.491
parkinsoms
protein
0.505
0.505
0.505
0.607
0.497
slice
0.564
0.564
0.564
0.559
0.555
XGBoost
aba
1.091
1,091
1.091
1.067
1.040
sustralian
0.827
0.827
0.827
0.873
0.758
german.numer
heart
0.818
0.818
0.818
0.793
0.702
ijenal
0.951
0.951
0.951
0.936
0.917
mudelon
0.908
0.908
0.908
0.887
0.834
spambase
0.931
0.931
0.931
0.950
0.818
svmguidel
0.849
0.849
0.849
0.912
6.798
w6a
1.039
1,039
1.039
1.054
1.003
Table 5: RMSE comparison for prior estimators when predicting the blackbox error given its param- eters.
12

---

Under review as a conference paper at ICLR 2020
dataset
ASLR
Box
CGP
GP
all
best
DeepAR
electricity
-19.7
-17.2
10.0
10.3
0..3
18.5
-1.00
exchange-rate
38.7
39.1
0.2
-0.3
7.7
m4-Daily
-17.5
-15.7
20.1
20.6
-1.7
-16.9
-14.8
m4-Houurl
102.0
m4-Monthly
3.7
6.7
5.0
4.6
-0.6
6.5
7.6
m4 uut-Quatterly
-10.3
-9.6
-0.4
-0.6
-2.
-9.6
md Weekly
26.7
-24.6
-1.4
-0.6
23,0
23.5
Yearly
-3.8
-3.1
2.1
-0.1
-3.7
-2.9
solar
3.4
traflic
336.7
-34.5
3.6
3.8
0.4
-33.7
-33.6
wiki-rolling
-1.0
FCNel
naval
80.6
69.4
90.1
89.5
-15.00
45.7
58.8
parkinsons
338.5
30.3
53.2
46.4
22.3
27.0
26.9
protein
-6.9
10.1
12.7
10.8
12.6
slice
-14.5
6.1
83.3
80,9
22.4
8.1
8.4
Table 6: Relative improvements over random search averaged over time.
lask
dataset
ABLR
Box GP
Box RS
Copula-GP
Copula-TS TS
GP
WS GP all
WS GP best
DecpAR
cloctricity
4.1
-1.2
14
1.0
0.1
-0.1
-1.3
-1.4
exchange-rate
m4 Daily
-3.4
-1.3
L.0
0.5
0.6
0.3
-1.1
0.6
Mouroy
m4-Mouthly
-1.5
1.0
1.6
1.6
1.5
0.2
1,5
1.7
m4- Quarterly
-1.3
0.0
D.O
1.0
0.7
0.0
0.7
-0.1
m4- Wockly
0.2
m4- Yearly
-0.8
0.5
D.I
1.3
0,7
0.3
0.1
0.3
traffic
-3.0
-1.1
LO
0.2
-0.1
0.1
40.5
-0.5
wiki-rolling
0.1
FCNe
79.8
95.8
71.0
parkinsons
5.9
32.7
42.9
46,9
45.1
-3.3
34,9
24.6
slice
16.7
59.5
74.0
80.0
78.9
-2.5
59.7
59.7
XGBoosk
0..1
0.1
4.2
0.2
0.2
0..
australian
-7.9
1.6
5.3
1.2
-1.5
german
nemean numer
-0.7
0.6
L6
0.9
0.6
-0.2
0.7
-0.3
ijcnn
13.99
-0.4
9.1
8.0
5.2
2.9
5.9
6.6
madelon
3.0
-1.2
1.0
1.2
spambuse
-5.5
-2.1
0.2
-2.7
-1.6
-1.9
-0.6
-0.5
svmguide
-2.9
0.9
2.0
1.2
0.0
wha
1.6
2.2
Table Relative improvements over random search iteration 10.
task
dataset
ABLR
Box GP
Box RS
Copula-GP
Copula-TS
GP
WS GP all
WS GP best
DeepAR
electricity
0.3
0.1
exchange-rate
1.8
-0.2
-0.1
2.1
1.3
0.6
1.2
1.0
m4-Daily
-0.2
m4- Hourly
-0.33
-4.2
-4.7
1.9
-3.8
-4.1
-2.0
-3.9
m4-Monthly
-0.3
-0.1
m4 Quarrtterty
-0.5
-0.1
0.4
0.1
022
Weekly
.0.1
1.0
0.2
0.4
0.5
m4. Yearly
0.2
0.4
solar
0.2
0.4
0.4
1,0
0.4
0.5
40.7
-0.2
traffic
wiki rolling
40.2
-0.1
4.2
0.1
-0.1
0.0
0.0
00.5
PCNet
naval
$9.6
72.7
80.4
81.5
77.9
46.2
64.4
70.9
parkinsons
16.9
26.1
23.1
33.5
24.4
24.9
19.5
27.5
protein
0.0
7.0
4.2
6.8
5.1
3.2
5.3
3.8
XGBoost
40.1
0.1
0.1
0.2
40.1
0.2
4.11
-0.1
australian
-0.9
1.3
-0.3
1.8
german
-0.5
beart
2.6
2.5
233
3.1
1.8
3.7
-0.8
6.6
ijennl
madelon
3.6
3.5
0.9
4.3
-0.1
4,9
0.5
1.0
spambase
-0.6
svmguide!
-3.1
0.9
1.2
-0.3
1.3
0.6
0.7
wha
1.9
-23
4.8
-0.6
Table 8: Relative improvements over random search at iteration 50.
13

---

Under review as a conference paper at ICLR 2020
task
dataset
ABLR
Box GP
Box RS
Copola-GP
Copola-TS
GP
WS GP all
WS best
DeepAR
electricity
exchonge-rate
1.0
-0.4
-0.4
0.8
0.1
0.3
0.4
0.6
m4-Daily
-0.5
-0.1
-0.1
-0.3
-0.3
md m4-Hourly
-6.2
-0.6
-5.6
4.4
-1.5
2.2
3.8
-1.5
m4-Monthly
-0.1
0.1
-0.2
0.3
-0,1
0.2
0.1
0.2
au--Quareety
md Weekly
0.2
0.2
0.1
0.4
0.1
0.2
0.2
0.1
Yearly
solar
0.3
0.0
0.2
0.7
0,1
0,3
-0.4
-0.3
traflic
-0.2
-0.1
-0.1
0.2
0.0
wiki-rolling rolling
0.0
40.1
-0.1
PCNet
naval
41.1
58.2
63.0
66.0
60.8
26.1
54.1
51.5
parkinsons
18.0
27.3
protcin
2.4
6,3
344
6.3
4.8
3.5
588
slice
27.3
30.9
42.4
36.5
18.2
31.6
24.2
XGBoost
03
australian
0.2
1.8
1.8
4,3
0.0
3.5
1.1
1.2
german numer
-02
beart
1.5
0.9
40.3
3.3
-1.3
244
-3.5
3,6
ijennl
-2.7
4.0
1.7
madelon
3.3
4.0
-0.1
48
spambase
-1.6
1.4
-0.7
2.2
-1,4
3.0
-0.7
-0.9
svmguidel
-0.3
wha
1.5
1.2
-3.3
4.6
0.1
4.6
-0.6
2.0
Table 9: Relative improvements over random search at iteration 100.
14


Reviews:
[{"review_id": "rJgxW_iQ5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a new Bayesian optimization (BO) based hyperparameter searching method that can transfer across different datasets and different metrics. The method is to build a regression model based on Gaussian Copula distribution, which maps from hyperparameter to metric quantiles. The paper shows that by leveraging this estimation using some specific (sampling) strategies, it is able to improve over other BO methods.\n\nThe high-level idea of this paper seems sound to me -- that improves standard BO to generalize across datasets and metrics by learning a mapping between the space of hyperparameters and metrics. While I am not an expert in this area, the derivation looks sound to me, and the evaluation results of this paper are comprehensive to show that CGP seems to outperform a number of methods.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HkeNJPwAFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of black-box hyperparameter optimization when multiple related optimization tasks are available simultaneously, performing transfer learning between tasks. Different tasks correspond to different datasets and/or metrics. Gaussian copulas are used to synchronize the different scales of the tasks.\n\nI have several reservations with this paper. First and foremost, it seems to be lacking a fair and trivial baseline (I will describe it below) that justifies the apparently unnecessary complicated path followed in this paper. Second, there are a few small incorrect or improperly justified technical details throughout the paper.\n\n\n1) Mistaken/unjustified technical details:\n\n- In equation 1, the last term seems to be constant. For each task, the function psi is not parametric, so its gradient is also not parametric and the input is the inverse of z, i.e., y, which is also fixed. So why is it included in the cost function? This sort of probabilistic renormalization is important in e.g. warped GPs because the transformation is parametric. In this case, I don't see the point. It can be treated as a normalization of the input data, prior to its probabilistic modeling.\n\n- Before equation 1, the text says \"by minimizing the Gaussian negative log-likelihood on the available evaluations (x, z)\" But then, equation 1 is not the NLL on z but on y.\n\n- In section 4.2 the authors model the residuals of the previous model using a powerful Matern-5/2 GP. Why modeling the residuals this way and not the observations themselves? The split of modeling between a parametric and non-parametric part is not justified.\n\n- One of the main points of the variable changes is to normalize the scales of the different tasks. However, equations 1 adds together the samples of the different tasks (which, as pointed out by the authors might have different sizes). Even if the scales of the outputs are uniform, the different dataset sizes will bias the solutions towards larger datasets. Why would that be a good thing? This is not mentioned and doesn't seem correct: there should not be a connection between a dataset size and the prior influence of the corresponding task. In fact, this will have the same effect as if the cost had different scales for different tasks, which is precisely the problem that the authors are trying to avoid.\n\n\n2) Trivial baseline\n\nGiven that the authors are trying to aggregate information about the optimal hyperparameters from several tasks, they should not compare with single-task approaches, but with the simplest way to combine all the tasks. For instance:\n    a) Normalize the outputs of every task. This can be accomplished in the usual way by dividing by the standard deviation, or even better, by computing the fixed transform z = psi(y), separately for each task.\n    b) Collect the z of all tasks and feed them into an existing GP black-box Bayesian optimizer.\n\nThis is a very simple way to get \"transfer learning\" and it's unclear that the extra complexities of this paper (copulas, changes of variable with proper renormalization when the transformation is parameter free, etc) are buying much else.\n\n\nMinor improvements:\n\n- Page 2: \"is the output of a multi-layer perceptron (MLP) with d hidden nodes\" Is d really the number of hidden nodes of the MLP? Or the number of outputs? Given that d is also the size of w, it seems it's actually the latter.\n\n- Explain why the EI approach is used for the second model (with the GP), but not for the first model.\n\nEdit after rebuttal:\n“The term is not constant over z” -> Sure, it’s not constant over z. But z is constant. So the term is constant.\n\n“The NLL is minimized in z and there is indeed no y in equation 1.” -> Sure, there’s no y in the equation, that’s correct. But it is still the NLL of y, and not the NLL of z.\n\nAbout the new baseline: Instead of simply renormalizing using mean and standard deviation, I suggested above using the same z=psi(y) that is used in the paper for the normalization. Is that where the advantage of the proposed method is coming from?\n\n\"Note that this is orthogonal to the scale issues we focus on: larger tasks will have larger gradient contributions but the scaling we propose still allows us to learn tied parameters across tasks as their scales are made similar. \" Both issues affect the scaling of the task, so I don't see how they can be orthogonal. Their scales are not made similar precisely because of the different sample sizes.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkecWwptYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new way of normalizing the labels of the meta-data. They propose a Thompson sampling strategy as a new hyperparameter optimization warmstarting strategy and an optimization method that leverages transfer learning.\n\n\nThe transfer learning baselines are surprisingly weak and show no improvement over the random search baseline for two of your tasks. You should consider stronger baselines. One interesting work is \"Collaborative hyperparameter tuning\" by Bardenet et al. which overcomes the problem of different scales by considering the problem as a ranking problem. You discussed further works in your related work. Another interesting work based on GPs is \"Scalable Gaussian Process based Transfer Surrogates for Hyperparameter Optimization\".\nThe warm-start GP currently seems to be your strongest baseline. However, I have doubts that it is implemented correctly. You say that you estimate the most similar dataset and then evaluate its best 100 hyperparameter configuration. First of all, I don't understand why you decided to choose 100 (DeepAR only has 220 configurations in total!). Second, this is not how this method works. Instead, you estimate the k most similar tasks and evaluate its best hyperparameter configuration. In your case k is upper bounded by 10 (number of tasks). This is notably smaller than 100 and gives more time to the Bayesian optimization which will likely improve your results.\n\n\nYou observe worse results on XGBoost and good ones on FCNet. You refer to Table 6 and connect it to the RMSE. This might be true but I have a much simpler explanation: the search space of FCNet is orders of magnitudes larger and it contains many configurations that lead to very high MSE and only few with low. Therefore, a random search will on average provide poor results where a warmstarted search will obtain decent results for the first iterations. XGBoost is less sensitive to hyperparameters such that the overall variance in the losses is smaller. Maybe you can provide some insights into the complexity of the optimization tasks (plot the distributions )and add it to the appendix?\n\n\nTable 6 contains more datasets than Table 2. Why did you drop some tasks?\n\n\nThe aggregated results in Table 2 are nice but actually we are interested in the outcome after the search after a given budget. Can you add such a table?\n\n\nI have few suggestions to improve the readability of the paper:\nThat there is a choice of copula estimators is mentioned at the very end of the paper. Can you add it to the section where you describe them first? In section 5.1 you already argue by referring to Table 6. However, Table 6 is explained first in section 5.2 which makes it hard to follow your argumentation.\nYou use the term \"metric\" to refer to objectives. This is confusing, you might consider changing this. You propose to use scalarization to address the multi-objective optimization problem. Why would an average of both objectives be the optimal solution? Does the unit you use to measure the time matter? What if you use machine learning algorithms that scale super-linear in the number of data points? How is this novel and why can't your baselines employ the same idea? A discussion of related work on autoML for multiple objectives is missing.\nThe second paragraph of section 5 is confusing. You cite some work, discuss it and then conclude that your setup is entirely different. Would any information be lost if you say that you precomputed the values?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the problem of transferring learning between tasks when performing Bayesian hyperparameter optimization. In this setting, tasks can correspond to different datasets or different metrics. The proposed approach uses Gaussian copulas to synchronize the different scales of the considered tasks and uses Thompson Sampling from the resulting Gaussian Copula Process for selecting next hyperparameters.

The main weakness of the paper resides in the concerns raised about the experiments. First, the results are hard to interpret, leading to a misunderstanding of performances. Moreover, the considered baselines may not be adapted (they may be trivial). This might be due to a misunderstanding of the paper, which would align with the third major concern, that is the lack of clarity. These points could be addressed in a future version of the work, but it would need to be reviewed again and therefore would be too late for the current camera-ready.

Hence, I recommend rejecting this paper.
Decision: Reject