Title: A COPULA APPROACH FOR HYPERPARAMETER TRANS-FER LEARNING

Abstract: Bayesian optimization (BO) is a popular methodology to tune the hyperparameters of expensive black-box functions. Despite its success, standard BO focuses on a single task at a time and is not designed to leverage information from related functions, such as tuning performance objectives of the same algorithm across multiple datasets. In this work, we introduce a novel approach to achieve transfer learning across different datasets as well as different objectives. The main idea is to regress the mapping from hyperparameter to objective quantiles with a semi-parametric Gaussian Copula distribution, which provides robustness against different scales or outliers that can occur in different tasks. We introduce two methods to leverage this estimation: a Thompson sampling strategy as well as a Gaussian Copula process using such quantile estimate as a prior. We show that these strategies can combine the estimation of multiple objectives such as runtime and accuracy, steering the optimization toward cheaper hyperparameters for the same level of accuracy. Experiments on an extensive set of hyperparameter tuning tasks demonstrate significant improvements over state-of-the-art methods.

1. INTRODUCTION:

Tuning complex machine learning models such as deep neural networks can be a daunting task. Object detection or language understanding models often rely on deep neural networks with many tunable hyperparameters, and automatic hyperparameter optimization (HPO) techniques such as Bayesian optimization (BO) are critical to find the good hyperparameters in short time. BO addresses the black-box optimization problem by placing a probabilistic model on the function to minimize (e.g., the mapping of neural network hyperparameters to a validation loss), and determine which hyperparameters to evaluate next by trading off exploration and exploitation through an acquisition function. While traditional BO focuses on each problem in isolation, recent years have seen a surge of interest in transfer learning for HPO. The key idea is to exploit evaluations from previous, related tasks (e.g., the same neural network tuned on multiple datasets) to further speed up the hyperparameter search.

A central challenge of hyperparameter transfer learning is that different tasks typically have different scales, varying noise levels, and possibly contain outliers, making it hard to learn a joint model. In this work, we show how a semi-parametric Gaussian Copula can be leveraged to learn a joint prior across datasets in such a way that scale issues vanish. We then demonstrate how such prior estimate can be used to transfer information across tasks and objectives. We propose two HPO strategies: a Copula Thompson Sampling and a Gaussian Copula Process. We show that these approaches can jointly model several objectives with potentially different scales, such as validation error and compute time, without requiring processing. We demonstrate significant speed-ups over a number of baselines in extensive experiments.

The paper is organized as follows. Section 2 reviews related work on transfer learning for HPO. Section 3 introduces Copula regression, the building block for the HPO strategies we propose in Section 4. Specifically, we show how Copula regression can be applied to design two HPO strategies, one based on Thompson sampling and an alternative GP-based approach. Experimental results are given in Section 5 where we evaluate both approaches against state-of-the-art methods on three algorithms. Finally, Section 6 outlines conclusions and further developments.

2. RELATED WORK:

A variety of methods have been developed to induce transfer learning in HPO. The most common approach is to model tasks jointly or via a conditional independence structure, which has been been explored through multi-output GPs (Swersky et al., 2013) , weighted combination of GPs (Schilling et al., 2016; Wistuba et al., 2018; Feurer et al., 2018) , and neural networks, either fully Bayesian (Springenberg et al., 2016) or hybrid (Snoek et al., 2015; Perrone et al., 2018; Law et al., 2018) . A different line of research has focused on the setting where tasks come over time as a sequence and models need to be updated online as new problems accrue. A way to achieve this is to fit a sequence of surrogate models to the residuals relative to predictions of the previously fitted model (Golovin et al., 2017; Poloczek et al., 2016) . Specifically, the GP over the new task is centered on the predictive mean of the previously learned GP. Finally, rather than fitting a surrogate model to all past data, some transfer can be achieved by warm-starting BO with the solutions to the previous BO problems (Feurer et al., 2015; Wistuba et al., 2015b) .

A key challenge for joint models is that different black-boxes can exhibit heterogeneous scale and noise levels (Bardenet et al., 2013; Feurer et al., 2018) . To address this, some methods have instead focused on search-space level, aiming to prune it to focus on regions of the hyperparameter space where good configurations are likely to lie. An example is Wistuba et al. (2015a) , where related tasks are used to learn a promising search space during HPO, defining task similarity in terms of the distance of the respective data set meta-features. A more recent alternative that does not require meta-features was introduced in Perrone et al. ( 2019), where a restricted search space in the form of a low-volume hyper-rectangle or hyper-ellipsoid is learned from the optimal hyperparameters of related tasks. Rank estimation can be used to alleviate scales issues however the difficulty of feeding back rank information to GP leads to restricting assumptions, for instance (Bardenet et al., 2013) does not model the rank estimation uncertainty while (Feurer et al., 2018) uses independent GPs removing the adaptivity of the GP to the current task. Gaussian Copula Process (GCP) (Wilson & Ghahramani, 2010) can also be used to alleviate scale issues on a single task at the extra cost of estimating the CDF of the data. Using GCP for HPO was proposed in Anderson et al. (2017) to handle potentially non-Gaussian data, albeit only considering non-parametric homoskedastic priors for the single-task and single objective case.

3. GAUSSIAN COPULA REGRESSION:

For each task denote with f j : R p → R the error function one wishes to minimize, and with D = {x i , y i } N i=1 the evaluations available for an arbitrary task. Given the evaluations on M tasks D M = 1≤j≤M {x j i , y j i } Nj i=1 , we are interested in speeding up the optimization of an arbitrary new task f , namely in finding arg min x∈R p f (x) in the least number of evaluations. One possible approach to speed-up the optimization of f is to build a surrogate model f (x). While using a Gaussian process is possible, scaling such an approach to the large number of evaluations available in a transfer learning setting is challenging. Instead, we propose fitting a parametric estimate of fθ (x) distribution which can be later used in HPO strategies as a prior of a Gaussian Copula Process.

A key requirement here is to learn a joint model, e.g., we would like to find θ which fits well on all observed tasks f j . We show how this can be achieved with a semi-parametric Gaussian Copula in two steps: first we map all evaluations to quantiles with the empirical CDF, and then we fit a parametric Gaussian distribution on quantiles mapped through the Gaussian inverse CDF.

First, observe that as every y i comes from the same distribution for a given task, the probability integral transform results in u i = F (y i ), where F is the cumulative distribution function of y. We then model the CDF of (u 1 , . . . , u N ) with a Gaussian Copula:

C(u 1 , . . . , u N ) = φ µ,Σ (Φ -1 (F (y 1 )), . . . , Φ -1 (F (y N ))),

where Φ is the standard normal CDF and φ µ,Σ is the CDF of a normal distribution parametrized by µ and Σ. Assuming F to be invertible, we define the change of variable z = Φ -1 • F (y) = ψ(y) and g = ψ • f . 1 We regress the marginal distribution of P (z|x) with a Gaussian distribution whose mean and variance are two deterministic parametric functions given by

P (z|x) ∼ N (µ θ (x), σ θ (x)) = N (w T µ h w h (x) + b µ , Ψ(w T σ h w h (x) + b σ )),

where h w h (x) ∈ R d is the output of a multi-layer perceptron (MLP) where 

w h , w µ ∈ R d , b µ ∈ R, w σ ∈ R d , b σ ∈ R

EQUATION

with SGD. Here, the term ψ (ψ -1 (z)) accounts for the change of variable z = ψ(y). Due to the term ψ (ψ -1 (z)), errors committed where the quantile function changes rapidly have larger gradient than when the quantile function is flat. Note that while we weight evaluations of each tasks equally, one may alternatively normalize gradient contributions per number of task evaluations.

The transformation ψ requires F , which needs to be estimated. Rather than using a parametric or density estimation approach, we use the empirical CDF

F (t) = 1 N N i=1 1 yi≤t . While this

estimator has the advantage of being non-parametric, it leads to infinite value when evaluating ψ at the minimum of maximum of y. To avoid this issue, we use the Winsorized cut-off estimator

F (t) ≈    δ N if F (t) < δ N F (t) if δ N ≤ F (t) ≤ 1 -δ N 1 -δ N if F (t) > 1 -δ N

where N is the number of observations of y and choosing δ N = 1 4N 1/4 √ π log N strikes a bias-variance trade-off (Liu et al., 2009) . This approach is semi-parametric in that the CDF is estimated with a non-parametric estimator and the Gaussian Copula is estimated with a parametric approach.

The benefit of using a non-parametric estimator for the CDF is that it allows us to map the observations of each task to comparable distributions as z j ∼ N (0, 1) for all tasks j. This is critical to allow the joint learning of the parametric estimates µ θ and σ θ , which share their parameter θ across all tasks. As our experiments will show, one can regress a parametric estimate that has a standard error lower than 1. This means that information can be leveraged from the evaluations obtained on related tasks, whereas a trivial predictor for z would predict 0 and yield a standard error of 1. In the next section we show how this estimator can be leveraged to design two novel HPO strategies.

4. COPULA BASED HPO:

4.1 COPULA THOMPSON SAMPLING Given the predictive distribution P (z|x) ∼ N (µ θ (x), σ θ (x)), it is straightforward to derive a Thompson sampling strategy (Thompson, 1933) exploiting knowledge from previous tasks. Given N candidate hyperparameter configurations x 1 , . . . , x N , we sample from each predictive distribution zi ∼ N (µ θ (x i ), σ θ (x i )) and then evaluate f (x i ) where i = arg min i zi . Pseudo-code is given in the appendix.

While this approach can re-use information from previous tasks, it does not exploit the evaluations from the current task as each draw is independent of the observed evaluations. This can become an issue if the new black-box significantly differs from previous tasks. We now show that Gaussian Copula regression can be combined with a GP to both learn from previous tasks while adapting to the current task.

4.2. GAUSSIAN COPULA PROCESS:

Instead of modeling observations with a GP, we model them as a Gaussian Copula Process (GCP) (Wilson & Ghahramani, 2010) . Observations are mapped through the bijection ψ = Φ -1 • F , where we recall that Φ is the standard normal CDF and that F is the CDF of y. As ψ is monotonically increasing and mapping into the line, we can alternatively view this modeling as a warped GP (Snelson et al., 2004) with a non-parametric warping. One advantage of this transformation is that z = ψ(y) follows a normal distribution, which may not be the case for y = f (x). In the specific case of HPO, y may represent accuracy scores in [0, 1] of a classifier where a Gaussian cannot be used. Furthermore, we can use the information gained on other tasks with µ θ and σ θ by using them as prior mean and variance. To do so, the following residual is modeled with a GP:

r(x) = g(x) -µ θ (x) σ θ (x) ∼ GP(m(x), k(x, x )),

where g = ψ • f . We use a Matérn-5/2 covariance kernel and automatic relevance determination hyperparameters, optimized by type II maximum likelihood to determine GP hyperparameters (Rasmussen & Williams, 2006) . Fitting the GP gives the predictive distribution of the residual surrogate r(x) ∼ N (µ r (x), σ r (x)).

Because µ θ and σ θ are deterministic functions, the predictive distribution of the surrogate ĝ is then given by ĝ

(x) = r(x)σ θ (x) + µ θ (x) ∼ N (µ g (x), σ g (x)) ∼ N (µ r (x)σ θ (x) + µ θ (x), σ r (x)σ θ (x))

Using this predictive distribution, we can select the hyperparameter configuration maximizing the Expected Improvement (EI) (Mockus et al., 1978) of g(x). The EI can then be defined in closed form as

EI(x) = E[max(0, g(x min ) -ĝ(x))] = σ 2 (x)(v(x)Φ(v(x)) + φ(v(x))), where v(x) := µ g (x) -g(x min ) σ 2 g (x)

, with Φ and φ being the CDF and PDF of the standard normal, respectively. When no observations are available, the empirical CDF F is not defined. Therefore, we warm-start the optimization on the new task by sampling a set of N 0 = 5 hyperparameter configurations via Thompson sampling, as described above. Pseudo-code is given in Algorithm 1.

Algorithm 1 Gaussian Copula process (CGP)

Learn the parameters θ of µ θ (x) and σ θ (x) on hold-out evaluations D M by minimizing equation 1.

Sample an initial set of evaluations D = {(x i , f (x i ))} N0 i=1 via Thompson sampling 2. while Has budget do Fit the GP surrogate r to the observations {(x, ψ(y)-µ θ (x)

σ θ (x) ) | (x, y) ∈ D} Sample N candidate hyperparameters x 1 , . . . , x N from the search space Compute the hyperparameter x i where i = arg max i EI(x i ) Evaluate y i = f (x i ) and update D = D ∪ {(x i , y i ))}. end while

4.3. OPTIMIZING MULTIPLE OBJECTIVES:

In addition to optimizing the accuracy of a black-box function, it is often desirable to optimize its runtime or memory consumption. For instance, given two hyperparameters with the same expected error, the one requiring fewer resources is preferable. For tasks where runtime is available, we use both runtime and error objectives by averaging in the transformed space, e.g., we set z(x) = 1 2 (z error (x) + z time (x)), where z error (x) = ψ(f error (x)) and z time (x) = ψ(f time (x)) denote the transformed error and time observations, respectively. This allows us to seamlessly optimize for time and error when running HPO, so that the cheaper hyperparameter is favored when two hyperparameters lead to a similar expected error. Notice many existing multi-objective methods can potentially be combined with our Copula transformation as an extension, which we believe is an interesting venue for future work. 2012), more details and statistics can be found in the appendix.

We compare against a number of baselines. We consider random search and GP-based BO as two of the most popular HPO methods. As a transfer learning baseline, we consider warm-start GP (Feurer et al., 2015) , using the best-performing evaluations from all the tasks to warm start the GP on the target task (WS GP best). As an extension of WS GP best, we apply standardization on the objectives of the evaluations for every task and then use all of them to warm start the GP on the target task (WS GP all). We also compare against two recently published transfer learning methods for HPO: We assess the transfer learning capabilities of these methods in a leave-one-task-out setting: we sequentially leave out one dataset and then aggregate the results for each algorithm. The performance of each method is first averaged over 30 replicates for one dataset in a task, and the relative improvements over random search are computed on every iteration for that dataset. The relative improvement for an optimizer (opt) is defined by (y random -y opt )/y random , which is upper bounded by 100%. Notice that all the objectives y are in R + . By computing the relative improvements, we can aggregate results across all datasets for each algorithm. Finally, for all copula-based methods, we learn the mapping to copulas via a 3-layer MLP with 50 units per layer, optimized by ADAM with early-stopping.

5.1. ABLATION STUDY:

To give more insight into the components of our method, we perform a detailed ablation study to investigate the choice of the MLP and compare the copula estimation to simple standardization.

Choice of copula estimators For copula-based methods, we use an MLP to estimate the output. We first compare to other possible options, including a linear model and a k-nearest neighbor estimator in a leave-one-out setting: we sequentially take the hyperparameter evaluations of one dataset as test set and use all evaluations from the other datasets as a training set. We report the RMSE in Table 5 of the appendix when predicting the error of the blackbox. From this table, it is clear that MLP tends to be the best performing estimator among the three. In addition, a low RMSE indicates that the task is close to the prior that we learned on all the other tasks, and we should thus expect transfer learning methods to perform well. As shown later by the BO experiments, FCNET has the lowest RMSE among the three algorithms, and all transfer learning methods indeed perform much better than single-task approaches.

Homoskedastic and Heteroskedastic noise:

The proposed Copula estimator (MLP) uses heteroskedastic noise for the prior. We now compare it to a homoskedastic version where we only estimate the mean. The results are summarized in Table 2 where average relative improvements over random search across all the iterations and replicates are shown. It is clear that heteroskedasticity tends to help on most datasets.

Copula transformation and standardization In our method, we map objectives to be normally distributed in two steps: first we apply the probability integral transform, followed by a Copula transform using the inverse CDF of a Gaussian. To demonstrate the usefulness of such transformation, we compare it to a simple standardization of the objectives where mean and std are computed on each datasets separately. Results are reported in Table 2 . It is clear that standardization performs significantly worse than the Copula transformation, indicating that it is not able to address the problem of varying scale and noise levels across tasks. Note that the relative improvement objective is not lower bounded, so that when random search finds very small values the scale of relative improvement can be arbitrary large (such as for the Protein dataset in FCNET). Table 2 : Relative improvements over random search. TS std and GP std respectively using a simple standardization instead of the copula transformation. Ho and He stand for Homoskedastic and Heteroskedastic noise.

5.2. RESULTS:

We now compare the proposed methods to other HPO baselines. The results on using only the error information are shown first followed by the results using both time and error information.

Results using only error information:

We start by studying the setting where only error objectives are used to learn the copula transformation. Within each task, we first aggregate 30 replicates for each method to compute the relative improvement over random search at every iteration, and then average the results across all iterations. The results are reported in Table 3 , showing that CGP is the best method for almost every task except XGBoost. In XGBoost, there are several tasks on which methods without transfer learning perform quite well. This is not surprising as we observe in an ablation study on copula estimators (see Table 5 in the appendix) that some tasks in XGBoost have relatively high test errors, implying that the transferred prior will not help. In those tasks, CGP is usually the runner-up method after standard GP. We also report the results at iteration 10, 50 and 100 in the Tables 7, 8 and 9 in the appendix where we observe CGP and Box RS are the most competitive methods at 10th iteration but at 100 iteration, CGP is clearly the best transfer learning method. This highlights the advantage of being adaptive to the target task of our method while making effective transfer in the beginning. We also show results on two example datasets from each algorithm in Figure 1 , reporting confidence intervals obtained via bootstrap. Note that the performance of the methods in the examples for DeepAR and XGBoost exhibit quite high variations, especially in the beginning of the BO. We conjecture this is due to an insufficient number of evaluations in the lookup tables. Nevertheless, the general trend is that CTS and CGP outperform all baselines, especially in the beginning of the BO. It can also be observed that CGP performs at least on par with the best method at the end of the BO. Box RS is also competitive at the beginning, but as expected fails to keep its advantage toward the end of the BO.

Results using both error and time information:

We then studied the ability of the copula-based approaches to transfer information from multiple objectives. Notice it is possible to combine Copula transformation with other multi-objective BO methods and we will leave this direction as future work. We show two example tasks in DeepAR and FCNET in Figure 2 , where we fix the total number of iterations and plot performance against time with 2 standard error. To obtain distributions over seeds for one method, we only consider the time range where 20 seeds are available ,which explains why methods can start and end at different times. With the ability to leverage training time information, the copula-based approaches have a clear advantage over all baselines, especially at the beginning of the optimization.

We also report aggregate performance over all the tasks in Table 6 in the appendix. Due to the different methods finishing the optimization at different times, we only compare them up to the time taken by the fastest method. For each method we first compute an average over 30 replicates, then compute the relative improvement over random search, and finally average the results across all time points. The copula based methods converge to a good hyperparameter configuration significantly faster than all the considered baselines. Note that we obtain similar results as for Hyperband-style methods (Li et al., 2016) , where the optimization can start much earlier than standard HPO, with the key difference that we only require a single machine.

6. CONCLUSIONS:

We introduced a new class of methods to accelerate hyperparameter optimization by exploiting evaluations from previous tasks. The key idea was to leverage a semi-parametric Gaussian Copula prior, using it to account for the different scale and noise levels across tasks. Experiments showed that we considerably outperform standard approaches to BO, and deal with heterogeneous tasks more robustly compared to a number of transfer learning approaches recently proposed in the literature. Finally, we showed that our approach can seamlessly combine multiple objectives, such as accuracy and runtime, further speeding up the search of good hyperparameter configurations.

A number of directions for future work are open. First, we could combine our Copula-based HPO strategies with Hyperband-style optimizers (Li et al., 2016) . In addition, we could generalize our approach to deal with settings in which related problems are not limited to the same algorithm run over different datasets. This would allow for different hyperparameter dimensions across tasks, or perform transfer learning across different black-boxes. 

:

Note that if z is regressed perfectly, then finding the minimum of f is solved as a parameter x minimizing ψ(f (x)) also minimizes f (x) since ψ is monotonically increasing.

Figures and Tables:

ABLR (Perrone et al., 2018)  and a search space-based transfer learning method(Perrone et al., 2019). ABLR is a transfer learning approach consisting of a shared neural network across tasks on top of which lies a Bayesian linear regression layer per task. Finally, Perrone et al. (2019) transfers information by fitting a bounding box to contain the best hyperparameters from each previous task, and applies random search (Box RS) or GP-based BO (Box GP) in the learned search space.

Comparison on heart for XGBoost.

Figure1: Results using only error information, with the current optimum plotted against the number of used evaluations.

Comparison on dataset m4-Yearly over time.

Comparison on dataset protein over time.

Figure 2: Two example tasks using both error and time information, with the current optimum plotted against time.

time series prediction model proposed inSalinas et al. (2017) (DeepAR). We tuned XGBoost on 9 libsvm datasets(Chang & Lin, 2011) to minimize 1-AUC, and FCNET on 4 datasets fromKlein & Hutter (2019) to minimize the test mean squared error. As for DeepAR, the evaluations were collected on the data provided by GluonTS(Alexandrov et al., 2019), consisting of 6 datasets from the M4-competition(Makridakis et al., 2018) and 5 datasets used inLai et al. (2017), and the goal is to minimize the quantile loss. Additionally, for DeepAR and FCNET the runtime to evaluate each hyperparameter configuration was available, and we ran additional experiments exploiting this objective. More details on the HPO setup are in Table1, and the search spaces of the three problems is in Table4of the appendix. Lookup tables are used as advocated inEggensperger  et al. (

Table content: <table><tr><td>tasks</td><td># datasets</td><td># hyperparameters</td><td># evaluations per dataset</td><td>available objectives</td></tr><tr><td>DeepAR</td><td>11</td><td>6</td><td>∼ 220</td><td>quantile loss, time</td></tr><tr><td>FCNET</td><td>4</td><td>9</td><td>62208</td><td>MSE, time</td></tr><tr><td>XGBoost</td><td>9</td><td>9</td><td>5000</td><td>1-AUC</td></tr><tr><td colspan="5">Table 1: A summary of the three HPO problems we consider.</td></tr><tr><td>5 EXPERIMENTS</td><td/><td/><td/><td/></tr><tr><td colspan="5">We considered the problem of tuning three algorithms on multiple datasets: XGBoost (Chen &amp;</td></tr><tr><td colspan="5">Guestrin, 2016), a 2-layer feed-forward neural network (FCNET) (Klein &amp; Hutter, 2019), and the</td></tr><tr><td>RNN-based</td><td/><td/><td/><td/></tr></table>

Relative improvements over random search averaged over all the iterations. The best methods are highlighted in bold.

Table content: <table><tr><td/><td>task</td><td/><td>dataset</td><td>ABLR</td><td>Box GP</td><td>Box RS</td><td>CGP</td><td>CTS</td><td>GP</td><td>WS GP all</td><td>WS GP best</td></tr><tr><td/><td colspan="2">DeepAR</td><td>electricity</td><td>-2.7</td><td>-2.1</td><td>0.8</td><td>0.8</td><td>0.8</td><td>0.2</td><td>-2.4</td><td>-2.3</td></tr><tr><td/><td/><td/><td>exchange-rate</td><td>0.5</td><td>-0.2</td><td>2.1</td><td>3.3</td><td>2.9</td><td>0.6</td><td>0.6</td><td>0.3</td></tr><tr><td/><td/><td/><td>m4-Daily</td><td>-1.2</td><td>-1.0</td><td>0.1</td><td>1.4</td><td>1.1</td><td>-0.1</td><td>-0.9</td><td>-0.7</td></tr><tr><td/><td/><td/><td>m4-Hourly</td><td>-18.0</td><td>-12.1</td><td>-2.8</td><td>3.0</td><td>-0.8</td><td>-1.3</td><td>-9.8</td><td>-11.7</td></tr><tr><td/><td/><td/><td>m4-Monthly</td><td>-0.7</td><td>0.2</td><td>0.6</td><td>1.0</td><td>0.6</td><td>0.2</td><td>0.2</td><td>0.6</td></tr><tr><td/><td/><td/><td>m4-Quarterly</td><td>-0.4</td><td>-0.1</td><td>0.6</td><td>1.0</td><td>0.8</td><td>0.1</td><td>0.1</td><td>0.0</td></tr><tr><td/><td/><td/><td>m4-Weekly</td><td>-3.8</td><td>-3.2</td><td>0.5</td><td>0.6</td><td>0.4</td><td>0.1</td><td>-2.9</td><td>-3.2</td></tr><tr><td/><td/><td/><td>m4-Yearly</td><td>-0.1</td><td>0.2</td><td>0.3</td><td>1.1</td><td>0.7</td><td>0.5</td><td>-0.2</td><td>0.3</td></tr><tr><td/><td/><td/><td>solar</td><td>-0.3</td><td>0.3</td><td>0.8</td><td>1.1</td><td>0.5</td><td>0.3</td><td>-0.5</td><td>-0.1</td></tr><tr><td/><td/><td/><td>traffic</td><td>-1.3</td><td>-0.7</td><td>0.5</td><td>0.5</td><td>0.0</td><td>0.2</td><td>-0.4</td><td>-0.3</td></tr><tr><td/><td/><td/><td>wiki-rolling</td><td>0.1</td><td>0.1</td><td>0.2</td><td>0.5</td><td>0.3</td><td>0.1</td><td>0.3</td><td>0.0</td></tr><tr><td/><td colspan="2">FCNet</td><td>naval</td><td>57.1</td><td>70.6</td><td>80.6</td><td>82.3</td><td>78.9</td><td>-64.3</td><td>61.2</td><td>65.9</td></tr><tr><td/><td/><td/><td>parkinsons</td><td>14.1</td><td>27.1</td><td>27.3</td><td>38.5</td><td>29.4</td><td>20.2</td><td>19.9</td><td>27.6</td></tr><tr><td/><td/><td/><td>protein</td><td>0.3</td><td>6.8</td><td>5.8</td><td>8.3</td><td>6.6</td><td>3.3</td><td>6.3</td><td>5.9</td></tr><tr><td/><td/><td/><td>slice</td><td>1.8</td><td>37.1</td><td>48.3</td><td>58.7</td><td>53.3</td><td>21.4</td><td>41.9</td><td>35.9</td></tr><tr><td/><td colspan="2">XGBoost</td><td>a6a</td><td>-0.1</td><td>0.1</td><td>0.1</td><td>0.2</td><td>-0.1</td><td>0.2</td><td>-0.1</td><td>-0.1</td></tr><tr><td/><td/><td/><td>australian</td><td>-1.2</td><td>1.4</td><td>3.1</td><td>3.7</td><td>0.4</td><td>1.7</td><td>1.5</td><td>0.8</td></tr><tr><td/><td/><td/><td>german.numer</td><td>-1.3</td><td>0.2</td><td>1.3</td><td>0.7</td><td>0.5</td><td>-0.3</td><td>1.3</td><td>0.2</td></tr><tr><td/><td/><td/><td>heart</td><td>1.5</td><td>1.5</td><td>2.3</td><td>5.1</td><td>0.9</td><td>2.8</td><td>-2.1</td><td>4.4</td></tr><tr><td/><td/><td/><td>ijcnn1</td><td>-8.1</td><td>3.1</td><td>5.0</td><td>5.8</td><td>3.0</td><td>4.1</td><td>3.4</td><td>4.4</td></tr><tr><td/><td/><td/><td>madelon</td><td>2.4</td><td>3.5</td><td>1.7</td><td>4.2</td><td>-0.4</td><td>4.4</td><td>0.3</td><td>1.4</td></tr><tr><td/><td/><td/><td>spambase</td><td>-2.9</td><td>0.9</td><td>0.0</td><td>1.0</td><td>-0.9</td><td>1.8</td><td>-0.3</td><td>-0.5</td></tr><tr><td/><td/><td/><td>svmguide1</td><td>-3.2</td><td>0.8</td><td>1.7</td><td>1.2</td><td>-0.3</td><td>1.1</td><td>0.4</td><td>0.4</td></tr><tr><td/><td/><td/><td>w6a</td><td>0.8</td><td>1.1</td><td>-2.5</td><td>3.4</td><td>0.5</td><td>3.8</td><td>-0.8</td><td>0.9</td></tr><tr><td/><td>1.0</td><td/><td colspan="2">electricity_lookup</td><td>optimizer</td><td/><td>1.2</td><td/><td/><td>m4-Yearly_lookup</td><td>optimizer</td></tr><tr><td>log10_best</td><td>1.6 1.4 1.2 1.8</td><td/><td/><td/><td>RS GP Copula-RS Copula-GP ABLR WS GP best WS GP all Box GP Box RS</td><td>log10_best</td><td>1.6 1.4 1.8</td><td/><td/><td>RS GP Copula-RS Copula-GP ABLR WS GP best WS GP all Box GP Box RS</td></tr><tr><td/><td>2.0</td><td>10 0</td><td colspan="2">10 1 iteration</td><td>10 2</td><td/><td>2.0</td><td>10 0</td><td/><td>10 1 iteration</td><td>10 2</td></tr><tr><td/><td colspan="6">Comparison on electricity for DeepAR.</td><td colspan="4">(b) Comparison on m4-Yearly for DeepAR.</td></tr><tr><td>log10_best</td><td>1.50 1.25 1.00 0.50 0.75</td><td/><td colspan="2">fcnet_parkinsons_lookup</td><td>optimizer RS GP Copula-RS Copula-GP ABLR WS GP best WS GP all</td><td>log10_best</td><td>0.75 1.50 1.25 1.00</td><td/><td colspan="2">fcnet_protein_lookup</td><td>optimizer RS GP Copula-RS Copula-GP ABLR WS GP best WS GP all</td></tr><tr><td/><td>1.75</td><td/><td/><td/><td>Box GP Box RS</td><td/><td>1.75</td><td/><td/><td>Box GP Box RS</td></tr><tr><td/><td>2.00</td><td>10 0</td><td colspan="2">10 1 iteration</td><td>10 2</td><td/><td>2.00</td><td>10 0</td><td/><td>10 1 iteration</td><td>10 2</td></tr><tr><td/><td colspan="6">(c) Comparison on parkinsons for FCNET.</td><td colspan="4">(d) Comparison on protein for FCNET.</td></tr><tr><td/><td/><td/><td colspan="3">xgboost_w6a_max_resource_lookup</td><td/><td/><td/><td/></tr><tr><td/><td>1.5</td><td/><td/><td/><td>optimizer</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td>RS</td><td/><td/><td/><td/></tr><tr><td>log10_best</td><td>1.8 1.7 1.6</td><td/><td/><td/><td>GP Copula-RS Copula-GP ABLR WS GP best WS GP all</td><td/><td/><td/><td/></tr><tr><td/><td>1.9</td><td/><td/><td/><td>Box GP Box RS</td><td/><td/><td/><td/></tr><tr><td/><td>2.0</td><td>10 0</td><td colspan="2">10 1 iteration</td><td>10 2</td><td/><td/><td/><td/></tr><tr><td/><td/><td colspan="4">(e) Comparison on w6a for XGBoost.</td><td/><td/><td/><td/></tr></table>

Relative improvements over random search averaged over time.

Table content: <table><tr><td/><td>task</td><td>dataset</td><td/><td>ABLR</td><td>Box GP</td><td>CGP</td><td>CTS</td><td>GP</td><td colspan="2">WS GP all</td><td>WS GP best</td></tr><tr><td/><td>DeepAR</td><td>electricity</td><td/><td>-19.7</td><td>-17.2</td><td>10.0</td><td>10.3</td><td>-0.3</td><td/><td>-18.5</td><td>-17.0</td></tr><tr><td/><td/><td colspan="2">exchange-rate</td><td>7.4</td><td>6.4</td><td>38.7</td><td>39.1</td><td>0.2</td><td/><td>-0.3</td><td>7.7</td></tr><tr><td/><td/><td>m4-Daily</td><td/><td>-17.5</td><td>-15.7</td><td>20.1</td><td>20.6</td><td>-1.7</td><td/><td>-16.9</td><td>-14.8</td></tr><tr><td/><td/><td>m4-Hourly</td><td/><td>-103.6</td><td>-99.7</td><td>30.9</td><td>29.8</td><td>3.0</td><td/><td>-102.0</td><td>-102.0</td></tr><tr><td/><td/><td colspan="2">m4-Monthly</td><td>3.7</td><td>6.7</td><td>5.0</td><td>4.6</td><td>-0.6</td><td/><td>6.5</td><td>7.6</td></tr><tr><td/><td/><td colspan="2">m4-Quarterly</td><td>-10.3</td><td>-9.6</td><td>0.7</td><td>-0.4</td><td>-0.6</td><td/><td>-9.2</td><td>-9.6</td></tr><tr><td/><td/><td>m4-Weekly</td><td/><td>-26.7</td><td>-24.6</td><td>0.2</td><td>-1.4</td><td>-0.6</td><td/><td>-23.0</td><td>-23.5</td></tr><tr><td/><td/><td>m4-Yearly</td><td/><td>-3.8</td><td>-3.1</td><td>2.2</td><td>2.1</td><td>-0.1</td><td/><td>-3.7</td><td>-2.9</td></tr><tr><td/><td/><td>solar</td><td/><td>1.6</td><td>4.0</td><td>5.6</td><td>5.6</td><td>0.1</td><td/><td>3.4</td><td>3.3</td></tr><tr><td/><td/><td>traffic</td><td/><td>-36.7</td><td>-34.5</td><td>3.6</td><td>3.8</td><td>0.4</td><td/><td>-33.7</td><td>-33.6</td></tr><tr><td/><td/><td>wiki-rolling</td><td/><td>-1.1</td><td>-0.9</td><td>2.3</td><td>2.5</td><td>0.1</td><td/><td>-0.3</td><td>-1.0</td></tr><tr><td/><td>FCNet</td><td>naval</td><td/><td>-80.6</td><td>69.4</td><td>90.1</td><td>89.5</td><td>-150.0</td><td/><td>45.7</td><td>58.8</td></tr><tr><td/><td/><td>parkinsons</td><td/><td>-38.5</td><td>30.3</td><td>53.2</td><td>46.4</td><td>22.3</td><td/><td>27.0</td><td>26.9</td></tr><tr><td/><td/><td>protein</td><td/><td>-6.9</td><td>10.1</td><td>12.7</td><td>10.8</td><td>4.1</td><td/><td>12.6</td><td>7.4</td></tr><tr><td/><td/><td>slice</td><td/><td>-14.5</td><td>6.1</td><td>83.3</td><td>80.9</td><td>22.4</td><td/><td>8.1</td><td>8.4</td></tr><tr><td>task</td><td>dataset</td><td colspan="2">ABLR</td><td>Box GP</td><td>Box RS</td><td colspan="2">Copula-GP</td><td colspan="2">Copula-TS</td><td>GP</td><td>WS GP all</td><td>WS GP best</td></tr><tr><td>DeepAR</td><td colspan="2">electricity</td><td>-4.1</td><td>-1.2</td><td>1.4</td><td/><td>1.0</td><td/><td>0.1</td><td>-0.1</td><td>-1.3</td><td>-1.4</td></tr><tr><td/><td colspan="2">exchange-rate</td><td>-2.0</td><td>4.5</td><td>5.7</td><td/><td>3.6</td><td/><td>5.8</td><td>0.9</td><td>4.8</td><td>3.4</td></tr><tr><td/><td colspan="2">m4-Daily</td><td>-3.4</td><td>-1.3</td><td>1.0</td><td/><td>0.5</td><td/><td>0.6</td><td>0.3</td><td>-1.1</td><td>0.6</td></tr><tr><td/><td colspan="2">m4-Hourly</td><td>-1.9</td><td>2.9</td><td>4.7</td><td/><td>4.7</td><td/><td>5.0</td><td>5.4</td><td>3.1</td><td>3.9</td></tr><tr><td/><td colspan="2">m4-Monthly</td><td>-1.5</td><td>1.0</td><td>1.6</td><td/><td>1.6</td><td/><td>1.5</td><td>0.2</td><td>1.5</td><td>1.7</td></tr><tr><td/><td colspan="2">m4-Quarterly</td><td>-1.3</td><td>0.0</td><td>0.0</td><td/><td>1.0</td><td/><td>0.7</td><td>0.0</td><td>0.7</td><td>-0.1</td></tr><tr><td/><td colspan="2">m4-Weekly</td><td>-2.6</td><td>-0.1</td><td>2.1</td><td/><td>1.1</td><td/><td>1.8</td><td>0.2</td><td>0.8</td><td>1.0</td></tr><tr><td/><td colspan="2">m4-Yearly</td><td>-0.8</td><td>0.5</td><td>0.1</td><td/><td>1.3</td><td/><td>0.7</td><td>0.3</td><td>0.1</td><td>0.3</td></tr><tr><td/><td>solar</td><td/><td>-1.3</td><td>0.5</td><td>1.8</td><td/><td>1.4</td><td/><td>1.4</td><td>0.6</td><td>0.3</td><td>0.0</td></tr><tr><td/><td>traffic</td><td/><td>-3.0</td><td>-1.1</td><td>1.0</td><td/><td>0.2</td><td/><td>-0.1</td><td>0.1</td><td>-0.5</td><td>-0.5</td></tr><tr><td/><td colspan="2">wiki-rolling</td><td>0.2</td><td>0.5</td><td>0.6</td><td/><td>0.8</td><td/><td>0.6</td><td>0.1</td><td>1.0</td><td>0.4</td></tr><tr><td>FCNet</td><td>naval</td><td colspan="2">79.8</td><td>87.9</td><td>95.7</td><td/><td>95.8</td><td/><td>94.9</td><td>-841.0</td><td>63.2</td><td>71.0</td></tr><tr><td/><td colspan="2">parkinsons</td><td>5.9</td><td>32.7</td><td>42.9</td><td/><td>46.9</td><td/><td>45.1</td><td>-3.3</td><td>34.9</td><td>24.6</td></tr><tr><td/><td>protein</td><td/><td>-2.6</td><td>5.9</td><td>8.7</td><td/><td>11.4</td><td/><td>8.3</td><td>0.8</td><td>10.9</td><td>5.7</td></tr><tr><td/><td>slice</td><td colspan="2">-16.7</td><td>59.5</td><td>74.0</td><td/><td>80.0</td><td/><td>78.9</td><td>-42.5</td><td>59.7</td><td>59.7</td></tr><tr><td>XGBoost</td><td>a6a</td><td/><td>-0.5</td><td>-0.1</td><td>0.0</td><td/><td>-0.1</td><td/><td>-0.2</td><td>-0.2</td><td>-0.2</td><td>-0.3</td></tr><tr><td/><td colspan="2">australian</td><td>-7.9</td><td>1.6</td><td>5.3</td><td/><td>3.1</td><td/><td>1.2</td><td>1.2</td><td>-1.5</td><td>0.5</td></tr><tr><td/><td colspan="2">german.numer</td><td>-0.7</td><td>0.6</td><td>1.6</td><td/><td>0.9</td><td/><td>0.6</td><td>-0.2</td><td>0.7</td><td>-0.3</td></tr><tr><td/><td>heart</td><td/><td>-0.8</td><td>2.1</td><td>2.5</td><td/><td>4.2</td><td/><td>2.8</td><td>1.5</td><td>-0.3</td><td>3.8</td></tr><tr><td/><td>ijcnn1</td><td colspan="2">-13.9</td><td>-0.4</td><td>9.1</td><td/><td>8.0</td><td/><td>5.2</td><td>2.9</td><td>5.9</td><td>6.6</td></tr><tr><td/><td>madelon</td><td/><td>-1.4</td><td>2.1</td><td>4.3</td><td/><td>3.0</td><td/><td>-1.2</td><td>1.0</td><td>1.2</td><td>4.3</td></tr><tr><td/><td colspan="2">spambase</td><td>-5.5</td><td>-2.1</td><td>0.2</td><td/><td>-2.7</td><td/><td>-1.6</td><td>-1.9</td><td>-0.6</td><td>-0.5</td></tr><tr><td/><td colspan="2">svmguide1</td><td>-2.9</td><td>0.9</td><td>2.0</td><td/><td>1.2</td><td/><td>0.2</td><td>0.0</td><td>0.1</td><td>1.2</td></tr><tr><td/><td>w6a</td><td/><td>-0.9</td><td>1.6</td><td>-0.7</td><td/><td>2.1</td><td/><td>1.7</td><td>2.2</td><td>0.5</td><td>2.1</td></tr></table>

Relative improvements over random search at iteration 10.

Table content: <table><tr><td>task</td><td>dataset</td><td>ABLR</td><td>Box GP</td><td>Box RS</td><td>Copula-GP</td><td>Copula-TS</td><td>GP</td><td>WS GP all</td><td>WS GP best</td></tr><tr><td>DeepAR</td><td>electricity</td><td>-0.6</td><td>-0.2</td><td>0.3</td><td>0.1</td><td>0.1</td><td>0.1</td><td>-0.6</td><td>-0.4</td></tr><tr><td/><td>exchange-rate</td><td>1.8</td><td>-0.2</td><td>-0.1</td><td>2.1</td><td>1.3</td><td>0.6</td><td>1.2</td><td>1.0</td></tr><tr><td/><td>m4-Daily</td><td>-0.2</td><td>-0.2</td><td>0.2</td><td>0.6</td><td>-0.2</td><td>0.1</td><td>0.0</td><td>-0.2</td></tr><tr><td/><td>m4-Hourly</td><td>-10.3</td><td>-4.2</td><td>-4.7</td><td>1.9</td><td>-3.8</td><td>-4.1</td><td>-2.0</td><td>-3.9</td></tr><tr><td/><td>m4-Monthly</td><td>-0.3</td><td>0.2</td><td>-0.1</td><td>0.8</td><td>0.2</td><td>0.4</td><td>0.1</td><td>0.5</td></tr><tr><td/><td>m4-Quarterly</td><td>-0.5</td><td>-0.1</td><td>0.2</td><td>0.4</td><td>0.1</td><td>0.2</td><td>0.1</td><td>0.0</td></tr><tr><td/><td>m4-Weekly</td><td>-0.1</td><td>0.3</td><td>0.3</td><td>1.0</td><td>0.2</td><td>0.4</td><td>0.5</td><td>0.1</td></tr><tr><td/><td>m4-Yearly</td><td>0.1</td><td>0.2</td><td>-0.1</td><td>0.7</td><td>0.4</td><td>0.5</td><td>-0.3</td><td>0.3</td></tr><tr><td/><td>solar</td><td>0.2</td><td>0.4</td><td>0.4</td><td>1.0</td><td>0.4</td><td>0.5</td><td>-0.7</td><td>-0.2</td></tr><tr><td/><td>traffic</td><td>-0.5</td><td>-0.4</td><td>0.0</td><td>0.3</td><td>-0.2</td><td>0.3</td><td>0.1</td><td>0.2</td></tr><tr><td/><td>wiki-rolling</td><td>-0.2</td><td>-0.1</td><td>-0.2</td><td>0.1</td><td>-0.1</td><td>0.0</td><td>0.0</td><td>-0.5</td></tr><tr><td>FCNet</td><td>naval</td><td>59.6</td><td>72.7</td><td>80.4</td><td>81.5</td><td>77.9</td><td>46.2</td><td>64.4</td><td>70.9</td></tr><tr><td/><td>parkinsons</td><td>16.9</td><td>26.1</td><td>23.1</td><td>33.5</td><td>24.4</td><td>24.9</td><td>19.5</td><td>27.5</td></tr><tr><td/><td>protein</td><td>0.0</td><td>7.0</td><td>4.2</td><td>6.8</td><td>5.1</td><td>3.2</td><td>5.3</td><td>5.8</td></tr><tr><td/><td>slice</td><td>4.5</td><td>39.2</td><td>43.3</td><td>55.3</td><td>48.0</td><td>34.2</td><td>44.9</td><td>38.8</td></tr><tr><td>XGBoost</td><td>a6a</td><td>-0.1</td><td>0.1</td><td>0.1</td><td>0.2</td><td>-0.1</td><td>0.2</td><td>-0.1</td><td>-0.1</td></tr><tr><td/><td>australian</td><td>-0.9</td><td>1.3</td><td>2.0</td><td>4.3</td><td>-0.3</td><td>1.8</td><td>2.1</td><td>0.0</td></tr><tr><td/><td>german.numer</td><td>-1.5</td><td>0.1</td><td>1.0</td><td>0.4</td><td>0.5</td><td>-0.5</td><td>1.5</td><td>0.3</td></tr><tr><td/><td>heart</td><td>2.6</td><td>2.5</td><td>2.3</td><td>5.1</td><td>1.8</td><td>3.7</td><td>-0.8</td><td>6.6</td></tr><tr><td/><td>ijcnn1</td><td>-8.1</td><td>4.2</td><td>4.5</td><td>7.5</td><td>3.2</td><td>5.8</td><td>4.2</td><td>5.6</td></tr><tr><td/><td>madelon</td><td>3.6</td><td>3.5</td><td>0.9</td><td>4.3</td><td>-0.1</td><td>4.9</td><td>0.5</td><td>1.0</td></tr><tr><td/><td>spambase</td><td>-2.8</td><td>1.5</td><td>-0.5</td><td>1.6</td><td>-0.6</td><td>2.2</td><td>-0.1</td><td>-0.4</td></tr><tr><td/><td>svmguide1</td><td>-3.1</td><td>0.9</td><td>1.2</td><td>1.1</td><td>-0.3</td><td>1.3</td><td>0.6</td><td>0.7</td></tr><tr><td/><td>w6a</td><td>1.8</td><td>1.9</td><td>-2.3</td><td>4.3</td><td>1.0</td><td>4.8</td><td>-0.6</td><td>1.0</td></tr></table>

Relative improvements over random search at iteration 50.

Table content: <table><tr><td>task</td><td>dataset</td><td>ABLR</td><td>Box GP</td><td>Box RS</td><td>Copula-GP</td><td>Copula-TS</td><td>GP</td><td>WS GP all</td><td>WS GP best</td></tr><tr><td>DeepAR</td><td>electricity</td><td>-0.5</td><td>-0.2</td><td>0.0</td><td>0.0</td><td>0.0</td><td>-0.1</td><td>-0.2</td><td>-0.2</td></tr><tr><td/><td>exchange-rate</td><td>1.0</td><td>-0.4</td><td>-0.4</td><td>0.8</td><td>0.1</td><td>0.3</td><td>0.4</td><td>0.6</td></tr><tr><td/><td>m4-Daily</td><td>-0.2</td><td>-0.5</td><td>-0.1</td><td>0.1</td><td>-0.1</td><td>-0.3</td><td>-0.2</td><td>-0.3</td></tr><tr><td/><td>m4-Hourly</td><td>-6.2</td><td>-0.6</td><td>-5.6</td><td>4.4</td><td>-1.5</td><td>2.2</td><td>3.8</td><td>-1.5</td></tr><tr><td/><td>m4-Monthly</td><td>-0.1</td><td>0.1</td><td>-0.2</td><td>0.3</td><td>-0.1</td><td>0.2</td><td>0.1</td><td>0.2</td></tr><tr><td/><td>m4-Quarterly</td><td>0.0</td><td>-0.1</td><td>-0.1</td><td>0.1</td><td>-0.1</td><td>0.0</td><td>0.0</td><td>-0.1</td></tr><tr><td/><td>m4-Weekly</td><td>0.2</td><td>0.2</td><td>0.1</td><td>0.4</td><td>0.1</td><td>0.2</td><td>0.2</td><td>0.1</td></tr><tr><td/><td>m4-Yearly</td><td>0.2</td><td>0.1</td><td>-0.1</td><td>0.3</td><td>0.3</td><td>0.3</td><td>-0.1</td><td>0.3</td></tr><tr><td/><td>solar</td><td>0.3</td><td>0.0</td><td>0.2</td><td>0.7</td><td>0.1</td><td>0.3</td><td>-0.4</td><td>-0.3</td></tr><tr><td/><td>traffic</td><td>-0.2</td><td>-0.1</td><td>0.0</td><td>0.2</td><td>-0.1</td><td>0.2</td><td>0.0</td><td>0.1</td></tr><tr><td/><td>wiki-rolling</td><td>0.0</td><td>-0.2</td><td>-0.3</td><td>0.0</td><td>-0.1</td><td>-0.1</td><td>-0.1</td><td>-0.2</td></tr><tr><td>FCNet</td><td>naval</td><td>41.1</td><td>58.2</td><td>63.0</td><td>66.0</td><td>60.8</td><td>26.1</td><td>54.1</td><td>51.5</td></tr><tr><td/><td>parkinsons</td><td>13.7</td><td>24.2</td><td>14.1</td><td>34.2</td><td>18.0</td><td>24.4</td><td>10.4</td><td>27.3</td></tr><tr><td/><td>protein</td><td>2.4</td><td>6.3</td><td>3.4</td><td>6.3</td><td>4.8</td><td>3.5</td><td>4.7</td><td>5.8</td></tr><tr><td/><td>slice</td><td>-1.3</td><td>27.3</td><td>30.9</td><td>42.4</td><td>36.5</td><td>18.2</td><td>31.6</td><td>24.2</td></tr><tr><td>XGBoost</td><td>a6a</td><td>0.1</td><td>0.2</td><td>0.1</td><td>0.3</td><td>0.0</td><td>0.3</td><td>-0.1</td><td>-0.1</td></tr><tr><td/><td>australian</td><td>0.2</td><td>1.8</td><td>1.8</td><td>4.3</td><td>0.0</td><td>3.5</td><td>1.1</td><td>1.2</td></tr><tr><td/><td>german.numer</td><td>-1.2</td><td>0.3</td><td>1.3</td><td>0.8</td><td>0.3</td><td>-0.2</td><td>1.2</td><td>0.6</td></tr><tr><td/><td>heart</td><td>1.5</td><td>0.9</td><td>-0.3</td><td>3.3</td><td>-1.3</td><td>2.4</td><td>-3.5</td><td>3.6</td></tr><tr><td/><td>ijcnn1</td><td>-2.7</td><td>3.1</td><td>2.4</td><td>4.8</td><td>1.6</td><td>4.0</td><td>1.7</td><td>2.8</td></tr><tr><td/><td>madelon</td><td>3.3</td><td>4.0</td><td>-0.3</td><td>4.0</td><td>-0.1</td><td>4.8</td><td>0.3</td><td>1.0</td></tr><tr><td/><td>spambase</td><td>-1.6</td><td>1.4</td><td>-0.7</td><td>2.2</td><td>-1.4</td><td>3.0</td><td>-0.7</td><td>-0.9</td></tr><tr><td/><td>svmguide1</td><td>-2.7</td><td>1.2</td><td>1.1</td><td>0.8</td><td>-0.3</td><td>1.7</td><td>0.8</td><td>0.2</td></tr><tr><td/><td>w6a</td><td>1.5</td><td>1.2</td><td>-3.3</td><td>4.6</td><td>0.1</td><td>4.6</td><td>-0.6</td><td>2.0</td></tr></table>

Relative improvements over random search at iteration 100.

Table content: <table/>

Peer review (iclr2020):
This paper tackles the problem of black-box hyperparameter optimization when multiple related optimization tasks are available simultaneously, performing transfer learning between tasks. Different tasks correspond to different datasets and/or metrics. Gaussian copulas are used to synchronize the different scales of the tasks. I have several reservations with this paper. First and foremost, it seems to be lacking a fair and trivial baseline (I will describe it below) that justifies the apparently unnecessary complicated path followed in this paper. Second, there are a few small incorrect or improperly justified technical details throughout the paper. 1) Mistaken/unjustified technical details: - In equation 1, the last term seems to be constant. For each task, the function psi is not parametric, so its gradient is also not parametric and the input is the inverse of z, i.e., y, which is also fixed. So why is it included in the cost function? This sort of probabilistic renormalization is important in e.g. warped GPs because the transformation is parametric. In this case, I don*t see the point. It can be treated as a normalization of the input data, prior to its probabilistic modeling. - Before equation 1, the text says *by minimizing the Gaussian negative log-likelihood on the available evaluations (x, z)* But then, equation 1 is not the NLL on z but on y. - In section 4.2 the authors model the residuals of the previous model using a powerful Matern-5/2 GP. Why modeling the residuals this way and not the observations themselves? The split of modeling between a parametric and non-parametric part is not justified. - One of the main points of the variable changes is to normalize the scales of the different tasks. However, equations 1 adds together the samples of the different tasks (which, as pointed out by the authors might have different sizes). Even if the scales of the outputs are uniform, the different dataset sizes will bias the solutions towards larger datasets. Why would that be a good thing? This is not mentioned and doesn*t seem correct: there should not be a connection between a dataset size and the prior influence of the corresponding task. In fact, this will have the same effect as if the cost had different scales for different tasks, which is precisely the problem that the authors are trying to avoid. 2) Trivial baseline Given that the authors are trying to aggregate information about the optimal hyperparameters from several tasks, they should not compare with single-task approaches, but with the simplest way to combine all the tasks. For instance: a) Normalize the outputs of every task. This can be accomplished in the usual way by dividing by the standard deviation, or even better, by computing the fixed transform z = psi(y), separately for each task. b) Collect the z of all tasks and feed them into an existing GP black-box Bayesian optimizer. This is a very simple way to get *transfer learning* and it*s unclear that the extra complexities of this paper (copulas, changes of variable with proper renormalization when the transformation is parameter free, etc) are buying much else. Minor improvements: - Page 2: *is the output of a multi-layer perceptron (MLP) with d hidden nodes* Is d really the number of hidden nodes of the MLP? Or the number of outputs? Given that d is also the size of w, it seems it*s actually the latter. - Explain why the EI approach is used for the second model (with the GP), but not for the first model. Edit after rebuttal: “The term is not constant over z” -> Sure, it’s not constant over z. But z is constant. So the term is constant. “The NLL is minimized in z and there is indeed no y in equation 1.” -> Sure, there’s no y in the equation, that’s correct. But it is still the NLL of y, and not the NLL of z. About the new baseline: Instead of simply renormalizing using mean and standard deviation, I suggested above using the same z=psi(y) that is used in the paper for the normalization. Is that where the advantage of the proposed method is coming from? *Note that this is orthogonal to the scale issues we focus on: larger tasks will have larger gradient contributions but the scaling we propose still allows us to learn tied parameters across tasks as their scales are made similar. * Both issues affect the scaling of the task, so I don*t see how they can be orthogonal. Their scales are not made similar precisely because of the different sample sizes.