Deep Ranking Ensembles for Hyperparameter Optimization
Authors: Abdus Salam Khazi
Sebastian Pineda Arango
Josif Grabocka
Venue: ICLR 2023 poster (2023)
Keywords: Hyperparameter Optimization
Meta-learning
Deep Ensembles
Ranking Losses

Abstract: Automatically optimizing the hyperparameters of Machine Learning algorithms is one of the primary open questions in AI. Existing work in Hyperparameter Optimization (HPO) trains surrogate models for approximating the response surface of hyperparameters as a regression task. In contrast, we hypothesize that the optimal strategy for training surrogates is to preserve the ranks of the performances of hyperparameter configurations as a Learning to Rank problem. As a result, we present a novel method that meta-learns neural network surrogates optimized for ranking the configurations' performances while modeling their uncertainty via ensembling. In a large-scale experimental protocol comprising 12 baselines, 16 HPO search spaces and 86 datasets/tasks, we demonstrate that our method achieves new state-of-the-art results in HPO.

Published as a conference paper at ICLR 2023
DEEP RANKING ENSEMBLES FOR HYPERPARAMETER OPTIMIZATION
Abdus Salam Khazi', Sebastian Pineda Arango,, University of Freiburg Correspondence to Sebastian Pineda Arango: pineda@es. ceai-freibuggec.m
Baango",Josa Josif Grabocka
ABSTRACT
Automatically optimizing the hyperparameters of Machine Learning algorithms is one of the primary open questions in AL Existing work in Hyperparameter Op- timization (HPO) trains surrogate models for approximating the response surface regression task. In contrast, we hypothesize that the op- timal strategy for training surrogates is to preserve the ranks of the performances of hyperparameter configurations as a Learning Rank problem. As result, we present a novel method that meta-learns neural network surrogates optimized for ranking the configurations' performances while modeling their uncertainty via en- sembling. In large-scale experimental protocol comprising 12 baselines, 16 HPO search spaces and 86 datasets/tasks, we demonstrate that our method achieves new state-of- oote-of-the--oe the-art results in HPO.
of hyperparameters
INTRODUCTION
Hyperparameter Optimization (HPO) is a crucial ingredient in training state-of-the-art Machine Learning (ML) algorithms. The three popular families of HPO techniques Bayesian Optimiza- tion (Hutter et al. )019), Evolutionary Algorithms (Awad et al. 2021a), and Reinforcement Learn-
ing (Wu & Frazier 2019 Jomaa al. 2019). Among these paradigms, Bayesian Optimization (BO) stands out as the most popular approach to guide the HPO search. At its core, BO fits a paramet- ric function (called surrogate) estimate the evaluated performances (e.g. validation error rates) of a set of hyperparameter configurations. The task of fitting the surrogate to the observed data points is treated as probabilistic regression, where the common choice for the surrogate is Gaus- sian Processes (GP) (Snoek et al. 2012). Consequently, BO uses the probabilistic predictions of the configurations' performances for exploring the search space of hyperparameters. For an introduction to BO, we refer the interested reader to Hutter et al. (2019).
In this paper, we highlight that the current BO approach of training surrogates through a regression task is sub-optimal. We furthermore hypothesize that fitting surrogate to evaluated configurations instead a learning-to-rank (L2R) problem (Burges et al. 2005). The evaluation criterion for HPO is the performance of the top-ranked configuration. In contrast, the regression loss measures the surrogate's 's ability to estimate all observed performances and does not pay any special consideration to the top-perfore perforrmn configuration(s). We propose that BO surrogates must be learned estimate the ranks of the configurations with a special emphasis on correctly predicting the ranks of the top-performing configurations.
Unfortunately, the current BO machinery cannot be naively extended for L,R,, because Gaussian Processes (GP) are not directly applicable ranking. In this paper, we propose novel paradigm to train probabilistic surrogates for learning to rank in HPO with neural network ensembles] Our networks are learned minimize L2R listwise losses (Cao et al. 2007). and the ensemble's uncer- tainty estimation is modeled by training diverse networks via the Deep Ensemble paradigm (Laksh- minarayanan et al. 2017). While there have been a few HPO-related works using flavors basic
ranking losses (Bardenet et al. 2013 Wistuba & Pedapati 2020 Öztürk et al. 2022), ours is the first
* "quual contribution
"Our code is available in the following repository:
httpss:/giithbb.com/reeeee/ frei burg/
Deepkank inghnnemblee

---

Published as a conference paper at ICLR 2023
systematic treatment of HPO through a wttpooooooogcallllpppiiiipe L2R formulation. To achieve state-of-the-art HPO results, we follow the established practice of transfer-leaming the ranking sur- rogates from evaluations previous datasets (Wistuba & Grabocka 202T). Furthermore, we boost the transfer quality by using dataset meta-features as an extra source of information (Jomaa et al. 2021a).
We conducted large-scale experiments using HPO-B (Pineda Arango et al.) 2021). the largest pub- lic HPO benchmark and compared them against 12 state-of-the-art HPO baselines. We ultimately demonstrate that our method Deep Ranking Ensembles (DRE) sets the new state-of-the-art in HPO by a statistically-sinifiiee margin. This paper introduces three main technical contributions:
We introduce a novel neural network BO surrogate (named Deep Ranking Ensembles) optimized with Learning-to-Re (L2R) losses;
We propose new technique for meta-learning our ensemble surrogate from large-scale public meta-datasets;
Deep Ranking Ensembles achieve the new state-of-the-art in HPO, demonstrated through a very large-scale experimental protocol.
RELATED WORK
Hyperparameter Optimization (HPO) is problem that has been well elaborated on during the last decade. The mainstream HPO strategies are Reinforcement Learning (RL) (Wu & Frazier 2019).
evolutionary search (Awad et al. 2021b), and Bayesian optimization (BO) (Hutter et al. 2019)) The latter comprises two main components: surrogate function that approximates the response function given some observations, and an acquisition function that leverages the probabilistic output of the surrogate to explore the search space, ultimately deciding which point observe next. Previous work covers various choices for the surrogate model family, including Gaussian Processes (Snoek ctt al., 2012). and Bayesian Neural Networks (Springenberg et al.j (006aa). Other authors report the advantages of using ensembles as a surrogate, such as Random Forests Hutter et al. (2011), or ensembles of neural networks White et al. (202T). In contrast, we train BO surrogates using
Saaming-to-rank problem definition (Cao et al. 2007).
Transfer HPO refers the problem definition of speeding up HPO by transferring knowledge
from evaluations of hyperparameter configurations on other auxiliary datasets (Wistuba & Grabocka]
2021 Feurer et al.) 2015 2018). For example, the
of a Gaussian Process can be meta-learned on previous datasets and then transferred to new tasks (Wang et al. (002T). Similarly,
deep GP's kernel parameters can also be meta-learned across auxiliary tasks (Wistuba & Grabocka 2021). Another method trains ensembles of GPs weighted proportionally to the similarity between the new task and the auxiliary ones [Wistuba et al. 2016). When performing transfer HPO, is use- ful to embed additional information about the dataset. Some approaches use dataset meta-features to warm-initialize the HPO (Feurer et al. 2015 Wistuba et al. 2015), or to condition the surrogate during pre-training (Bardenet et al. 2013). Recent works propose attention mechanism train dataset-aware surrogates (Wei et al. 2019), or utilize deep sets to extract meta meta-fcatures (Jomaa et al., 2021b). In complement to the prior work, we meta-learn ranking surrogates with meta-features.
Learning to Rank (L2R) is a problem definition that demands estimating the rank (a.k.a. relevance, or importance) of an instance in
(Burges al.) 2005). The primary application domain for L2R is information retrieval (ranking websites in search engine) (Ai et al., 2018). or e-commerce systems (ranking recommended products or advertisements) (Tang & Wang 2018 Wu al.) 2018). However, 1.2R is applicable in diverse applications, from learning distance functions among images in computer vision (Cakir et al., 2019) up to ranking financial events (Feng et al., 2021). this paper, we emphasize the link between HPO and L2R and train neural surrogates for BO with L2R.
Learning to Rank for HPO is a strategy for conducting HPO with an L2R optimization approach. There exist some literature on transfer-learning HPO methods that employ ranking objective within their transfer mechanisms. SCoT uses a surrogate- paarod ranking mechanism for transferring hyper- parameter configurations across datasets (Bardenet et al.) 2013). On the other hand, Feurer et al. (2018) use a weighted ensemble of Gaussian Processes with one GP per auxiliary dataset, while the ensemble weights are learned with pairwise ranking- loss. Modeling the ranks of the learning
2

---

Published as a conference paper at ICLR 2023
Concatenator
z
81
Aggregator
MLP Scorer
Ranker
Variable
MM
SM
fM
Figure The neural architecture of our Deep Ranking Ensembles (DRE) with imputs æ (query points) and (meta-features).
curves also helps estimate the performance of configurations in multi-fidelity transfer setup Wiis- tuba & Pedapati 2020). Recent work has demonstrated that pair-wise ranking losses can be used for transter-learnin
surrogates in a zero-shot HPO protocol (Öztürk et al. (022). However, none of these approaches extensively study the core HPO problem with L2R, nor do they analyze which ranking loss types enable us to learn accurate BO surrogates.
3 DEEP RANKING ENSEMBLES (DRE)
3.1 PRELIMINARIES
Hyperparameter Optimization is defined as the problem of tuning the hyperparameters E X of a ML algorithm to minimize the validation error achieved on a dataset D as arg min Val (x. D). The mainstream approach for tuning hyperparameters is Bayesian Optimization (BO), an introduc- tion of which is offered by Hutter et al. 2019). BO relies on fitting a surrogate function for approx- imating the validation error on evaluated hyperparameter configurations. Consider having evaluated N configurations on a dataset and their respective validation errors as H = {( y:)
Jim where y: = va C (xx, D). We train a surrogate function ğ (ax)) = [(x;; 0), typically a Gaussian Process, to estimate the observed y as arg maxg E ((s,)))>p log (y, [2,, H/ //[[zz :).
Learning to Rank (L2R) differs from a standard supervised regression because instead of directly estimating the target variable it learns to estimate the rank of the target values. In the context of HPO, define the rank of configuration r(vi, (y1,,..,,'')
Σø= 1y:su The core of a typical L2R method (Burges al. 2005) includes training a parametric ranker f (x;) == (a;; 0) that correctly estimates the ranks of observed configurations' validation errors. Instead of naively estimating the ranks as direct regression task, i.e. arg max, ((g,rr)))p log p (ra [z;; 0), L2R techniques prioritize estimating the ranks of top-performing configurations more than bottom- performing ones (Cao et al., 2007). In general ranking losses can be defined on the basis of sin- gle objects (point-wise approach), pairs of objects (pair-wise approach) the whole list of objects (list-wise approach) (Chen et al. 2009).
3.2
DEEP RANKING ENSEMBLE (DRE) SURROGATE
In this paper, introduce novel ranking model based ensemble of diverse neural net- works optimized for L2R. We aim to learn neural networks that output the ranking score of a hyperparameter configuration 8 X R. The ranks of the estimated scores should match the true ranks Σ Lw Σÿ= Le(z, however, with higher priority in approximat- ing the ranks of the perforrnnnn configurations using weighted list-wise L2R loss (Cao al. 2007]. First of all, we define the indices of the ranked/ordered configurations as (1....,N] N [1....,N] N... Concretely, the l-th observed configuration the k-th ranked configuration π (f)
k Σ)= Sme Ultimately, we train the scoring network using the following loss:
N arg min
(Ii, Ve. y, 0), where L (1,, U6. v. 0) w (TT(i) exp= (x=(0)1)))))
(I)
Σne exp" (==())10)
3

---

Published as a conference paper at ICLR 2023
Algorithm 1: Meta-learning the Deep Ranking Ensembles
Input : Set of datasets D, Number of iterations J, Number of ensemble scorers M
,.....,..... Output: DRE parameters ..,.... OM, Meta-feature network parameters +
Initialize scorer networks with parameters ......, ,,.
Initialize the parameters & of the meta-feature network from Jomaa et al. (2021a)
for j = to J do
Sample dataset index i E {I,..., D}, sample scorer network index m (1,..., M)
Sample query set ((x) : {(2(()) (1) (a) (d)
(c) (() from D;
M/
Sample support set (xx) from Dj 1765;;
Compute meta reta-feetures z 2(H(i);
Compute rank scores for the query set s; = 8 (a/5), () ; =...... NO0)
Compute true ranks (1),..., π (N()));
10
Compute loss L (F, s; mm 3) using Equation
Update the meta-feature network x- 170
(L(m, 0m
Bb
12
Update the ranker network On < Om 1/0,..
aC(m0 0m
08
13
end
return 0I....,0xx 00. Ø
The weighting functions w
+ R., defined as w (I(1)) +11( and is used to assign a higher penalty to the top-performing prpee-parameter configurations, whose correct rank is more important in HPO (Chen al.) (017). After having trained the scoring model of Equation we estimate the rank of an unobserved configuration as ê (x; f) Σ) (=(m):0))<(xx00) Further- more, Bayesian Optimization (BO) needs uncertainty estimates to be able explore the search space (Hutter et al. 2019). As result, we model uncertainty by training M diverse neural scorers $1 (1, 61)...., As (z. 6M) with stochastic gradient descent. The diversity of the ensemble scorers is ensured through the established mechanism of applying different per-scorer seeds for sampling mini-batches of hyperparameter configurations (Lakshminarayaaan et al. 2017). Finally, the pos- terior mean and variance of the estimated ranks is computed trivially as (ux) = N E f(z; 0s) and a²
(x)) (a:) N (f(a;; (1;;;;;) 0t) /²(:))² The BO pseudo-code and the details for using Deep
Rankers in HPO are explained in Appendix A
3.3 META-LEARNING THE DEEP RANKING ENSEMBLES
HPO is a very challenging problem due to the limited number of evaluated hyperparameter con- figurations. As result, the current best practice in HPO relies transfer-learning the knowledge of hyperparameterg" from evaluations on previous datasets (Wistuba Grabocka 2021 Wistuba ct al. 2016 Salinas et al. 2020). In this paper, we meta-learn ranker from K datasets assuming (k)
we have a set of observations H := {(21 31 ...... (I N } k = 1,..,, K with Na evaluated hyperparameter configurations on the k-th dataset. We meta-learn our ensemble M Deep Rankers with the meta-learning objective in Equation where we learn to estimate the ranks of all observations on all evaluations for all the previous datasets using the loss of Equation]
K N M ΣΣ
arg min Gs.....or
yn
(2) 0m
"Even in manually-designel MI. systems, experts start their initial guess about hyper-parameters by transfer-leaming the configurations that worked well past projects (a.k.a. datasets).
4

---

Published as a conference paper at ICLR 2023
BO Step 1
BO Step
BO Step 3
BO Step 4
BO Step
0
10
T0
-10
True Function
Expected Improvement
Observed Samples
Next Sample
Figure BO Steps Example with Random Initialized DRE. El scaled and shifted for clarity.
Scorers' Output
Rarkers
Distributions Storees' Output
Distributious of Rankers' Output
Unoertainty Estimatioos
Oulput
200
200
200
150
150
150
100
100
10
2
Figure 3: Understanding the outputs of DRE's modules.
Transfer-learning for HPO suffers from the regative-transfer phenomenon, where the distribution of the validation errors given hyperparameters changes across datasets. In such cases, using dataset meta-features helps condition the transfer only from evaluations on similar datasets (Rakotoarison et al.] 2022, Jomaa et al. 2021a). We use the meta-features of Jomaa et al. (2021a) which are based on a deep set formulation (Zaheer al. 2017) the pairwise interactions between hyperparameters and their validation emrors. The meta-fcature network with parameter & takes history of evaluations
H (xi,, ):) its input and outputs L-dimensional representation of the history as =(H, Φ) (X' x R) - RL_. Afterward, the scorer function becomes 8 (x, 2(H; o); 0) x TRL + TR. In other words, the dataset meta-features additional features to the scorers. graphical depiction of our architecture is shown in Figure [
We update all the scorer networks of the ensemble independently using the loss of Equation [ The pseudo-code of Algorithm|] draws an evaluation set (called query set) which is used as the training batch for updating the parameters of the sampled scorer network. We also meta-learn the meta- feature network (Jomaa et al.) 2021a), however, by using different batch of evaluations (called support set). We do not meta-learn both the scorer and the meta-feature networks using the same batch of evaluations in order to promote generalization.
EXPERIMENTS AND RESULTS
4.1 MOTIVATING EXAMPLE
We demonstrate our DRE with 10 base models a simple sinusoid function y sin( (2)zz for E [-10, 10] sampled with a step size of 0.1 in equally spaced intervals. Further details on the architecture are explained in Section B In Figure 2 we conduct BO with variant of DRE without meta-lcarning, and we start the HPO with 3 initial random observations. We observe that a BO procedure with the Expected Improvement acquisition reaches optimum after observations.
Furthermore, we plot the scorers' and rankers' outputs of the second BO step in Figure The analysis illustrates that the distributions of the scorers' outputs have different ranges because the loss function Equation] I models only the target rank, but not the scale of the target. However, the outputs of the rankers display similar distributions in the rank space, which is more adequate for

---

Published as a conference paper at ICLR 2023
1.0
MetinTToin
Meta Thin Tack
0.5
Meta. Trun Tusk 3
0.0
Meto: Tran Tisk
0.5
Meta: Test Task
0.55
Expocted Imprexement
1.0
Observod Sumplo
Next Semple
-1.0
1.55
10
10
10
10
Figure 4: Meta-train and Meta-test Tasks (left) for optimizing the function. The meta- DRE finds the optimum in one step (right).
computing the ranks' uncertainties. Moreover, the rank distributions differ in certain regions of the search space, enabling BO conduct exploration.
To showcase the power of transfer-leaminng we meta-learn DRE on 5 auxiliary tasks, corresponding
different sinusoidal functions y = sin( - + 3) with varying 3 E {11, 15), as illustrated in (lefr) Subsequently, we deploy meta-learned DRE surrogate on a test task (blue line with 8) which was not part of the meta-training set. Figure reveals that DRE directly discovers global optimum within one BO step (4 total observations). The success is attributed to the fact that the surrogate has been meta-learned to recognize sinusoidal shapes given the 3 initial observations in green, is clearly shown by the acquisition in Figure 4 (right).
4.2 DATASETS AND BASELINES
We base our experiments HPO-B (Pineda Arango et al. 202T), the largest public benchmark for HPO. contains 16 search spaces, each which comprises a meta-train, meta-test, and meta- validation split. Every split is set of datasets, and for every dataset, the benchmark contains the validation errors of evaluated hyperparameter configurations. The benchmark also includes the re- sults of several HPO methods run in those datasets, including transfer and non-transfer algorithms Moreover, we generated new results for three additional state-of-the-ee baselines (GCP, HEBO, and DKLM) that are not released by HPO-B. The benchmark provides sets initial random seeds for every task in the meta-test split (86 in total). We use the meta-test datasets to compare the per- formance of the Deep Ranker Ensembles against the baselines. Specifically, our non-transfer HPO baselines are listed below:
Random Search (RS) (Bergstra & Bengio 2012) is simple yet strong baseline that
selects a random configuration at every step.
Gaussian Processes (GP) (Sooek et al. 2012) model the response function by computing the posterior distribution of functions induced by the observed data.
DNGO (Snoek al. 2015) uses neural network that models the uncertainty with Bayesian linear regression on the last network layer.
BOHAMIANN (Springenberg et al. 201 is also a Bayesian neural network that per- forms Bayesian inference via Hamiltonian Monte Carlo.
Deep-Kernel Gaussian Processes (DKGP) (Wilson et al. 2016) learn latent representa- tion of the features that are fed to GP kernel function.
НЕВО (Cowen-Rivee Rivers et al. 2020) is a state-of-the-art Bayesian optimization method. It combines input and output transformations and multi-objective acquisition function. We use the implementation contained in the original repository
Transfer HPO methods use the evaluations of the tasks included in the meta- meta-train split to leta-learn surrogates, that are subsequently applied for HPO on the meta-test tasks within the same search space. We consider the following baselines:
"Available in htntt::s//gitttuuccoooo/ee
Available in http:://ggttto..coo//aaaaaa frriiiirbbbbii freibura/HoOQ

---

Published as a conference paper at ICLR 2023
TST (Wistuba al. 2016) constructs an ensemble of Gaussian Processes aggregated with a kerghl- average. Alternativlyy, TAF builds an ensemble of acquisition functions.
RGPE (Feurer et 2018) trains Gaussian Process per each meta-train task and then combines for new task through a weighting scheme, which accounts for the ranking per- formance of every base GP model.
FSBO (Wistuba & Grabocka 2021) pre-trains Deep Kernel Gaussian Process using meta- train tasks and then fine-tures tie-ttnns the parameters when observations for new tasks are available.
2020) pre-trains neural network predict the residual performance on the auxiliary tasks and applies Gaussian Copulas to combine results for new task.
GCP (Salinas et al.
DKLM (Jomaa et al. 2021b) adds a Deep Set as task contextualization on top of FSBO. We use the same hyperparameters suggested in the original paper.
4.3 TRE-EXPERIMENTAL SETUP
The meta-fcature extractor z is based on the Deep Set architecture proposed by Jomaa et al. (202 lab with five hidden layers and 32 neurons per layer. The ensemble of scorers is composed of 10 MLPs with identical architectures: four layers and 32 neurons that we selected using the meta-valinn veta-validation split from HPO-B. We meta-learn DRE for 5000 epochs with Adam optimizer, learning rate 0.001 and batch size 100. Every element of the batch is a list of 100 elements. We select 20% of the samples in each list as input to the meta-feature extractor. During meta-test in every BO iteration, update the pre-trained weights for 1000 epochs. For DRE-RI, initialize randomly the scorers and train them for 1000 epochs using Adam Optimizer with learning rate of 0.02. Every epoch, use 20% of the observations to feed the meta-feature extractor.
4.4 RESEARCH HYPOTHESIS AND EXPERIMENTAL RESULTS
(a) Transfer Methods
(b) Non- Kon-Transfer Methods
5.5
Average Rank
Average Rank
5.0
4.5
6
4.0
4-
3.5-
20
20
35
30
65
80
95
Number of Evaluations
Number Evaluations
DRE
TST TAP
HEBO -CP
DRE-RI DKGP -DNGO HEBO -BOHA RRadom GP
FSBO
DKLM
Random
RGPE
Figure 5: Results for Transfer and Non-transfer methods.
Hypothesis 1. Deep Ranking Ensembles (DRE) achieve state-of-the-ar results in transfer IPPO.
We compare against the transfer HPO baselines listed in Section 4.2 and report the average ranks across all the tasks in the meta-test split of all the HPO-B search spaces. Our protocol uses 5 initial configurations plus 100 BO iterations across search spaces (the default HPO-B protocol). Our method uses meta-features (Jomaa et al. 2021a and the scorer parameters are fine-tuned after each BO observation.
Figure 5 (left) shows that DRE clearly outperforms all baselines 100 BO iterations based on the rank among the HPO methods averaged among 86 datasets and runs. We compute the critical dif- ference diagram (Demšar 2006) for 25, 50, and 100 iterations, and show the statistical significance

---

Published as a conference paper at ICLR 2023
(a) Ranking Loss
(b) Meta-featuree
(c) Acquisition Function
4.0
3.50
3.25
Avenae Rank
3.3-
Average Rank
Awerage Rank
3.00
2.75
33:00
3.0
2.50
2.25
2.50
2.5
35
65
96
20
50
80
35
65
95
Number of Evaluations
Number of Evaluations
Number of Evaluations
List
Pair Point
-og
F.M.T FM
FTT
Avg UCH Rondom
W-List
M.T
E1
Figure 6: Results after testing hypothesis 3-5.
of the results Figure 9a
(Appendix Eb. HEBO is not transfer HPO method but is presented as a reference. These results demonstrate the advantage of training neural ensembles with 1,2R since our method outperforms other rivals which also meta-train neural networks (FSBO, DKLM), or en- sembles of neural networks (TST, TAF, RGPE). DRE also attains competitive results in individual search space, as shown in Figure L3 at Appendix
Hypothesis 2. The randomly-initializz DRE performs competitivell in non-transfer HPO.
We test the hypothesis by comparing the performance DRE against the non non-transfed basclines compute the average rank over 100 BO iterations, aggregating across all the meta-test tasks of all the search spaces in HPO-B.
mentioned in Section 4.2 Similar Experiment
The results of Figure S (right) show that random initialized DRE (i.e. non meta-learned) is still a competitive surrogate for HPO. exhibits good performance for up to 30 iterations compared to the other baselines and is second only HEBO (notice our meta-learned DRE actually outperforms HEBO, Figure ((guree](llft))) (left)). This demonstrates the usefulness of deep ensembles with L2R as general- purpose HPO surrogates. Interestingly, DRE outperforms other surrogates using neural networks, such as BOHAMIANN, DNGO, and DKGP. We present the statistical significance of the results after 25, 50, and 100 BO iterations in Figure 96
Hypothesis 3. A weighted list-wise ranking loss is the best L2R strategy for DRE.
We test DRE (meta-learned) with three different L2R losses: point-wise, pair-wise, and list-wise (weighted and non-weige son-weighted) ranking losses. Additionally, we compare to surrogate predicting the performance in the original scale using Mean Squared Error as loss, i.c. regression. Moreover, compare the performance to DRE trained with regression loss. We omit the meta-features from all variants to avoid confounding factors from the analysis and use Expected Improvement as the acquisition function.
The results in Figures 66 and [ Ia (Appendix D show the advantage of the list-wise ranking losses over the other type of ranking losses. Moreover, the results highlight the advantage of list-weighted ranking losses, as it attained the best performance over the average rank among 100 BO iterations. Additionally, we observe that pairwisc-loss losssss also give boost in performance compared point- wise estimations. The message is: "Any L2R loss is better than the regression one".
Hypothesis 4. Meta-feature help the transfer HPO performance of DRE.
We evaluate DRE with and without the meta-features extracted by the DeepSet module (Jomaa et al.) 202 ablating the scenarios with and without meta-learning. Again all 16 search spaces from HPO-B for 100 BO iterations, starting with random initial configurations. DRE uses the weighted list-wise loss, and Expected Improvement as the acquisition.
8

---

Published as a conference paper at ICLR 2023
Figure 6b shows the performance obtained with meta-features (F) considering meta-learning (M) and fine-tuning (T). A missing capital letter in the label stands for an experiment without that aspect (e.g. no M means meta-learning, etc). The results indicate that the meta-features help DRE achieve better performance, both with and without meta-learning. The results also highlight that fine-tuning (i.e. updating the scorer network's parameters the target tasks after each BO step) the meta-Icarned surrogate is important for achieving the best HPO performance. Further evidence of the significance of these results showcased in Figure [lb (Appendix E
Hypothesis 5. Expected Improvement is the best acquisition frnnutittionfuunctioon... for BO with DRE.
We run experiments to address how DRE performs with different acquisition functions, which use DRE's estimated rank uncertainty to explore the search space with Bayesian Optimization. Con- cretely, we ablate the Upper Confidence Bound (UCB) and Expected Improvement (EI) acquisitions. Additionally, we added Average Rank (Avg) which simply recommends the configuration with the highest estimated average rank, without using the posterior variance of the rank. We also add Ran- dom Search as reference baseline. Further details on how apply acquisitions in the BO loop are discussed in Appendix A losses.
In this experiment, we use meta-features and weighted list-wise ranking
The results in Figures 6c and TOb (Appendix F demonstrate that is the best choice for the ac- quisition function. As UCB and EI attained overall better performances than the simple average rank (no uncertainty), we conclude the uncertainties computed by DRE are effective in exploring the search space.
4.5 DISCUSSION ON DRE HYPERPARAMETERS
Given that DRE achieves state-of-the-ar results across all the 16 search spaces (see Figure 12 in Appendix H of HPO-B by using the same configuration (e.g. number of layers for the scor- ers, number of layers for meta-feature extractor), we assume our settings thyper-hyperparameters are applicable straightfor- wardly to new search spaces. Such a generalization of the hyper- hyperparameters is desirable for any HPO method and liberates practitioners and rescarchers from having to tune DRE hyper- hyperparameters. In Figure 7 we show ablation study compar- ing the performance of DRE for different numbers of layers (2, 3, 4), and different numbers of neurons per layer (16, 32, 64) on all the tasks of the meta-validation split from HPO-B. Given the critical difference diagram Figure 10a we observe the perfor- mance does not change significantly when we vary any of these However, we notice that the depth of the scorer is slightly more important to tune than the number of neu- rons per layer. We also notice that even an expressive ensemble of scorers (32x4) is able to generalize well on the meta-test split, as we have shown in our previous experiments.
Scorer Size
5.5
Average Rank
5.0
1.5-
4.0
3.5
20
50
65
95
Number of Evaluations
16x2
32x3
48x2
Randomn
32x2
64x2
Figure 7: Average Rank on the meta-validation split from HPO-
CONCLUSION
The presented empirical results based on a very large-scale experimental protocol provide strong evidence of the state-of-the-art performance of deep ensembles optimized through learning to rank. We demonstrated that our method outperforms a large number of 11 baselines in both transfer and non-transfer HPO. In addition. validated the design choices of our method through detailed ablations and analyses. Particularly, the results indicate the power of meta-learning surrogates from evaluations other datasets. Overall, we believe that this paper will set new trend in the HPO community for moving away from regression-learne surrogate functions in Bayesian Optimization. Finally, our surrogate DRE opens up an effective way improve the HPO performance different sub- uub-problems, such as multi-fidelin HPO, multi-objective HPO, or neural architecture search.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This research was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foun- dation) under grant number 417962828 and grant INST 39/963-1 FUGG (bwForCluster NEMO). In addition, Josif Grabocka acknowledges the support of the BrainLinks- BroinTools Center of Excel- lence.
REFERENCES
Qingyao Ai, Keping Bi, Jiafeng Guo, and W. Bruce Croft. Learning a deep listwise context model for ranking refinement. The 41st International ACM SIGIR Conference on Research and De- velopment in Information Retrieval, SIGIR '18, pp. 135-144, New York, NY, USA, 2018. Asso- ciation for Computing Machinery. ISBN 9781450356572. doi: 10.1 01144///299788200099999
URL
https://doi org/10 1 145/3209978.3209983 3209985
N. Awad, N. Mallik, and F. Hutter. DEHB: Evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. In Z. Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, ISCAI-21, pp. 2147-2153. ijcai.org, 2021a.
Noor H. Awad, Neeratyoy Mallik, and Frank Hutter. DEHB: evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. CoRR, abs/2105.09821 2021b.
Rémi Bardenet, Mátyás Brendel, Balázs Kégl, and Michèle Sebag. Collaborative hyperparameter tuning. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, pp. 199-207, 2013.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. J. Mach. Learn. Res., 13:281-305, 2012.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hul- lender. Learning to rank using gradient descent. In Proceedings of the 22nd international confer- on Machine learning, pp. 89-96, 2005.
Fatih Cakir, Kun He, Xide Xia, Brian Kulis, and Stan Sclaroff. Deep metric learning to rank. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In Zoubin Ghahramani (ed.), Machine Learning, Proceedings of the Twenty-Fouur Fourth International Conference (ICML 2007), Corvallis, Oregon, USA, June 20- 24, 2007, volume 227 of ACM International Conference Proceeding Series, pp. 129- 136.. ACM, 2007. URL
http://o...g/11.../11/1. wtt..o////...1//23333333366 1273512
Huadong Chen, Shujian Huang, David Chiang, Xinyu Dai, and Jiajun Chen. Top-rank enhanced listwise optimization for statistical machine translation. arXiv preprint arXiv:1707.05433 2017.
Wei Chen, Tie-yan Liu, Yanyan Lan, Zhi-ming Ma, and Hang Li. Ranking measures and loss functions in learning to rank. In Y. Bengio. D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 22. Curran As- sociates, Inc., 2009. URL https::/pproceeeeeeeet www.:eeeeeddii.....e..........s. wttpppppp/p////////....// 2E570744199d2111abbbbe T11111101111191711111111 Papee7766--Ppp..... pdf
Alexander Imani Cowen-Rivers Wenlong Lyu, Zhi Wang, Rasul Tutunov, Jianye Hao, Jun Wang, and Haitham Bou-Ammas HEBO: heteroscedastic evolutionary bayesian optimisation. CoRR, abs/2012.03826, 2020.
Janez Demăar. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res., 7:1-30, dee 2006. ISSN 1532-4435.
Fuli Feng, Moxin Li, Cheng Luo, Ritchie Ng, and Tat-Seng Chua. Hybrid learning to rank for finan- cial event ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, pp. 233-243, New York, NY, USA, 2021. Association for Computing Machinery, ISBN 45038503.... doi: 00.1145/3404835.4462299 http:////b..oggg....223///303330 94699
10

---

Published as a conference paper at ICLR 2023
Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Initializing bayesian hyperparameter optimization via meta-learning. In Blai Bonet and Sven Koenig (eds.), Proceedings of the Twenty- Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pp- 1128-1135. AAAI Press, 2015.
Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for bayesian opti- mization using monking-weighte gaussian process ensembles. In AutoML Workshop at ICML, volume 7, 2018.
Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization 5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers, pp. 507-523, 2011. doi:
11-10779788-3-62225556---
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.). Automated Machine Learning Meth- ods, Systems, Challenges. The Springer Series on Challenges in Machine Learning, Springer, 2019. ISBN 978-3-030-05317-88 doi: 1111000/978-3-03005511111
Hadi S. Jomaa, Josif Grabocka, and Lars Schmidt- Thieme. Hyp-rl: Hyperparameter optimization by reinforcement learning. CoRR, abs/1906. 11527, 2019. URL http://arxiv.org abs/
1906. 11527
Hadi S Jomaa, Lars Schmidt- Schmidt-Thieme, Thieme, and Josif Grabocka. Dataset2vec: Learning dataset meta- features. Data Mining and Knowledge Discovery, pp. 1-22, 2021a.
Hadi Samer Jomaa, Sebastian Pineda Arango, Lars Schmidt-Thiene and Josif Grabocka. Transfer learning for bayesian hpo with end-to-end landmark meta-features. In Fifth Workshop on Meta- Learning at the Conference on Neural Information Processing Systems, 2021b.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Ad- vances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6402-6413, 2017.
Ekrem Öztürk, Fabio Ferreira, Hadi S. Jomaa, Lars Schmidt-Thiemee Josif Grabocka, and Frank Hutter. Zero-shot automl with pretrained models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 17138-17155. PMLR, 2022. URL https: //proceedings ings mlr. pre.s/ vv 62 wwt.ootw...aaooooo htmml
Sebastian Pineda Arango, Hadi Jomaa, Martin Wistuba, and Josif Grabocka. b: A large-scale reproducible benchmark for black-box hpo based on openml. In J. Vanschoren and S. Yeung (eds.). Proceedings of the Neural Information Process- ing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL https: Siaaaastts-eenchmaakr-rrooeee www.iinnn....n.seeeeeeeee cc/ppaper//222///11ee ec 1c996337a397777777777aaaaaa
Hpo-
Paper round2.pdl
Herilalaina Rakotoarison, Louisot Milijaona, Andry RASOANAIVO, Michele Sebag, and Marc Schoenauer. Learning meta-features for autoML. In International Conference on Learning Rep- resentations, 2022. URL
net wwt.:neeeeeeinieeeeeeeee www.emoienneeEEEeee...
David Salinas, Huibin Shen, and Valerio Perrone. A quantile-based approach for hyperparameter transfer learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, pp. 8438-8448, 2020.
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of meeting held De- cember 3-6, 2012, Lake Tahoe, Nevada, United States, pp. 2960-2968. 2968, 2012.
11

---

Published as a conference paper at ICLR 2023
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat, and Ryan P. Adams. Scalable bayesian optimization using deep neural networks. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-1 July 2015, pp- 2171-2180, 2015.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In D. Lee. M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Asso- ciates, Inc., 6a. URL hagaaa/ceeeeeeeboeeeaaaaaaaa1112
htw.:/ddiinnnnneuiieeeeeee cc//paper/2016/ille
pdf
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In Advances in Neural Information Processing Systems 29: Annual Conference Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain. pp. 4134 4342, 2016b.
Jiaxi Tang and Ke Wang. Ranking distillation: Learning compact ranking models with high performance for recommender system. In Proceedings of the 24th ACM SIGKDD Interna- fional Conference on Knowledge Discovery and Data Mining, KDD '18, pp. 2289-2298, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450355520. 3220021
doi:
112.0..11433222 10.1145/32198119.22220 122200... URL http::ttts///cctt httppps///a...r//1111111144.331.
Zi Wang, George E Dahl. Kevin Swersky, Chansoo Lee, Zelda Mariet, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. Pre-trained gaussian processes for bayesian optimization. arXiv preprint arXiv 109.08215, 2021.
Ying Wei, Peilin Zhao, Huaxiu Yao, and Junzhou Huang. Transferable neural processes for hy- perparameter optimization. CoRR, 10s/1999,,,29,, 2019. URL http:://aaxiii/ wtt.ivs/rrg/b///o/ 1909. 03209
Colin White, Willie Neiswanger, and Yash Savani. BANANAS: bayesian optimization with neural architectures for neural architecture search. In Thirty-Fii Fifth AAAI Conference on Artificial Intelli- gence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 10293-10301, 2021.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. In Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th International Con- ference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pp. 370-378, 2016.
Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Martin Wistuba and Tejaswini Pedapati. Learning to rank learning curves. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JML.. 2020.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thiend Learning hyperparameter optimiza- tion initializations. In 2015 IEEE International Conference on Data Science and Advanced Ana- lytics, DSAA 2015, Campus des Cordeliers, Paris, France, October 19-21, 2015, pp. 1-10, 2015. doi: 10.1109/DAAA.215..34449
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thiema Two-stage transfer surrogate model for automatic hyperparameter optimization. Paolo Frasconi, Niels Landwehr, Giuseppe Manco, and Jilles Vreeken (eds.), Machine Learning and Knowledge Discovery in Databases- European Conference, ECML PKDD 2016, Riva del Garda. Italy, September 19-23, 2016, Proceedings, Part I, volume 9851 of Lecture Notes in Computer Science, pp. 199 194-214. Springer, 2016.
Jian Wu and Peter I. Frazier. Practical Two-Step Ahoad Bayesian Optimization. Curran Asso- ciates Inc., Red Hook, NY, USA, 2019.
12

---

Published as a conference paper at ICLR 2023
Liang Wu, Diane Hu, Liangjie Hong, and Huan Liu. Turning clicks into purchases: Revenue opti- mization for product search in e-commerce. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 365-374, 2018
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.). Advances in Neural Information Processing Systems, nww.ips. cC/ #B80:22777744422221aaaa222262668
volume 30. Curran Associates, Inc., 2017. URL https:
13

---

Published as a conference paper at ICLR 2023
A BAYESIAN OPTIMIZATION WITH DEEP RANKING ENSEMBLES
Once the Deep Ensembles are trained, we aggregate the predictions for an input x following the procedure explained in Section 3.2 obtain μ(x), (a)) and conditioning to set of observations D. For the sake of simplicity, we omit this conditioning in our notation. These outputs can be fed in several types of acquisition functions and decide for the next point z to observe from the set of pending points to evaluate X. Notice that the lower rank, the better the configuration, therefore we formulate the cast the acquisition function as minimization problem. Specifically, we consider:
Average Rank: a(x,) = (l(2,)
Lower Confidence Bound: a(z,) = m(x,) - --
Expected Improvement: (zzz) = max (0, u(xx) r) N (1iA(zz),
Where 33 is a factor that trades of exploitation and exploration and my is the best-observed configura- tion, i... k = arg ( [P.I) 3i and µ(хх) is the average rank predicted for that configuration and yk is its validation error. The previous formulation assumes a minimization, thus to choose the next query point you apply: arg min a(zz).
Algorithm 2: Bayesian Optimization with DRE
Input A prior distribution over datasets p(D), initial observations
H = H ;i1))))))) ).... (IN, UN)). pending points X, number of BO iterations K, black-box function to optimize f
Output: Best observed configuration r.
Train ensemble of MLP scorers following Algorithm I and prior p(D);
for -
1 to K do Fine-tune/Train MLP scorers Suggest next candidate x = arg , ex a(z,,
Observe response y f(z);
Update history H = H U [(,,))}:
end
Return top performing configuration: arg min (r..))e y:
B EXPERIMENTAL SETUP FOR DEEP RANKING ENSEMBLES
Meta-Feature Extractor The DRE model has two configurable components: the meta-feature net- work and the scorers. The meta-feature extractor a DeepSet with an architecture similar to the one used by Jomaa et al. (2021a). However, we used 2 fully connected layers with 32 neurons each for both & and p (Deep Set parameters) instead of fully connected layers. The output size set to 16 by default.
Ensemble of Scorers The ensemble of scorers a group of MIP (Multilayer Perceptrons) with identical architectures. Each neural network has 4 hidden layers and each hidden layer has 32 neurons. The neural networks are initialized independently and randomly (for DRE-RI) or warm- initialized with the meta-learned weights. The input size of each neural network is 16 (the dimesi- ionality of the meta-features), plus the HP search space dimensionality. their output size is 1.
Setup for Motivating Example. For the creation of the Figure 2 MLP with 2 hidden layers and 10 neurons per layer. The meta-feature extractor has 4 layers and 10 neurons, and output dimensions equal to 10. The network is meta-trained for 1000 epochs, with batch size 10, learning rate 0.001, Adam Optimizer, and 10 models the ensemble. For the meta-learning example, we do not fine-tune the networks, while we fine-tune the networks for the non-meta-learned example for 500 iterations.
as
network an
14

---

Published as a conference paper at ICLR 2023
Table 1: Average Cost per BO Step (in seconds)
4796 (3 Dims)
5636 (6 Dims)
5527 (8 Dims)
5965 (I0 Dims)
5906 (16 Dims)
HEBO
0.27 + 0.18
3.11 + 1.68
2.66 0.95
3.21 1.78
2.85 + 2.43
FSBO
10.49 + 2.92
10.13 + 1.51
10.61 + 4.47
11.45 + 4.35
12.13 = 6,41
DRE
22.29 + 3.81
18.8 + 3.57
22.61 3.85
19.39 + 3.81
22.29 3.79
C DISCUSSION ON LIST SIZE AND LIST WEIGHTS
We present an additional ablation on the list size. We report the aver- age rank on the meta-validation split for different list sizes during meta- training on Figure 8a Notice, that un-
(a) List Size Ablation
(h) List Weights
4.0
1.5
small list size (10) leads derperforming setting. Therefore, it important to consider relative large list sizes (100- S n.).
Average Rank
3.5
Weight
3.0
During meta-testing i.e. by perform- ing BO, there is no significant over- head in terms of having a larger list size, because the true rank is derived from the observed validation accu- racy of configurations. During both meta-training, as well as the BO step, fit surrogate estimate the rank previously observed configu- rations that have been already evalu- ated. Given n observations, comput- ing the true rank is simple ((n-log(n)) sorting operation. Notice that in BO settings is typically small.
0.0
35
50
80
96
20
Number Evaluations
Position
Randon -1 -1-50
-1.-200
Log Linear
PDA
L-10
11-100
Figure 8: Effect of parameters in list-wise loss
There are several weighting schemes. Two alternatives to the weighting factor we use (inverse log weighting) are inverse linear weighting and position-dependen attention (PDA) (Chen et al.) (017)) As you can see in Figure inverse linear gives very small weight lower ranks, while the position- dependent gives too much importance. In this plot, PDA weights were scaled to make it comparable to the other schemes. We decided use the inverse log weighting because gives neither too low nor too high weight to lower ranks. For the j-th position in list with & elements, these weights can be described as follows:
Inverse Log: w(j) log(F+T)
Inverse Linear: w(j) =
Position-ddpendennt attention: w(j)
k-1+1 D
D DISCUSSION ON COMPUTATIONAL COST
We provide here a cost comparison between DRE, FSBO and HEBO. In the Table I we provide the average cost BO step (#: standard deviation) for different search spaces (with different di- mensions). DRE effectively incurs a cost higher than FSBO and HEBO, but <30 seconds, which is very small overhead compared to the cost of actually evaluating hyperparameter configurations (evaluation means the expensive process of training classifiers given the hyperparameter configura- tions and computing the validation accuracy).
15

---

Published as a conference paper at ICLR 2023
E ADDITIONAL PLOTS
We present additional results on the critical difference diagrams for i) Transfer methods results (Fig-
ure 9ap. ii) Non-Transfer (Figure 9b iii) Scorer size (Figure I0a iv) Acquisition Function (Figure
10b Ranking Loss (Figure Tab and vi Meta-features (Figure lb). These CD plots show the com-
parison of the performance different number of trials (e.g. at 25 trials Rank @25). The vertical lines connecting two methods indicate that their performances are not significantly different.
R2 Thansfer
Ranki125 Non-transder
Randomn
DRE
Rendem BOHA.
DRE-RI DNGO
TST
DKLM
TAF
FSBO
DKGP
HEBO
HEBO
RGPE
GP
050 Thansfe
Rankf150 Non-trauseer
Random
Randoma
HEBO
TST
DKLM
BOHA
DRE-RI
TAF
FSB0
CP
DNCO
RCPE
BEBO
DKGP
Rankö 100 Transfer
Banks1100 Non-transfer
Rendon
DRE
Randem
TAF TST RCPE
GP
DHE-RI
PSHO
BOHAA
DRGP
HEBO
DNGO
(a) Comparison vs. transfer methods
(b) Comparison vs. non-transfer methods
Figure 9: Critical Difference Diagram for a) Transfer and b)
Ranke025 Scorer size
R2 Acquisition Func.
Randonn
16x22
32x2
9233
Random Avg
1882
UCB
Ranku50 Scorer size
Ba Aequisition Fune.
Randoo 32x2
32xd
Randonn
64x2
Avg
UCB
disc2
Rank@100 Sooter size
Rank Acquisition Funr.
Randoon 16x2
32xc
Randon
Avg
UCB
32:2
64x2
(b) Ablation of the Acquisition Function
(a) Ablation of the DRE Scorer size
Figure 10: Critical Difference Diagram for the results of the ablation of DRE hyperparameters in (a) and the choice the acquisition function from Hypothesis (b).
16

---

Published as a conference paper at ICLR 2023
Rankw22 Ranking Loss
Hankw225 Meta- Features
Beg
W-List List
F.M
F.M.T M.T
Print
FT
Rank Ranking Loss
Rank 555 Meta- Features
Rog
FMT
Point
List
M.T
Pair
E.T
Rank@100 Ranking Loss
Rank@100 Meta- Features
Beg
W-F.ist Pair
P.M
F F.M.T
Point
M.T
(a) Ablation of Ranking Loss
(b) Ablation of Meta-Featte Neta-Features
Figure 11: Critical Difference Diagrams for the results Hypothesis in a) and Hypothesis 4 in b).
Space
Space
Space BE36
Space
Aretage Bants
Ranic
Aveung Bank
Avtige
Numile Evaltaatine
Number of Evaluntions
Numine Evalustion
Numner Evalintions
Space No. 1961
Syare 2801
Sparn No. 5006
Spare No. 70663
a
Averuge Hank
Avtrnge Bunk
Averuge Bank
30 65
86 20
50
56
20
60
30 10
Exalndoos
Fonluttion
Spece No. 1970
Space No. 2071
Space No. 6706
Spece No. 6777
Angure Rack
Anergge Bank
Averue Bankk
Ancage
20 3o
20
36 50
Nymber Poalattion
Number of Praluntions
Number of Bouluation
Number of Poalmations
Space No. 6754
Space No. 7017
No. 7030
Space No. 5895
Anerac Honcs
Anrrge Bonik
Aveuue Ronis
Number Eoainations
Number of Evaliations
Number of Evalnations
Number of Evalinatios
DRE
F390
RGPE
TAP
HEBO
TRLM
DCP
Rarsless
Figure 12: Average Rank per Search Space (Transfer Methods)
17

---

Published as a conference paper at ICLR 2023
No. 4796
Space No. 3527
Space No. 3636
5839
Average Rank
Avergge Rank 15
Avringe Hank
Average Hank
Number Evaluations
Number of Evaluations
Number Evaluations
Number Evaluations
Space
Space No.
Space 3906
Space
Average Btak
Avernge Bank 60
Averagg Bank
Averugg Hank
25 50
56
20 35
80 96
5 20
50 &
Number Exaluation
Number of Foaluatoos
Nummer of Hvalmations
Number Foulnations
Space No.
Space No. 3871
Space No. 6766
Space 6767
Averng Rank
Avnrage Bank
Average Bank
Avorage Hank
20 30 50 85 80
96
20 35
56 80
96
28 36 96 3
2 55 50 & 80 96
Number of Evaluation
Number of Evaluation
Number of Evaluations
Numaer of Evaluations
Space No. 6794
Space No. 7007
Space No. 7809
Space No. 3889
Average Bank
Averuge Banik
Averagr Back
16 Avrrngg Bank
20 3 50 G5 SI 50 Number of Doaluations
30 30 MI G SI Number of Enduutions
96
s0 Number of Evalustions
3
30
5 50 16
Number of Evahuation
DRE-R
HEDC
OKCP
DOHA.
DNGO
Random
Figure 13: Average Rank per Search Space (Non-Trass Transfer Methods)
18


Reviews:
[{"review_id": "ysb_86O8b-", "reviewer": "Reviewer_MGU4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new idea on hyperparameter optimization (HPO) by formulating it as a weighted learning to rank problem. By using a novel loss function, coupled with existing work on neural ensembles and transfer learning, the proposed method is shown to achieve state-of-the-art results in HPO.", "summary_of_the_review": "Overall I believe this paper propose a new idea in the important area of HPO, which is proved through extensive benchmark datasets to yield state-of-the-art results. I think it's a good paper for ICLR.  ", "strength_and_weaknesses": "Strength:\n- The idea presented is well motivated with clear mention to related work and background. \n- The method is proposed clearly.\n- The demonstration of the toy example is very intuitive. \n- Experiments are thorough and discussions on the 5 hypotheses are clear. \n\nWeaknesses: \nNot something significant but I have several questions for discussion:\n- The weight in loss function (1) seems to be from an older paper, and is not tuned (in experiment seems only compare with the non-weighted version)? Do we think there could be better weights available, especially for different tasks? \n- The rank of the configurations is a less well behaved surface to model compared to the original loss. Even if the underlying loss function is smooth, the rank could still be of large variability. Intuitively, how the method is able to perform better in this case? Consider a example where there are two local extremes, then modeling the original surface should be fine, but modeling the ranking can be harder due to the similarities of ranks near two local extremes? \n- The authors should discuss the efficiency of the algorithm compared to its competitors. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cEC7t9frAAf", "reviewer": "Reviewer_Eztd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new approach for hyperparameter optimization (Deep Ranking Ensembles) in both the transfer and non-transfer learning setting. \n\nThe are 3 main components to this approach:\n1. Use neural networks that learn to rank configurations as a surrogate model in the Bayesian optimization setting. \n2. Create diverse ensembles of these networks in order to perform uncertainty quantification (Lakshminarayanan et al., 2017), which is required to use acquisition functions like expected improvement \n3. Enhance the surrogate models by using additional learned dataset meta-feature representations (Jomaa et al., 2021).\n\nAn illustrative example of the method is provided, followed by an extensive evaluation on the HPO-B benchmark.\n", "summary_of_the_review": "This is a well-written paper with some interesting new ideas and impressive experimentation. Given the general applicability of the approach, I think the potential for impact is high. ", "strength_and_weaknesses": "Strengths:\n- While not all components here are novel (e.g., the method re-uses deep ensembles for uncertainty quantification and the dataset meta-feature approach from existing works), the core idea of using learning to rank inside BO surrogates is novel to the best of my knowledge, and the combination of this with the existing ideas is innovative and worthy of publication. \n- The illustrative example provided really helps the reader to grasp the main concepts, which is much appreciated. \n- Extensive experiments are provided comparing with many baselines in both the transfer and non-transfer context, in addition to a variety of different ablation studies, and statistical hypothesis testing is provided in the appendix.\n\nWeaknesses:\n- In Section 4.3 it is stated that during the meta-test phase the scoring network is training for 1000 epochs. How sensitive is the algorithm to the number of epochs used? It seems rather a lot, given that in early phase of BO one might only have a handful of configurations that have been evaluated. Are mini-batches used during the online training phase? If so, what was the batch size? If not, how is diversity ensured when training the scoring networks (it is stated in Section 3.2 that diversity in ensured by sampling different mini-batches). \n- No mention is provided about the runtime of DRE compared to other baselines. I understand that in the limit when it is very expensive to evaluate a hyperparameter configuration, this may not matter very much. However. it would give a lot of insight into whether this scheme could be used to solve more general classes of black-box optimization problems in which evaluating a configuration may be less expensive than training an ensemble of neural networks for thousands of epochs. \n- It was not very clear to me what is the role of the dataset meta-features in the non-transfer setting. Based on last sentence at the end of Section 4.3, it seems that the meta-feature extractor is trained also for DRE-RI. Could the authors comment on how these features might be expected to help in this context?\n- I appreciate the authors including the somewhat negative result comparing to HEBO in non-transfer setting (even if it is clearly surpasses by DRE in the transfer setting). Can the authors provide any explanation or intuition why HEBO might be winning in the non-transfer setting when the number of trials gets large. \n\nMinor corrections:\nSection 4.3 - I believe the text says \"32 networks\" when it should say \"32 neurons\"\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "OLQW9eMiDV", "reviewer": "Reviewer_7HGf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an HPO method that learns to select the best HP from the rankings of HPs. The main insight comes from the authors that the optimal strategy for training surrogates is to preserve the ranks of the performances of HPO as an L2R problem. More specifically, the method meta-learns neural network surrogates optimized for ranking the configurations' performance while modeling their uncertain via ensembling. The ranking surrogate will be used in the BO framework to help with HP suggestion.", "summary_of_the_review": "This work proposes an HPO method that learns to select the best HP from the rankings of HPs.  To the best of my knowledge, the ranking-based HPO method is quite novel. A comprehensive evaluation is done to show the effectiveness of the proposed method. However, I have concerns about how sensitive the method is on the ranking list size. I also have confusions about how the evaluation is done (see the Strength And Weaknesses). ", "strength_and_weaknesses": "**Strengths**:\n\n1. To the best of my knowledge, the ranking-based HPO method is quite novel.\n2. A comprehensive evaluation is done to show the effectiveness of the proposed method.\n\n**Concerns and questions**:\n\n1. My main concern about this work is the impact of the ranking list size. I do not find the actual definition of $N^{(s)}$ in Alg.1, but it seems to be the size of the ranking list in the rank scorer. In my opinion, this ranking list size is a crucial factor in building the ranking surrogate. Presumably, a small-size list is not that informative (one extreme case is the size being 1), and a larger size shall be preferred. However, larger size also means that the HPO algorithm needs to evaluate a large batch of configurations at each iteration (according to my understanding, computing the true rank requires one to evaluate the actual performance of the configs in the list). Thus I believe it is important to investigate the impact of this factor in this method. Currently, there is no discussion and no evaluation of this factor.\n\n2. A minor comment on the main hypothesis, \"the optimal strategy for training surrogates is to preserve the ranks of the performances of HPO as an L2R problem.\": Although the idea of using L2R to learn the ranking of HPs to help with HPO is a reasonably good idea and can be effective, I do not see it necessarily being an \"optimal\" strategy. Since in HPO, the ultimate objective is to find one config which has the best performance, does the full ranking necessarily matter? This work verifies some hypotheses but does not discuss thoroughly on this main hypothesis.\n\n3. A question on the computation cost: I wonder does training the scorer, and the neural network surrogates bring significant computation overhead to the HPO process. The empirical evaluation in this work only evaluates the performance regarding the number of trials and does not evaluate the end-to-end computation cost. I hope the authors can provide more information on the computation overhead of the rank-based surrogate and also the anytime performance over computation resources used instead of the number of trials.\n\n4. On the meaning of BO iterations, trials, and the actual number of evaluated configurations: I have confusion on (1) Does # of BO iterations = # of trials? (2) Does # of trials in the Figure 5 = # of actual hyperparameter configurations evaluated?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster