Hyperparameter Transfer Across Developer Adjustments
Authors: Danny Stoll
Jörg K.H. Franke
Diane Wagner
Simon Selg
Frank Hutter
Venue:  (2021)
Keywords: Meta Learning
Hyperparameter Optimization
Transfer Learning

Abstract: After developer adjustments to a machine learning (ML) algorithm, how can the results of an old hyperparameter optimization (HPO) automatically be used to speedup a new HPO? This question poses a challenging problem, as developer adjustments can change which hyperparameter settings perform well, or even the hyperparameter search space itself. While many approaches exist that leverage knowledge obtained on previous tasks, so far, knowledge from previous development steps remains entirely untapped. In this work, we remedy this situation and propose a new research framework: hyperparameter transfer across adjustments (HT-AA). To lay a solid foundation for this research framework, we provide four simple HT-AA baseline algorithms and eight benchmarks
changing various aspects of ML algorithms, their hyperparameter search spaces, and the neural architectures used. The best baseline, on average and depending on the budgets for the old and new HPO, reaches a given performance 1.2-3.6x faster than a prominent HPO algorithm without transfer. As HPO is a crucial step in ML development but requires extensive computational resources, this speedup would lead to faster development cycles, lower costs, and reduced environmental impacts. To make these benefits available to ML developers off-the-shelf and to facilitate future research on HT-AA, we provide python packages for our baselines and benchmarks.

Under review as a conference paper at ICLR 2021
HYPERPARAMETER TRANSFER ACROSS DEVELOPER ADJUSTMENTS
Anonymous authors Paper under double-bling review
ABSTRACT
After developer adjustments to a machine learning (ML) algorithm, how can the results of an old hyperparameter optimization (HPO) automatically be used speedup new HPO? This question poses challenging problem. as developer adjustments can change which hyperparameter settings perform well, or even the hyperparameter search space itself. While many approaches exist that leverage knowledge obtained on previous tasks, so far, knowledge from previous develop- ment steps remains entirely untapped. In this work, remedy this situation and propose a new research framework: hyperparameter transfer across adjustments (HT-AA). To lay solid foundation for this research framework, we provide four simple HT-AA baseline algorithms and eight benchmarks changing various aspects of ML algorithms, their hyperparameter search spaces, and the neural architectures used. The best baseline, on average and depending on the budgets for the old and new HPO, reaches given performance 1.2-3.6x faster than a prominent HPO al- gorithm without transfer. As HPO is crucial step in ML development but requires extensive computational resources, this speedup would lead faster development cycles, lower costs, and reduced environmental impacts. To make these benefits available MI developers off-tf and to facilitate future research on HT-AA, we provide python packages for our baselines and benchmarks.
Algorithm A
Algorithm A2
Algorithm As
Search Space X
Search Space X2
Search Space X'
Current
HPO
HPOº
HPO
HPO,
HPOΩ
HPO
Our Proposition
Automated
Automated
Knowledge Transfer
Knowledge Transfer
Hyperparameter optimization (HPO) across adjustments the algorithm or hyperparameter search space. A common practice is to perform HPO from scratch after each adjustment or somehow manually transfer knowledge. In contrast, we propose new research framework about automatic knowledge transfers across adjustments for HPO.
Graphical Abstract:
INTRODUCTION: A NEW HYPERPARAMETER TRANSFER FRAMEWORK
The machine learning (ML) community arrived at the current generation of ML algorithms by perform- ing many iterative adjustments. Likely, the way to artificial general intelligence requires many more adjustments. Each algorithm adjustment could change which settings of the algorithm's 's hyperparam- eters perform well, or even the hyperparameter search space itself (Chen et al. 2018 Li et al. (020)). For example, when deep learning developers change the optimizer, the learning rate optimal value

---

Under review as a conference paper at ICLR 2021
likely changes, and the new optimizer may also introduce hyperparameters. Since ML algorithms are known to be very sensitive to their hyperparameters (Chen et al. 2018 Feurer Hutter 2011)), developers are faced with the question of how adjust their hyperparameters after changing their code. Assuming that the developers have results of one or several hyperparameter optimizations (HPOs) that were performed before the adjustments, they have two options:
1. Somehow manually transfer knowledge from old HPOs.
This is the option chosen by many researchers and developers, explicitly disclosed, e.g., in the seminal work AlphaGo (Chen et al. 2018). However, this is not satisfying option since manual decision making time-consuming, often individually designed, and has already lead to reproducibility problems (Musgrave et al. 2020).
2. Start the new HPO from scratch.
Leaving previous knowledge unutilized can lead to higher computational demands and worse perfor- mance (demonstrated empirically in Section This is especially bad the energy consumption of ML algorithms is already recognized as an environmental problem. For example, deep learning pipelines have CO2 emissions the order of magnitude of the emissions of multiple cars for a lifetime (Strubell et al., 2019), and their energy demands are growing furiously: Schwwrtzz et al. (2019) cite a 300x0000x increase from 2012 to 2018". Therefore, reducing the number of evaluated hyperparameter settings should be a general goal of the community.
The main contribution of this work is the introduction of new research framework: Hyperparame- ter transfer across adjustments (HT-AA), which empowers developers with a third option:
3. Automatically transfer knowledge from previous HPOs.
This option leads to advantages in two aspects: The automation of decision making and the utilization of previous knowledge. On the one hand, the automation allows to benchmark strategies, replaces expensive manual decision making, and enables reproducible and comparable experiments; on the other hand, utilizing previous knowledge leads to faster development cycles, lower costs, and reduced environmental impacts.
To lay solid foundation for the new HT-AA framework, our individual contributions are as follows:
We formally introduce a basic version of the HT-AA problem (Section
We provide four simple baseline algorithms for our basic HT-AA problem (Section 3
We provide comprehensive set of eight novel benchmarks for basic HT-AA problem (Section 4).
We show the advantage of transferring across developer adjustments: some of our simple baseline algorithms outperform HPO from scratch up 1.2-3.6x on average depending the budgets (Section 5)
We empirically demonstrate the need for well-vetted algorithms for HT-AA: two baselines modelled after citualll-practicee manual strategies perform horribly on our benchmarks (Section 5))
We relate the HT-AA framework existing research efforts and discuss the research opportunities it opens up (Section 6).
To facilitate future research on HT-AA, we provide open-source code for our experiments and benchmarks and provide python package with out-of-the-box usable implementation our HT-AA algorithms.
HYPERPARAMETER TRANSFER ACROSS ADJUSTMENTS
After presenting a broad introduction to the topic, we now provide a detailed description of hy- perparameter transfer across developer adjustments (HT-AA). We first introduce hyperparameter optimization, then discuss the types of developer adjustments, and finally describe the transfer across these adjustments.
2

---

Under review as a conference paper at ICLR 2021
Add Remove Hyperparameters
Change Hyperparameter Range
Change Algorithm
Search Space
H
H MIZZ
H2
H2
H2
H2
H2
H2
Figure 1: Developer adjustments from the perspective of hyperparameter optimization.
Hyperparameter optimization (HPO)
The HPO formulation we utilize in this work is follows:
minimize fa with b evaluations
(1)
#EX
where fa(2) is the objective function for ML algorithm A with hyperparameter setting æ, b is the number available evaluations, and X' is the search space. We allow the search space contain categorical and numerical dimensions alike and consider only sequential evaluations. We refer to specific HPO problem with the 3-tuple (X', fA. b). For discussion on potential extensions of our framework to different HPO formulations, we refer the reader to Section 6
Developer adjustments We now put developer adjustments concrete terms and introduce taxonomy of developer adjustments. We consider two main categories of developer adjustments: ones that do not change the search space X' (homogeneous adjustments) and that do (heterogenous adjustments). Homogeneous adjustments could either change the algorithm's implementation or the hardware that the algorithm run on. Heterogeneous adjustments can be further categorized into adjustments that add or remove hyperparameter (hyperparameter adjustments) and adjustments that change the search space for specific hyperparameter (range adjustments). Figure shows an illustration of the adjustment types.
Knowledge transfer across adjustments In general, continuous stream of developer adjustments could be accompanied by multiple HPOs. We simplify the problem in this fundamental work and only consider the transfer between two HPO problems; we discuss a potential extension in Section The two HPO problems arise from adjustments V to a MI. algorithm Anta and its search space Vosa. which lead to Anew: Vnow = V (Aood:: Xaia). Specifically, the hyperparameter transfer across adjustments problem is to solve the HPO problem (Nuow, f bnew), given the results for (Nalds fAu bola). Compared to HPO from scratch, developers can choose lower budget bnews given evidence for transfer algorithm achieving the same performance faster.
Relation to hyperaarameter transfer across tasks (HT-AT) There exists an extensive research field that studies the transfer across tasks for HPOs (Vanschoren, 2018). The main difference to hyperparameter transfer across adjustments is that HT-AT assumes that there ware, ML algorithm, and search space, whereas HT-AA is characterized by these changes. Another difference is that most work HT-AT considers large amounts of meta-data (up more than thousand previous HPO runs and function evaluations (Wang et al. 2018 Metz et al. 2020 whereas in the basic HT-AA problem described above, only one previous HPO run available. Compared to HT-AT approaches, the main technical challenge for HT-AA approaches are heterogeneous developer adjustments, as homogeneous HT-AA problems, where none of the adjustments changes the search space, are syntactically equivalent to HT-AT problems with only one prior HPO run. For this homo- geneous HT-AA, existing approaches for HT-AT could, principle, be applied without modification; this includes, for example the transfer acquisition function (Wistuba et al. 2018), multi-task bayesian optimization (Swersky et al. 2013), multi-task adaptive bayesian linear regression (Perrone et al. 2018), ranking- weighted gaussian process ensemble (Feurer et al. 2018). and difference difference-moddlllngg bayesian optimisation (Shilton et al. 2017). However, the low amount meta-data in (homogeneous) HT--A poses additional challenges.
changes to hard-

---

Under review as a conference paper at ICLR 2021
3 BASELINE ALGORITHMS FOR HT-AA
In this section we present four baselines for the specific instantiation of the hyperparameter transfer across adjustments (HT-AA) framework discussed in Section [ We resist the temptation introduce complex approaches alongside new research framework and instead focus on solid foundation. Specifically, we focus on approaches that do not use any knowledge from the new HPO for the transfer and include two baselines that are modelled after cectually-pacctiin manual strategies. We first introduce the two basic HPO algorithm that the transfer approaches build upon, then introduce two decompositions of HPO search spaces across adjustments, and finally, present the four baselines themselves.
3.1 BACKGROUND: HYPERPARAMETER OPTIMIZATION ALGORITTM
We instantiate our transfer algorithms with two different basic hy hyperaaraeeter optimization algo- rithms, and also evaluate against these two algorithms (Section 5). Both algorithms are based on Bayesian optimization (BO): Given observations D {( (x(x } k= BO fits probabilistic model AfA D) for the objective function f.:: then, to decide which hyperparameter setting to evaluate next, BO maximizes acquisition function axx 'D,, which uses this probabilistic model. For both approaches we follow standard practice and sample configurations randomly in fraction of cases (here V/3.. For recent review of BO refer to (Saahriari et al. (2016). In the following, provide details for the two basic HPO algorithms which we use.
Tree-Structured Parzen Estimator One of the two algorithms that we use is the Pree-Structured Parzen Estimator (TPE) BBergstra al. 20IT), which is the default algorithm in the popular HyperOpt package (Bergstra et al. 2013). TPE uses kernel density estimators to model the densities (xx) and g(æ)), for the probability of given hyperparameter configuration being worse (l), better (g), than the best already evaluated configuration. To decide which configuration to evaluate, TPE then maximizes the acquisition function x" arg = s(al/(()) approximately. In our experiments, we use the TPE implementation and hyperparameter settings from Falkner et al. (2018). which requires ((dim(XXee 1) evaluations build a model.
Bayesian optimization with Gaussian processes Bayesian optimization with Gaussian processes uses Gaussian process regression (Rasmussen & Williams 2006) to fit probabilistic model p( f D). To generate a sample, we optimize the widely used expected improvement (Jones et 1998 acquisition function. If there are less than dim(X') evaluations, we sample randomly. In our experiments, we use the implementation from the SMAC3 package (Lindauer al. 2017) with default kernels, hyperparameters, and optimization procedures. In the following we will refer to BO with Gaussian processes with GP.
3.2 PRELIMINARIES: SEARCH SPACE DECOMPOSITIONS
Hyperparameter adjustments For hyperparameter adjustments the new search space Nnew and the old search space Xold only differ in hyperparamtters, decompose the search spaces as Xnew = Xonly-new X Xnoeh and Xold Nooth X Konty-old. where Nooeh the part of the search space that remains unchanged across adjustments (see Figure 2 for reference). All baselines use this decomposition and project the hyperparameter settings that were evaluated in the old HPO from Vota to Abothh
in hyperparameter ranges, so we can
Range adjustments
range adjustment can remove values from the hyperparameter range or add values. For an adjustment of hyperparameter range XHI, to XHI, this can be expressed as XHH, = N U HHH
bech.raggo-oll--ne with N = XHI \ xHH Abbuh.anng-oonly-oo
3.3 BASELINE ALGORITHMS
Only Optimize New Hyperparameters A common developer strategy to manually transfer across adjustments is to set previous hyperparameters in Xnvth to the best setting of the previous HPO and only optimize hyperparameters in nonly (Agostinelli et al. 2014 Huang et al. 2017 Wu & He 2018).
4

---

Under review as a conference paper at ICLR 2021
oonly-odd
both
Xonly-new
Xold
H
H
H3
Xnew
H2
H3
H4
Transfer Sample
Regular Sample
Figure 2: Example search space decomposition for hyperparameter addition and removal.
the previous best setting not valid anymore, i.e., it has values in XH bath.ang--om old for a hyperpa- rameter H; still in Xbat, this strategy uses the best setting that still valid configuration. In the following, we refer to this strategy as only-optimize-new.
Drop Unimportant Hy perpaaarrsss A strategy inspired by manual HT-AA efforts is only optimize important hyperparameters. The utilization of importance statistics was, for example, explicitly disclosed in the seminal work AlphaGo (Chen al. 2018). Here, determine the importance of each individual hyperparameter with functional analysis of variance (fANOVA) (Hutter al., 2014) and do tune hyperparameters with below mean importance. Therefore, this strategy only optimizes hyperparameters in Konty-new and hyperparameters in Nooth with above mean importance. In the following, we refer this strategy cooop-unimporann..
First Evaluate Best The best-first strategy uses only-optimize-new for the first evaluation, and uses standard GP for the remaining evaluations. This strategy has a large potential speedup and low risk as falls back to standard GP.
Transfer GP (TGP) / Transfer TPE (T2PE) The main idea of Transfer GP (TGP) Trans- fer TPE (T2PE) is to fit GP TPE model the observations of the previous HPO Dala for the part of the search space that remained unchanged (i.c., Nooth for refer- ence see Figure 2p. To this end, TGP/T2PE first discards now invalid hyperparameter settings and projects the remaining settings I; Doia from Nata Xooth to yield Dold.btth Then, either GP or TPE model Mbon is fit on these transformed previous results Dold.both. and to generate sample x Xnew for the new HPO, sample from Mbath over Xnnth combined with a random sample over Konly- Further, as in the basic HPO algorithms, TGP/T2PE uses random sample over &new in 1/3 of cases, and after (nww (TGP) or 2(dim(X ) + 1) (T2PE) evaluations, TGP/T2PE uses a model fitted the new observations. We provide pseudocode for TGP/T2PE in Algorithm [
{(E,, fAm (ax)
Algorithm Sampling strategy in transfer GP/TPE
Input: Current search space Xnew, previous search space Vous previous results Dota. budget nnw
1: Decompose X (Nooth Xooth bbothh.ong--onl--now/
Konly-new
See Section 3.2
2: Discard hyperparameter settings Dota that have values Kbuth.range-only-nn
3: Project configs in Dota to space Abaus to yield Dold.ott
4: Fit TPE or GP model Mooth. for Xnoth on DolLboth
5: for in 1, ..,, Unew do
6: if is random fraction then
D. See Section
3.1
7: Sample Xnow from prior on Xeew
8: else if no model for Xnew then
>> Differs for TPE and GP, see Section 3.]
9:
Sample Xhuth from Xhoth using Mouth
10:
Sample new from prior on Xealy- Combine Xboth with Xonly-new to yield sample Xnew
12:
else
13:
Fit TPE or GP model Mnew for Xnew on current observations
14:
Sample Xaew from Knew using Maew

---

Under review as a conference paper at ICLR 2021
4 BENCHMARKS FOR HT-AA
We introduce eight novel benchmarks for the basic hyperparameter transfer across adjustments (HT-AA) problem discussed in Section 2
As is common hyperparameter optimization research, we employ tabular and surrogate benchmarks to allow cheap and reproducible benchmarking (Perrone) ett al.] 2018 Falkner al. 2018). Tabular benchmarks achieve this with a lookup table for all possible hyperparameter settings. In contrast, surrogate benchmarks fit surrogate model for the objectíve function (Eggensperger et al., 2014). We base our surrogate models and lookup tables on code and data from existing hyperparameter optimization (HPO) benchmarks (detailed in Appendix (A) that, together, cover four different machine learning algorithms: a fully connected neural network (FCN), neural architecture search for convolutional neural network (NAS), support vector machine (SVM), and XGBoost (XGB). While the NAS and FCN benchmarks are based on lookup tables, the SVM and XGB based benchmarks use surrogate models. For each of these base benchmarks, we simulate two different types of developer adjustments (Table D. to arrive at a total of eight benchmarks. Additionally. for each algorithm and adjustment, we consider multiple tasks in our benchmarks. Further, we provide a python package with all our benchmarks and refer the reader to for additional details on the benchmarks.
Appendix A
Table I: Developer adjustments and types of adjustments in the benchmarks.
Benchmark
Adjustments
Adjustment Type
FCN-A
Increase #units-per-layer 16x
Homogeneous
Double #epochs
Homogeneous
Fix batch size hyperparameter
Heterogeneous
FCN-B
Add per-layer choice of activation function Change learning rate schedule
Heterogeneous
Homogeneous
NAS-A
Add 3x3 average pooling as choice of operation to each edge
Heterogeneous
NAS-B
Add node to cell template (adds 3 hyperparameters)
Heterogeneous
XGB-A
Expose four booster hyperparameters
Heterogeneous
XGB-B
Change four unexposed booster hyperparameter values
Homogeneous
SVM-A
Change kernel
Homogeneous
Remove hyperparameter for old kernel Add hyperparameter for new kernel
Heterogeneous
Heterogeneous
SVM-B
Increase range for cost hyperparameter
Heterogeneous
EXPERIMENTS AND RESULTS
In this section, we empirically evaluate the four baseline algorithms presented in Section solutions for the hyperparameter transfer across adjustments problem. We first describe the evaluation protocol used through all studies and then present the results.
Evaluation protocol We the benchmarks introduced in Section 4 and focus on the speedup of transfer strategies over either Bayesian optimization with Gaussian processes (GP), or over Tree Parzen Estimator (TPE). Specifically, we measured how much faster a transfer algorithm reaches a given objective value compared to GP or TPE in terms of the number of evaluations. We repeated all measurements across 25 different random seeds and report results for validation objectives, as not all benchmarks provide test objectives, and to reduce noise in our evaluation. We terminate runs after 400 evaluations and report ratio means. To aggregate these ratios across tasks and benchmarks, we use the geometric mean. We use the geometric mean, as, intuitively, two speedups of e.g., 0x and 10x average to 1x, and not 5.05x. We want to note that the standard mean is an upper bound for the geometric mean, using the geometric mean in fact makes the speedups slightly less impressive than had we used the standard arithmetic mean. To determine the target objective values, we measured GP's average performance for 10, 20, and 40 evaluations. We chose this range of evaluations

---

Under review as a conference paper at ICLR 2021
Table 2: Average speedup across benchmarks for different #evaluations for the old and new HPO. For the GP based (left) and TPE based (right) evaluation.
#Evals Old
#Evals New
Best First
Transfer GP/TPE
Best First + Transfer GP/TPE
10
10
1.6
/
1.5
1.3
/
1.0
1.7
1.7
1,4
1.4
1.2
1.0
1.4
1.4
40
1.1
1.2
1.1
/ 1.1
1.2 1.2
20
10
2.3
/
1.8
1.7 / 1.3
2.6
2.1
20
1.7
1.4
1.5
/ 1.2
2.0
1.8
40
1.3
1.1
1.2
1.1
1.4
1.5
40
3.3
2.6
2.1
1.5
3.6
2.6
20
2.6
2.0
2.0
/
1.4
2.9
2.0
1.8
1.4
1.5
1.2
2.1
1.5
survey among NeurIPS2019 and ICLR2020 authors indicates that most hyperparameter optimizations (HPOs) do not consider more than 50 evaluations (Bouthillier & Varoquaux (020). Further, for transfer approaches, we perform this experiment for different evaluation budgets for the HPO before the adjustments (also for 10, 20, and 40 evaluations).
Results The transfer GP (TGP) / transfer TPE (T2PE) and best-first strategy lead to large speedups, while trop-unimporttnt and only-optimize-neet new perform poorly. Here, in this paragraph we focus on the GP based evaluation. On average and depending on the budgets for the old and new HPO, TGP reaches the given objective values 1.1-2.1x faster than GP, and best-first 1.1-3.3x faster. The combination of TGP/T2PH and best-first leads to further speedups over best-first if the budget for the 40, and when the old budget and new budget both equal 10. These additional average speedups when combining TGP with best-first are between 0.1 and 0.3. We visualize these results with violin plots (Figure 3 as they take into account the multi modality the results, and also provide table with exact averages (Table There two main trends visible: (1) The more optimal the target objective, the smaller the speedup, and (2) the higher the budget for the previous HPO, the higher the speedup. For a more fine-grained visualization that shows violin plots over task averages for each benchmark, we refer to Appendix manual strategies, drop-unimportant and only-optimize-neww do not reach the performance of GP in a large percentage of (20%-7095 on average), even while given at least 10x the budget compared GP These high failure rates makes an evaluation for the speedup unfeasible and indicates that paacttcce manual strategies perform worse than starting from scratch. For a visualization of the failure rates of trop-unimmoottat and only-optimize-nett and for GP, TGP, and best-first (0-0.8%) we refer the reader to Appendix C
old HPO was
B The approaches inspired by actually-practicas
Additionally, we provide study on the improvement in objective in Appendix D Appendix we show the results of a control study that compares GP and TPE with different ranges of random seeds; and in Appendix we compare GP and TPE to random search to validate their reliability.
RELATED WORK AND RESEARCH OPPORTUNITIES
In this section, we discuss work related hyperparameter transfer across adjustments (HT-AA) and present several research opportunities in combining existing ideas with HT-AA
Transfer learning Transfer learning studies how to use observations from one or multiple source tasks to improve learning one or multiple target tasks İZhuang et al. 2019). If we view the HPO problems before and after specific developer adjustments as tasks, we can consider HT-AA as specific transfer learning problem. As developer adjustments may change the search space, HT-AA would then be categorized as hetcrogeneous transfer learning problem (Day & Khoshgoftaar 2017).
Transfer learning across adjustments Recently, Berner et al. (009)) transferred knowledge be- tween deep reinforcement learning agents across developer adjustments. They crafted techniques to preserve, approximately preserve, the neural network policy for each type of adjustment they
7

---

Under review as a conference paper at ICLR 2021
10 Previous Evaluations
20 Previous Evaluations
40 Previous Evaluations
13
12
11
Speedup Over GP
9
8
7
3
12
Speedup Over TPE
11
6
2
10
20
40
10
20
40
10
20
40
GP/TPE Evaluations for Reference Objective [#]
Best First
Best First + Transfer GP/TPE
Transfer GP/TPE
Figure 3: Speedup reach given reference objective value compared GP/TPE for best-first, best-first combined with transfer GP/TPE, and transfer GP/TPE (from left to right each violin triplet) across 8 benchmarks. The violins estimate densities of benchmark geometric means. The horizontal line in each violin shows the geometric mean across these benchmark means. #Evaluations for the old HPO increases from left right. The x-axis shows the budget for the GP and TPE reference. Note that the GP and TPE runs we used to determine the reference objective are different the ones that we show the speedups over.
encountered. Their transfer strategies inspired Net2Net knowledge transfer (Chen et al. 2013). and they use the term surgery to refer to this practice. Their work indicates that transfer learning across adjustments is not limited to knowledge about hyperparameters, but extends more general setting, leaving room for many research opportunities.
Continuous knowledge transfer In this paper, we focus on transferring knowledge from the last HPO performed, but future work could investigate a continuous transfer knowledge across many cycles of adjustments and HPOs. Transferring knowledge from HPO runs multiple previous versions could lead further performance gains, as information from each version could be useful for the current HPO. Such continuous HT-AA would then be related to the field of continual learning (Thrun Mitchell [995] Lange et al. 2020
Hyperparameter transfer across tasks (HT-AT) While we have related HT-AA to HT-AT in Section 2 here, we want to complement this discussion with the research opportunities we see in НT-AA based on НT-AT. First, an adaptation of across-task strategies, e.g., the ones cites in Section 2 the across-adjustie arross-adjustments setting could lead to more powerful HT-AA approaches in the future. Second, the combination of across-tase task and across-adjustmet hyperparameter transfer could provide even larger speedups than either transfer strategy its
8

---

Under review as a conference paper at ICLR 2021
10 Previous Evaluations
20 Previous Evaluations
40 Previous Evaluations
80
Failed Runs [%)
60
40
20
0
10
20
40
10
20
40
10
20
40
GP Evaluations for
Reference Objective [#]
Drop Unimportant
Only Optimize New
Figure 4: Percent of runs that do not reach the reference objective for the GP based poop-unimporant and only-optimize-new. Each data point for the violins represents the mean percentage of failures for a benchmark. The line each violin shows the mean across these benchmark means. #Evaluations for the old HPO increases from left to right. The x-axis shows the budget for the GP reference.
Advanced hyperparameter optimization HT-AA be combined with the many extensions to hyperparameter optimization (HPO). One such extension multi multi-fidelity HPO, which allows the use of cheap-to-evaluate approximations the actual objective (Li al. 2017 Falkner et al. 20188. Similarly, cost-aware HPO adds a cost to each hyperparameter setting, so the evaluation of cheap settings can be prioritized over expensive evaluation noise into account |errstigg et al. 2007). or to consider not one. but multiple objectives to optimize for (Khan al.) (002). All these HPO formulations can be studied in conjunction with HT-AA, to either provide further speedups or deal with more general optimization problems.
(Snoek et al. 2012). Yet another extension is to take
Guided machine learning The field of guided machine learning (gML) studies the design of interfaces that enables humans guide ML processes (Westphal et al.) 2019). An HT-AA algorithm could be viewed as a ML algorithm that receives incremental guidance in the form of arbitrary developer adjustments; the interface would then be the programming language(s) the MI. algorithm is implemented in. On different note, gML. could provide HT-AA algorithms with additional information about the adjustments the ML algorithm. For example, when adding a hyperparameter, there are two distinctions component added to the algorithm that introduces a new hyperparameter From the HPO problem itself, cannot know which case it is, and neither which fixed value exposed hyperparameter had. Guided HT-AA algorithms could ask for user input to fill this knowledge gap, or HT-AA algorithms with code analysis could extract this knowledge from the source code.
can make: Either an existing hyperparameter is exposed
new
Programming by optimization The programming by optimization (PbO) framework (Hoos 2012) proposes the automatic construction of a search space of algorithms based on code annotations, and the subsequent search in this search space. While PbO considers changing search spaces developer actions, each task and development step restarts the search from scratch. This is in contrast HT-AA, which alleviates the need restart from seratch after each developer adjustment.
CONCLUSION
In this work, we introduced hyperparameter transfer across developer adjustments to improve effi- ciency during the development of ML algorithms. In light of rising energy demands of ML algorithms and rising global temperatures, more efficient ML development practices are an important issue now and will become more important in the future. As already considered in this work lead to large empirical speedups, our new framework represents a promising step towards efficient MI. development.
of the simple baseline algorithm

---

Under review as a conference paper at ICLR 2021
REFERENCES
Forest Agostinelli, Matthew Hoffman, Peter Sadowski, and Pierre Baldi. Learning activation functions to improve deep neural networks. arXiv preprint arXiv:14/2.6830, 2014.
James Bergstra, Daniel Yamins, and David Cox. Making science of model search: Hyperparameter optimization in hundreds dimensions for vision architectures. In International conference on machine learming, pp. 115- 123, 2013.
James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for parammter optimization. In Advances in neural information processing systems, pp. 2546-2554, 2011.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint
2019.
Xavier Bouthillier and Gaël Varoquaux. Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020. Research report, Inria Saclay Ile de France, January 2020. URL
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv: 1,05611, 2015.
Yutian Chen, Aja Huang, Ziyu Wang, Ioannis Antonoglou, Julian Schrittwieser, David Silver, and Nando de Freitas. Bayesian optimization in alphago. arXiv preprint arXiv:1812.06855, 2018.
Oscar Day and Taghi M Khoshgoftaar. survey on heterogeneous transfer learning. Journal of Big Data, 4(1):29, 2017.
Xuanyi Dong and Yang. Nas-hench-201: Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations, 2019.
K. Eggensperger, F. Hutter, Hoos, and K. Leyton-Brown Surrogate benchmarks for hyperparameter optimization. In ECAI workshop Metalearning and Algorithm Selection (MetaSel" 14), 2014.
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter op- timization at scale. In Proceedings of the 35th International Conference Machine Learning (ICML 2018), pp. 1436-1445, July 2018.
Matthias Feurer and Frank Hutter. Hyperparameter optimization. In Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.), Automatic Machine Learning: Methods, Systems, Challenges, pp. 3-38. Springer, 2019. Available at http: htt::///utmmmmm http:://autttoolooo/bbbb org/book
Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for bayesian optimiza- tion. arXiv preprint 1rXiv:1802.022. 2018.
Holger H Hoos. Programming by optimization. Communications the ACM, (5(2)570-88)3 80, 2012.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Hutter, H. Hoos, and K. -oyton-Broww. An efficient approach for assessing hyperparameter importance. In E. Xing and T. Jebara (eds.), Proceedings of the 31th International Conference on Machine Learning, (ICML' pp. 754-762. Omnipress, 2014.
Donald R Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455-492, 1998.
Kristian Kersting, Christian Plagemann, Patrick Pfaff. and Wolfram Burgard. Most likely het- croscedastic gaussian process regression. In Proceedings of the 24th international conference on Machine learming, pp. 993-400, 2007.
10

---

Under review as a conference paper at ICLR 2021
Nazan Khan, David E Goldberg. and Martin Pelikan. Multi-objective bayesian optimization algorithm. In Proceedings of the 4th Annual Conference on Genetic and Evolutionary Computation, pp. 684 684. Citeseer, 2002.
Aaron Klein and Frank Hutter. Tabular benchmarks for joint architecture and hyperparameter optimization. arXiv preprint UrXiv:1905.04977. 2019.
Daniel Kühn, Philipp Probst, Janek Thomas, and Bernd Bischl. Automatic exploration of machine learning experiments on openml. arXiv preprint 1rXiv: 2018.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting classification tasks, 2020.
Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Ste- fano Soatto. Rethinking the hyperparameters for fine-tuning. In International Conference on Learn- ing Representations, 2020. URL
L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: Bandit- configuration evaluation for hyperparameter optimization. In OpenReview.ne (ed.), Proceedings of the International Conference Learning Representations (ICLR' 2017.
M. Lindauer, K. Eggensperger, M. Feurer, S. Falkner, A. Biedenkapp. and Hutter F. Smac v3: Algorithm configuration in python. https://gil http://gttttttteeg//e wwwbb..ooaautoooto /SMAC3 2017.
Luke Metz, Niru Maheswaranathan, Ruoxi Sun, C. Daniel Freeman, Ben Poole, and Jascha Sohl- Dickstein. Using a thousand optimization tasks to learn hyperparameter search strategies, 2020.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. arXiv preprint arXiv:2003.08505, 2020.
Valerio Perrone, Rodolphe Jenatton, Matthias W Seeger, and Cédric Archambeau. Scalable hyperpa- rameter transfer learning. In Advances in Neural Information Processing Systems, pp. 6845-6855, 2018.
C. Rasmussen and C. Williams. Gaussian Processes for Machine Learning, The MIT Press, 2006.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. arXiv preprint arXiv: /907. 10597, 2019.
B. Shahriari, K. Swersky, Z. Wang, R. Adams, and N. de Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings the IEEE, 104(1):148 175, 2016.
Alistair Shilton, Sunil Gupta, Santu Rana, and Svetha Venkatesh. Regret Bounds for Transfer Learning in Bayesian Optimisation. In Aarti Singh and Jerry Zhu (eds.). volume 54 Proceedings of Machine Learning Research, pp. 307-315, Fort Lauderdale, FL, USA, 20-22 Apr 2017. PMLR. ngs m. wwwps/esssssp.. shi ton 7a html
URL
Snoek, H. Larochelle, and R. P. Adams. Practical Bayesian optimization of machine learning algorithms. In P. Bartlett, F. Pereira, C. Burges, L. Bottou, and K. Weinberger (eds.). Proceedings of the 26th International Conference on Advances in Neural Information Processing Systems (NIPS' (2), pp. 2960-2968, 2012.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3645-3650, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P199 1355. URL https: www. ac www.:awwwwb.c..m www.cal.web.oogg//ttttooo./////p 1355
Kevin Swersky, Jasper Snock, and Ryan P Adams. Multi-task bayesian optimization. In Advances in neural information processing systems, pp. 2004-2012, 2013.
11

---

Under review as a conference paper at ICLR 2021
Sebastian Thrun and Tom M. Mitchell.
Lifelong robot learning.
Robotics and Au-
tonomous Systems, 15(1):25
46. 1995. ISSN 0921-8890. doi: htps://doi.org///
10160091--89099500044.
URL http:/ www. rect.. com/ com/sciiecee
The Biology and Technology of Intelligent Au-
tonomous Agents.
Joaquin Vanschoren. Meta-learning: survey. arXiv preprint arXiv:1810.0354 2018.
Zi Wang, Beomjoon Kim, and Leslie Pack Kaelbling. Regret bounds for meta bayesian optimization with unknown gaussian process prior. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 10477-10488. Curran Associates, Inc., 2018.
Florian Westphal, Niklas Lavesson, and Häkan Grahn. A case for guided machine learning. In Andreas Holzinger, Peter Kieseberg, A Min Tjoa, and Edgar Weippl (eds.), Machine Learning and Knowledge Extraction, pp. 353-361, Cham, 2019. Springer International Publishing. ISBN 978-3-030-29726-8.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme Scalable gaussian process-basee transfer surrogates for hyperparameter optimization. Machine Learning, 107(():43-78, 2018.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongho Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. arXiv preprint arXiv:1911.02683 2019.
12

---

Under review as a conference paper at ICLR 2021
A
BENCHMARK SUITE DETAILS
A.1
OVERVIEW
Table Benchmarks overview
Benchmark
#Hyperparameters Old
#Hyperparameters New
#Tsks
Metric
FCN-A
5
MSE
FCN-B
8
MSE
NAS-A NAS-B
6
Accuracy
3
6
3
Accuracy
XGB-A
9
10
AUC
XGB-B
6
10
AUC
SVM-A
2
10
AUC
SVM-B
2
10
AUC
A.2 FCN-A & FCN-B
Base benchmark We use code and data from (Klein & Hutter 2019). We use the tasks HPO- Bench-Protein, PPO-Bench-SSlie,, JPPO-Benh-NNaall and Pr0--Bench-Parkinson
Budget For FCN-A the budget is set 100. For FCN-B, additional the changes in the search space (Table the budget is increased from 50 to 100 epochs.
Table 4: Values for integer coded hyperparameters FCN benchmarks
Hyperparameter
Values
Units Layer [1,2)
(16, 32, 64, 128, 256, 512) (0.0, 0.3, 0.6)
Dropout Layer [1,2)
Initial Learning Rate (0.0005, 0.001, 0.005, 0.01, 0.05, 0.1)
Batch Size
(8, 16, 32, 64)
Table Search spaces in FCN-A. Numerical hyperparameters are encoded as integers, see Table for specific values for these hyperparameters.
Steps
Hyperparameter
Range/Value
Prior
1
Units Layer 1
Units Layer 2
Batch Size
{0, ..., 3)
Uniform
1, 2
Dropout Layer
{0. .... 2}
Uniform
1,2
Dropout Layer
{0. ...., 2]
Uniform
1,2
Activation Layer
[Relu, tanh)
Uniform
1,2
Activation Layer 2
(ReLu, tanh)
Uniform
1,2
Initial Learning Rate
{0, ...,, 5}
Uniform
1,2
Learning Rate Schedule
Constant
Uniform
2
Units Layer 1
2
Units Layer 2
5
2
Batch Size
13

---

Under review as a conference paper at ICLR 2021
Table Search spaces in FCN-B. Numerical hyperparameters are encoded as integers, see Table for specific values for these hyperparameters.
Steps
Hyperparameter
Range/Value
Prior
Activation Layer 1
tanh
Activation Layer 2
tanh
1
Learning Rate Schedule
Constant
1,2 2
Units Layer 1
{0, ................................ 5}
Uniform
1,
2
Units Layer 2
{0, ................................ 5]
Uniform
1,2
Dropout Layer
{0, ................................ 2]
Uniform
1,2
Dropout Layer 2
(0...., 2]
Uniform
1, 2
Initial Learning Rate
{0, ..... 5)
Uniform
1,2
Batch Size
..,..., 3}
Uniform
2
Activation Layer 1
[ReLu, tanh)
Uniform
Activation Layer 2
[ReLu, tanh)
Uniform
2
Learning Rate Schedule
Cosine
A.3 NAS-A & NAS-B
Base benchmark We use code and data from (Dong & Yang 2019). We use the tasks CIFARIO, CIFARIO0, and ImageNet.
Table Search spaces in NAS-A.
Steps
Hyperparameter
Range/Value
Prior
1,2
0 4 2
none, skip-connect, conv 1x1, conv3x3, avg-poo13x3
Uniform
1,2
0
none, skip-connect, conv 1x1, conv3x3, avg-pool3x3
Uniform
1,2
2 - 3
none, skip-connect, conv lx1, conv3x3, avg-pool3x3
Uniform
2
0+1
none, skip-connect, conv conv3x3, avg-pool3x3 none, skip-connect, conv lx1, conv3x3, avg-pool3x3 none, skip-connect, conv lx1, conv3x3, avg-pool3x3
Uniform
1-2
Uniform
1
Uniform
Table Search spaces in NAS-B.
Steps
Hyperparameter
Range/Value
Prior
0+1 1
none, skip-connect, conv Ixl, conv3x3 none, skip-connect, convix1, conv3x3 none, skip-connect, convixl, conv3x3
Uniform
0 | 2
Uniform
0
Uniform
1
none, skip-connect, convixl, conv3x3
Uniform
1-3
none, skip-connect, conv lx1, conv3x3
Uniform
2 2
none, skip-connect, conv lxl, conv3x3
Uniform
0+1
none, skip-connect, convix1, conv3x3, avg-pool3x3
Uniform
2
0
none, skip-connect, conv lx1, conv3x3, avg-pool3x3 none, skip-connect, conv 1x1, conv3x3, avg-poo13x3 none, skip-connect, conv 1x1, conv3x3, avg-poo13x3 none, skip-connect, conv 1x1, conv3x3, avg-poo13x3 none, skip-connect, conv 1x1, conv3x3, avg-poo13x3
Uniform
0+3
Uniform
2
1-2
Uniform
1-3
Uniform
2
2 2
Uniform
A.4 SVM-A & SVM-B
Base benchmark We use an open source implementation by the HPOlib authors following Perrone
al. 2018). This implementation uses data from Kühn et al. (2018) and employs random forest
14

---

Under review as a conference paper at ICLR 2021
surrogate model (Eggensperger et al., 2014). For our benchmarks, randomly selected the ten tasks monks-problems- tic- vic-tac-toe, kc2, monks Bronks-robbles-- qsar-biodeg, nomao, pe3, bank- markmnan..ingg ada-agnostic, and hill-valley.
Table 9: Search spaces in SVM-A.
Steps
Hyperparameter
Range/Value
Prior
Kernel Degree
Radial
......, 5}
Uniform
1,2
Cost
[2-10, 210
Log-uniform
Kernel
Polynomial
[2 29)
Log-uniform
Table 10: Search spaces in SVM-B.
Steps
Hyperparameter
Range/Value
Prior Log-uniform
Cost
_2-5 5 25]
1, 2
1, 2
Degree
1,2 2
Kernel
(Polynomial, Linear, Radial)
Uniform
2
Cost
[ 10 21
Log-uniform
A.5 XGB-A & XGB-B
Base benehmark We use an open source implementation by the HPOlib authors following Perrone al. (0188). This implementation uses data from Kühn et al. (2018) and employs a random forest as surrogate model (Fggensperger et al., 2014). For our benchmarks, we randomly selected the ten tasks monks-problems-2, tic-tac-tos kc2, monks monks-problems-11 qsar-biodeg, nomao, pc3, bank-marketing, ada-agnostic, hill-vallcy
Table 11: Search spaces in XGB-A
Steps
Hyperparameter
Range/Valus
Prior
Colsample-by-tres
Colsample-by-leet
1
Minimum child weight
1
Maximum depth
6
1, 2
Booster
Tree
1.2
# Rounds
(..... 5, 5,000}
Uniform
2
Subsample
[0. 1]
Uniform
1,2
Eta
[2 [2-10 291
Log-uniform
1,2
Lambda
[2-10 210
Log-uniform
1,2 2
Alpha
[2 2- 210
Log-uniform
Colsample-by-tree
00 1
Uniform
Colsample-by-let
[0. 1
Uniform
Minimum child weight
[20_ 27
Log-uniform
Maximum depth
[I...., 15)
Uniform
15

---

Under review as a conference paper at ICLR 2021
Table 12: Search spaces in XGB-B
Steps
Hyperparameter
Range/Value
Prior
Colsample-by-tres
colsample-by-leve
1
Minimum child weight
Maximum depth
6
1,2
Booster
Linear, Tree
1.2
# Rounds
{1, 5, 5,000}
Uniform
1,2
Subsample
[0. 1]
Uniform
1.2
Eta
[2 [2-10_ 29
Log-uniform
1, 2
Lambda
[ 10 21
Log-uniform
1, 2
Alpha
[2 210
Log-uniform
Colsample-by-tree
1
2
Colsample-by-lewe
0.5
Minimum child weight
10
Maximum depth
10
16

---

Under review as a conference paper at ICLR 2021
B DETAILED SPEEDUPS
B. GP BASED EVALUATION
FCN-A
FCN-B
5
3
NAS-A
NAS-B
5
3
Speedup Over GP
Izz
TェX
SVM-A
SVM-B
6
4
2
0
XGB-A
XGB-B
4
2
0
10
20
40
10
20
40
GP Evaluations for Reference Objective [#]
Best First
Best First + Transfer GP
Transfer GP
Figure 5: Speedup compared to GP for best-first, best-first combined with transfer GP, and transfer GP across tasks for each of 8 benchmarks. The previous HPO has budget of 10 evaluations here.
17

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
NAS-A
NAS-B
Speedup Over
SVM-A
SVM-B
38
XGB-A
XGB-B
10
20
40
10
20
40
GP Evaluations for Reference Objective
[#]
Best First
Best First Transfer GP
Transfer GP
Figure 6: Speedup compared to GP for best-first, best-first combined with transfer GP, and transfer GP (from left to right in each violin triplet) across tasks for each of 8 benchmarks. The previous HPO has budget of 20 evaluations.
18

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
NAS-A
NAS-B
Speedup Over GP
SVM-A
SVM-B
XGB-A
XGB-B
10
20
40
10
20
40
GP Evaluations for Reference Objective
[#]
Best First
Best First Transfer GP
Transfer GP
Figure 7: Speedup compared to GP for best-first, best-first combined with transfer GP, and transfer GP (from left to right in each violin triplet) across tasks for each of 8 benchmarks. The previous HPO has budget of 40 evaluations.
19

---

Under review as a conference paper at ICLR 2021
B.2 TPE BASED EVALUATION
FCN-A
FCN-B
NAS-A
NAS-B
Speedup Over TPE
SVM-A
SVM-B
XGB-A
XGB-B
0
1
10
20
40
10
20
40
TPE Evaluations for Reference Objective [#]
Best First
Best First + Transfer TPE
Transfer TPE
Figure 8: Speedup compared to GP for best-first, best-first combined with transfer GP, and transfer GP across tasks for each of 8 benchmarks. The previous HPO has budget of 10 evaluations here.
20

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
13
12
10
0
NAS-A
NAS-B
13
Speedup Over TPE
ZIZ
SVM-A
SVM-B
XGB-A
XGB-B
10
20
40
10
20
40
TPE Evaluations for Reference Objective [#]
Best First
Best First + Transfer TPE
Transfer TPE
Figure 9: Speedup compared to GP for best-first, best-first combined with transfer GP, and transfer GP (from left to right in each violin triplet) across tasks for each of 8 benchmarks. The previous HPO has budget of 20 evaluations.
21

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
16
NAS-A
NAS-B
18
Speedup Over TPE
SVM-A
SVM-B
18
XGB-A
XGB-B
10
20
40
10
20
40
TPE Evaluations for Reference Objective [#]
Best First
Best First Transfer TPE
Transfer TPE
Figure 10: Speedup compared GP for best-first, best-first combined with transfer GP, and transfer GP (from left to right in each violin triplet) across tasks for each of 8 benchmarks. The previous HPO has budget of 40 evaluations.
22

---

Under review as a conference paper at ICLR 2021
C
FAILURE RATES
10 Previous Evaluations
20 Previous Evaluations
40 Previous Evaluations
0.8
Failed Runs (%)
0.6
0.4
0.2
0.0
10
20
40
10
20
40
10
20
40
GP Evaluations for Reference Objective [#]
Best First
GP
Transfer GP
Failure rates for best-first, GP, and transfer GP (from left to right in each violin triplet) across 8 benchmarks. The violins estimate densities of the task means. The horizontal line in each violin shows the mean across these task means. The plots from left to right utilize increasing budget for the pre-adjustment hyperparameter. The x-axis shows the budget for the GP reference.
Figure 11:
10 Previous Evaluations
20 Previous Evaluations
40 Previous Evaluations
Failed Runs [%)
10
20
40
10
20
40
10
20
40
TPE Evaluations for Reference Objective [#]
Best First
TPE
Transfer TPE
Failure rates for best-first, TPE, and transfer TPE (from left to right in each violin triplet) across 8 benchmarks. The violins estimate densities of the task means. The horizontal line in each violin shows the mean across these task means. The plots from left to right utilize increasing budget for the pre-ajjustment hyperparameter. The x-axis shows the budget for the GP reference.
Figure 12:
23

---

Under review as a conference paper at ICLR 2021
10 Previous Evaluations
20 Previous Evaluations
40 Previous Evaluations
80
Failed Runs [%)
60
40
20
0
10
20
40
10
20
40
10
20
40
TPE Evaluations for Reference Objective [#]
Drop Unimportant
Only Optimize New
Figure 13: Failure rates for the TPE based drop-unimportant and only-optimize-nett new across benchmarks. The violins estimate densities of the task means. The horizontal line in each violin shows the mean across these task means. The plots from left to right utilize increasing budget for the pre-adjustment hyperparameter. The x-axis shows the budget for the GP reference.
D OBJECTIVE IMPROVEMENTS
Here, we show the standardized improvement over the control algorithm. We compute the standard- ized mean improvement by averaging the final performance across repetitions, then subtract the mean performance of the control algorithm. and divide by the standard deviation of the control algorithm. This metric known as glass' delta and intuitively gives the improvement of an algorithm over control algorithm in the standard units of this control algorithm. As some benchmarks had a standard deviation of 0, we added small constant in those cases. We chose this constant according to the 0. --quunntlle of the observed values. For the plots we clip the improvement to |-10, oo), as for some plots there are extreme outliers. We provide results for two aggregation levels: the distribution over task standardized mean improvements, where compute the standardized mean improvement repetitions and show results on a per-benchmark level; and the distribution over benchmark means, i.e., the means across task standardized mean improvements for all tasks in given benchmark.
24

---

Under review as a conference paper at ICLR 2021
D.1 TRANSFER GP, BEST FIRST, AND TRANSFER GP COMBINED WITH BEST FIRST VS. GP
10 Previous Evaluations
20 Previous Evaluations
40 Previous Evaluations
Improvement Over GP [SDGp +&]
1.0
0.5
0.0
-0.5
10
20
40
10
20
40
10
20
40
Evaluations
[#]
Best First
Best First + Transfer GP
Transfer GP
Figure 14: Standardized objective improvements of best-first, best-first combined with transfer GP. and transfer GP (from left to right each violin triplet) over GP across benchmarks. The violins estimate densities of the benchmark means. The horizontal line in each violin shows the mean across these benchmark means. #Evaluations for the old HPO increases from left to right. In each plot, the evaluation budget increases.
25

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
0.0
2.5
5.0
-7.5
-10.0
NAS-A
NAS-B
0.0
2.5
55.0
Improvement Over
-75
GP [SDGp +6]
-10.0
SVM-A
SVM-B
0.0
-25
-5.0
7.5
-10.0
XGB-A
XGB-B
0.0
2.5
55.0
-7.5
-10.0
10
20
40
10
20
40
Evaluations
[#]
Best First
Best First Transfer GP
Transfer GP
Figure 15: Standardized objective improvements of best-first, best-first combined with transfer GP, and transfer GP (from left to right in each violin triplet) across tasks for each of benchmarks. The previous HPO has a budget of 10 evaluations.
26

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
0
-2
6
NAS-A
NAS-B
I
-2
-4
Improvement Over
GP [SDGr
SVM-A
SVM-B
0
-2
6
XGB-A
XGB-B
-2
4
6
10
20
40
10
20
40
Evaluations
[#]
Best First
Best First + Transfer GP
Transfer GP
Figure 16: Standardized objective improvements of best-first, best-first combined with transfer GP, and transfer GP (from left to right in each violin triplet) across tasks for each of 8 benchmarks. The previous HPO has budget of 20 evaluations.
27

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
i U
0
5
-10
NAS-A
NAS-B
-5
Improvement Over
GP [SDGP +
-10
SVM-A
SVM-B
0
-5
10
XGB-A
XGB-B
-5
-10
10
20
40
10
20
40
Evaluations
[#]
Best First
Best First Transfer GP
Transfer GP
Figure 17: Standardized objective improvements of best-first, best-first combined with transfer GP, and transfer GP (from left to right in each violin triplet) across tasks for each of 8 benchmarks. The previous HPO has budget of 40 evaluations.
28

---

Under review as a conference paper at ICLR 2021
D.2 TRANSFER TPE, BEST FIRST, AND TRANSFER TPE COMBINED WITH BEST FIRST VS. TPE
10 Previous Evaluations
20 Previous Evaluations
40 Previous Evaluations
Improvement Over TPE [SDap +-E]
0.5
0.0
-0.5
-1.0
10
20
20
40
20
40
Evaluations
[#]
Best First
Best First + Transfer TPE
Transfer TPE
Figure 18: Standardized objective improvements of best-first, best-first combined with transfer TPE, and transfer TPE (from left to right in each violin triplet) over GP across 8 benchmarks. The violins estimate densities of the benchmark means. The horizontal line in each violin shows the mean these benchmark means. #Evaluations for the old HPO increases from left to right. In each plot, the evaluation budget increases.
29

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
NAS-A
NAS-B
0
Improvement Over
TPE [SDop +E]
SVM-A
SVM-B
XGB-A
XGB-B
10
20
40
10
20
40
Evaluations
[#]
Best First
Best First + Transfer TPE
Transfer TPE
Figure 19: Standardized objective improvements of best-first, best-first combined with transfer TPE, and transfer TPE (from left to right in each violin triplet) across tasks for each of 8 benchmarks. The previous HPO has budget of 10 evaluations.
30

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
0.0
2.5
55.0
-7.
-10.0
NAS-A
NAS-B
0.0
22.
-5.0-
Improvement Over
-7.5
TPE [SDap +6]
-10.0
SVM-A
SVM-B
0.0
2.5
5.0
7.5
-00.
XGB-A
XGB-B
0.0
2.5
-5.0-
-7.5
-10.0
10
20
40
10
20
40
Evaluations
[#]
Best First
Best First + Transfer TPE
Transfer TPE
Figure 20: Standardized objective improvements of best-first, best-first combined with transfer TPE, and transfer TPE (from left to right in each violin triplet) across tasks for each of 8 benchmarks. The previous HPO has budget of 20 evaluations.
31

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
22
NAS-A
NAS-B
II
-2
-4
Improvement Over
TPE [SDor +E]
SVM-A
SVM-B
XGB-A
XGB-B
-2
-4
10
20
40
10
20
40
Evaluations
[#]
Best First
Best First + Transfer TPE
Transfer TPE
Figure 21: Standardized objective improvements of best-first, best-first combined with transfer TPE, and transfer TPE (from left to right in each violin triplet) across tasks for each of 8 benchmarks. The previous HPO has budget of 40 evaluations.
32

---

Under review as a conference paper at ICLR 2021
D.3 GP BASED ONLY OPTIMIZE NEW AND DROP UNIMPORTANT VS. GP
10 Previous Evaluations
20 Previous Evaluations
40 Previous Evaluations
0.0
Improvement Over GP [SDGp +-]
-2.
-5.0
-7.5
-10.0
10
20
40
10
20
40
10
20
40
Evaluations
[#]
Drop Unimportant
Only Optimize New
Figure 22: Standardized objective improvements of only-optimize-nent new and drop-unimportant over GP across benchmarks. The violins estimate densities of benchmark means. The horizontal line in each violin shows the mean across these benchmark means. #Evaluations for the old HPO increases from left right. each plot, the evaluation budget increases.
33

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
0.0
-2.
55.0
-7.5
-10.0
NAS-A
NAS-B
0.0
2.5
-5.0-
Improvement Over
77.5
GP [SDGp +6]
-10.0
SVM-A
SVM-B
0.0
-2.5
-5.0
-75
-00.0
XGB-A
XGB-B
0.0
22.5
55.0
-7.5
-10.0
10
20
40
10
20
40
Evaluations
[#]
Drop Unimportant
Only Optimize New
Figure 23: Standardized objective improvements of only-optimize-new and drop-unimportant across tasks for each of 8 benchmarks. The previous HPO has budget of 10 evaluations.
34

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
0.0
2.5
55.0
-7.
-10.0
NAS-A
NAS-B
0.0
225
-5.0-
Improvement Over
-7.5
GP [SDGp +6]
-10.0
SVM-A
SVM-B
0.0
2.5
5.0
7.5
-00.
XGB-A
XGB-B
0.0
2.5
5.0
-7.5
-10.0
10
20
40
10
20
40
Evaluations
[#]
Drop Unimportant
Only Optimize New
Figure 24: Standardized objective improvements of only-optimize-new and drop-unimportant across tasks for each of 8 benchmarks. The previous HPO has budget of 20 evaluations.
35

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
5
-10
NAS-A
NAS-B
R
0
-5-
Improvement Over
GP [SDGP +
-10
SVM-A
SVM-B
10
XGB-A
XGB-B
-5
-10
10
20
40
10
20
40
Evaluations
[#]
Drop Unimportant
Only Optimize New
Figure 25: Standardized objective improvements of only-optimize-new and drop-unimportant across lasks for each of 8 benchmarks. The previous HPO has a budget of 40 evaluations.
36

---

Under review as a conference paper at ICLR 2021
D.4 TPE BASED ONLY OPTIMIZE NEW AND DROP UNIMPORTANT VS. TPE
10 Previous Evaluations
20 Previous Evaluations
40 Previous Evaluations
Improvement Over
TPE [SDop +
-2
-4-
-6-
-8
10
20
40
10
20
40
10
20
40
Evaluations
[#]
Drop Unimportant
Only Optimize New
Figure 26: Standardized objective improvements of only-optimize-new and drop-unimportant over TPE across 8 benchmarks. The violins estimate densities of the benchmark means. The horizontal line each violin shows the mean across these benchmark means. #Evaluations for the old HPO increases from left to right. In each plot, the evaluation budget increases.
37

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
0.0
2.5
-5.0
77.
-10.0
NAS-A
NAS-B
0.0
2.5
55.0
Improvement Over
775
TPE [SDap +6]
-10.0
SVM-A
SVM-B
0.0
22.5
5.0
-7.5
-10.0
XGB-A
XGB-B
0.0
-2.
5.0
-7.
-10.0
10
20
40
10
20
40
Evaluations
[#]
Drop Unimportant
Only Optimize New
Figure 27: Standardized objective improvements of only-optimize-new and drop-unimportant across tasks for each of 8 benchmarks. The previous HPO has budget of 10 evaluations.
38

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
0.0
2.5
-5.0
77.5
-10.0
NAS-A
NAS-B
0.0
2.5
5.0
Improvement Over
-7.
TPE [SDap +6]
-10.0
SVM-A
SVM-B
0.0
22.5
5.0
-7.5
-10.0
XGB-A
XGB-B
0.0
2.5
5.0
775
-10.0
10
20
40
10
20
40
Evaluations
[#]
Drop Unimportant
Only Optimize New
Figure 28: Standardized objective improvements of only-optimize-new and drop-unimportant across tasks for each of 8 benchmarks. The previous HPO has budget of 20 evaluations.
39

---

Under review as a conference paper at ICLR 2021
FCN-A
FCN-B
0.0
225
-5.0
77.
-00.0
NAS-A
NAS-B
0.0
2.5
5.00-
Improvement Over
-7.
TPE [SDap +6]
-10.0
SVM-A
SVM-B
0.0
2.5
-5.0
77.5
-10.0
XGB-A
XGB-B
0.0
-2.5
5.0
775
-10.0
10
20
40
10
20
40
Evaluations
[#]
Drop Unimportant
Only Optimize New
Figure 29: Standardized objective improvements of only-optimize-new and drop-unimportant across tasks for each of 8 benchmarks. The previous HPO has budget of 40 evaluations.
40

---

Under review as a conference paper at ICLR 2021
E CONTROL STUDY: GP AND TPE FOR DIFFERENT RANDOM SEED RANGES
As sanity check, and to gauge the influece of random seeds, we compare GP (and TPE) to itself with different random seed ranges (GP) and GP2). We observe little differences in GP2 and GP2 (likewise for TPE) compared to the differences between GP and the transfer approaches (Figure 30 and Figure
10 Previous Evaluations
20 Previous Evaluations
40 Previous Evaluations
Speedup Over GP
1.2
1.0
0.8
0.6
0.4
0.2
0.0
10
20
40
10
20
40
10
20
40
GP Evaluations for Reference Objective [#]
Figure 30: Speedup of GP1 over GP2 across 8 benchmarks. The violins estimate densities of the benchmark geometric means. The horizontal line in each violin shows the geometric mean across these benchmark means. #Evaluations for the old HPO increases from left to right. The x-axis shows budget for the GP reference.
10 Previous Evaluations
20 Previous Evaluations
40 Previous Evaluations
1.6
Speedup Over TPEa
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0.0
10
20
40
10
20
10
20
40
TPE Evaluations for Reference Objective [#]
Figure 31: Speedup of TPE; over TPEg across benchmarks. The violins estimate densities of the benchmark geometric means. The horizontal line in each violin shows the geometric mean across these benchmark means. #Evaluations for the old HPO increases from left right. The x-axis shows the budget for the TPE reference.
41

---

Under review as a conference paper at ICLR 2021
F
CONTROL STUDY: RELIABILITY OF GP AND TPE
As sanity check for the reliability of GP and TPE, we compare them to random search.
10 Previous Evaluations
20 Previous Evaluations
40 Previous Evaluations
Speedup Over Random
3
2-
10
20
40
10
20
40
10
20
40
GP Evaluations for Reference Objective [#]
GP
TPE
Figure 32: Speedup of GP and TPE over random search across 8 benchmarks. The violins estimate densities of the benchmark means. The horizontal line in each violin shows the geometric mean across these benchmark means. #Evaluations for the old HPO increases from left to right. The x-axis shows the budget for the GP reference.
42


Reviews:
[{"review_id": "x4ucCPwLms", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is motivated by the situation where a machine learning algorithm has development adjustment and we would like to reuse the tuning results of previous hyperparameter optimization. The paper calls it HT-AA problem, which certainly is an interesting problem given software can get updated often. The paper proposes four simple baseline algorithms for the HT-AA problem. \n\nFor the empirical study, a set of eight benchmarks for basic HT-AA problem are presented. The experiment results show the transfer TPE (T2PE) and best-first strategy produce good speed up. To reach given objective values, T2PE can be 1.0–1.7x faster than TPE, and best-first 1.2–2.6x faster comparing with old HPO. \n\nThe pros of the paper include:\n1. Although the topic of the paper is not one of the most popular, some readers might find it interesting and can be benefited from it.\n2. The proposed methods are reasonable and acceptable.\n3. Numerical results show some of the proposed methods can help to speed up reoptimizing hyperparameter.\n\nThe cons include:\n1. Need explain when the only-optimize-new and drop-unimportant methods can be useful. If not useful as the experiments demonstrate, why propose them?\n2. Look like TPE is the method that show speedup for the benchmarks. How reliable the method is? Is the saving justifying use an extra tuning tool?\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "o8JKdk9Y_U", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is addressing a problem which is quite relevant in practice, namely how to warmstart HP optimization after small changes have been done to the ML model. Such changes may modify the HP search space, both by adding/removing HPs, or by changing their value ranges. The paper is clearly written. It introduces 3 potential baselines, as well as a simple transfer strategy. All work is based on TPE, which is frequently used, but not SotA for HPO.\n\nThe paper does not elaborate on the motivations of these \"developer changes\". One could suspect many are attempts to modify/improve the HPO process itself, over the *same* model. And this is not really new, there is lots of prior work to help shaping search spaces, both by quantifying HP relevance or by learning search ranges. Say, a developer modifies the value range of an HP. What other motivation would there be than mistrust in the previous range, but no change of algorithm. Same for adding/removing an HP, which normally just means going from fixed default to HPO or back. In fact, the 8 benchmarks are all of that sort. My feeling is that by viewing the problem in this way (namely, just HP search space optimization), there is suddenly a lot more related work not taken into account here. More difficult problems, such as learning ensembles from a range of models, and then adding/removing model types, are not tackled here. These would call for more difficult transfer strategies.\n\nThis paper does not really propose new methodology, except maybe T2PE, which is a pretty basic heuristic. There is a lot of prior work on transfer HPO, some of which could cewrtainly be adopted. Given that the paper is mainly empirical, one would expect a more thorough and wider evaluation. On the positive side, the paper introduces 8 new benchmarks, even though they are pretty simple setups. Their empirical evaluations are a little thin. Only the best-first baseline works well, results for the others are not shown. It should be noted that best-first is standard in HPO practice, this is the first thing one does for transfer. Their T2PE essentially works just as well, and a combination of the two works slightly better. While the paper categorizes types of modification, the empirical evaluation does not differentiate among them anymore. Also, the restriction to TPE is questionable. Why not also use GP-BO? All baselines would work just the same.\n\nMy main recommendation for this work would be to be clear about the modification for such limited developer changes. If this is just about the developer trying to twist HPO in itself, this work would have to compare against previous work for optimizing search spaces. Otherwise, please address more complex scenarios, such as ensemble learning, where HPO transfer becomes really difficult.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MR_WqhSu4k", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Major weaknesses of the paper:\n- My understanding is that these are surrogate models that are meant to simulate a real-world task. However there is no description as to how these surrogates were created or trained, nor how their fidelity to the original task was vetted.\n- The most simple and naive algorithm seems to provide similar speed-ups to the much more complicated proposed T2PE; especially when considering the much larger improvement of the naive method (see Fig. 11), none of the other proposed methods seem justified to me.\n- Furthermore, I don't consider this naive approach (Best First) as being an HPO approach that leverages transfer, since it does exactly what anyone would do when faced with a slightly altered set of hyperparameters.\n- This last point suggests that at least one of the following must be true:\n  * non-trivial transfer is not as important as intuition would lead us to think;\n  * this benchmark suite does not provide a good testbed for assessing an HPO method's ability to transfer; or\n  * none of the non-trivial proposed algorithms do a good job transferring and can therefore not argue against the previous point.\n  \nGiven this important contradiction, I must recommend a rejection. My recommendations would be to:\n- focus on creating a good benchmark suite (perhaps focus on a single or two domains as introduce many variants, instead of four domains with only two variants);\n- focus on vetting the surrogates in terms of their fidelity to the task they are meant to simulate; and\n- focus on demonstrating that accounting for the slight modifications in the subsequent HPO does indeed provide a benefit over the naive thing to do.\nFor the record, I don't think this is an easy task.\n\nMinor points:\n- Justify geometric mean. I'm not saying it's the wrong way to compare these, I just think it requires at least a sentence of justification.\n- Same for the violin plots. For such simple plots, simple boxes and whiskers, with perhaps data points to show the spread of measurements across seeds, would do just fine.\n- Figure 4, and indeed any mention of the two methods therein, can be entirely removed from the paper; other than to perhaps mention that they were tried and failed---results in the appendix.\n- A much more interesting replacement for that figure would be Figure 11.\n- Not sure what is the point of comparing random search to TPE in the appendix unless this means Best-first then Random-search/TPE? If the latter is true, please clarify.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DqhIHj9Qt9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new framework for hyperparameter optimization and transfer across incremental modifications to a given algorithm and its search space, a process called developer adjustments in the paper. The authors then propose a few strategies to transfer knowledge from previous HPO runs and evaluate them on a series of simulated benchmarks. Results show the value added by transferring information from previous runs, as well as the surprising efficiency of simply reusing the best found hyperparameters from the previous run. \n\nStrong points:\n\n- The framework is simple and clearly introduced. \n- Extensive experiments help bring to light the advantage of transferring across adjustments.\n- The paper is very well written. \n\nWeak points:\n\n- Not enough details on benchmarks, more on this below\n- The use of simulated benchmarks with surrogate models introduces noise in the evaluation\n- Comparisons with more baselines would be beneficial. RF and/or GP-based HPO methods are extremely popular and would have been easy to integrate with the best-first baseline. \n\nRecommendation:\n\nThe contributions are simple and incremental, and clearly rooted in machine learning engineering, however I still think they could be beneficial as a whole to the community given the extensive experiments realized. I have some issues with experiments, lack of details and baselines, but those issues are mostly fixable. I'll give the paper a weak accept for now.\n\nExtra comments:\n\nYou do not specify which benchmarks are based on lookup tables and which ones are based on surrogate models. From looking at the search spaces, I would assume that the SVM and XGB benchmarks are modeled via surrogate benchmarks and the FCN and NAS benchmarks are lookup tables, but this should be explicited in the paper (or appendix). Parameters used for the benchmark surrogate model should also be given (if defaults of Eggensperger are used, simply mention this). It is also not clear what underlying datasets are used, this bears some importance and should be mentioned, even if only in the Appendix.\n\nOn surrogate model benchmarks: It can be seen in (Eggensperger et al. 2015., Figure 2) that ordering of methods can shift due to noise in the surrogate model (a random forest?). This is likely going to have a bigger impact when trying to measure the speedup, which is measured when a method reaches a certain threshold of performance. This threshold is likely to be met during the convergence phase of algorithms, and this phase appears noisier  (i.e. looking at how the phases of transition differ between the true benchmark and the RF surrogate benchmark differ in Eggensperger et al. 2015). Have you given this any thought? Have you compared experiments with a few runs on a real benchmark?\n\nThe method you end up recommending only has its detailed performance shown in the appendix. This feels counterintuitive to me. This result should be featured in the paper itself. This is perhaps due to the used of those split violin plots, which force you to display only two methods per plot. Maybe you should display a group of X single-sided violin plots where X is the number of methods you are trying to compare.\n\nI think it is misleading to portray everything in terms of speedup or improvement over the \"TPE solution with X iterations\". A more strictly meaningful metric here is accuracy (assuming there is only one dataset per benchmark). Assuming the performance to beat by original TPE was an 11% error rate, there is a big difference between a method which was able to achieve a 10% error rate and a method which was able to achieve a 5% error rate, yet both will be assessed by how quickly they achieved x < 10% error rate. I can't seem to find such figures in the appendices.\n\nTypos:\n\n- Section 3.1 page 3, argmax g(x) / b(x) << you mean g(x) / l(x)?\n- appendix G, you wrote TPE2 instead of T2PE", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has been actively discussed, both during and after the rebuttal phase. I enjoyed, and I am thankful for, the active communication that took place between the authors and the reviewers.

On the one hand, the reviewers agreed on several pros of the paper, e.g.,
* Clear, well presented manuscript
* The presentation of practically-relevant setting
* A work that fosters reproducible research (both BO data and algorithms are made available)
* Careful experiments

On the other hand, several important weaknesses were also outlined, e.g.,
* _Novelty_: While the authors claim they “introduce a practically relevant and fundamentally novel research problem”, existing commercial HPO solutions already mention, and propose solutions for, the very same problem, e.g., [AWS](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-warm-start.html) (section “Types of Warm Start Tuning Jobs”) and [Google cloud](https://cloud.google.com/blog/products/gcp/hyperparameter-tuning-on-google-cloud-platform-is-now-faster-and-smarter) (section “Learning from previous trials”). The reviewers all agreed on the fact that this down-weights the novelty aspect (claimed many times in the rebuttal and the manuscript): The paper formalizes an already existing framework rather than introducing it.
* In the light of the weakened "novelty" contribution (see above), the reviewers regretted the absence of a novel transfer method _tailored to HT-AA_, which would have certainly strengthened the submission.
* _“Dynamic range” of the benchmark_: It is difficult to evaluate the capacity of the benchmark to discriminate between different approaches (e.g., see new Fig. 3 showing the violin plot with all three methods for transfer, as suggested by Reviewer 1: the improvements over "best first" seem marginal at best). To better understand the benchmark, it would be nice to illustrate its “dynamic range” by exhibiting a more powerful method that would substantially improve over “best first”.

As illustrated by its scores, the paper is extremely borderline. Given the mixed perspectives of pros and cons, we decided with the reviewers to recommend the rejection of the paper. 
Decision: Reject