Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper is motivated by the situation where a machine learning algorithm has development adjustment and we would like to reuse the tuning results of previous hyperparameter optimization. The paper calls it HT-AA problem, which certainly is an interesting problem given software can get updated often. The paper proposes four simple baseline algorithms for the HT-AA problem. 

For the empirical study, a set of eight benchmarks for basic HT-AA problem are presented. The experiment results show the transfer TPE (T2PE) and best-first strategy produce good speed up. To reach given objective values, T2PE can be 1.0–1.7x faster than TPE, and best-first 1.2–2.6x faster comparing with old HPO. 

The pros of the paper include:
1. Although the topic of the paper is not one of the most popular, some readers might find it interesting and can be benefited from it.
2. The proposed methods are reasonable and acceptable.
3. Numerical results show some of the proposed methods can help to speed up reoptimizing hyperparameter.

The cons include:
1. Need explain when the only-optimize-new and drop-unimportant methods can be useful. If not useful as the experiments demonstrate, why propose them?
2. Look like TPE is the method that show speedup for the benchmarks. How reliable the method is? Is the saving justifying use an extra tuning tool?



review_title: A framework for hyperparemeter transfer when ML algorithm changeshm 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a new framework for hyperparameter optimization and transfer across incremental modifications to a given algorithm and its search space, a process called developer adjustments in the paper. The authors then propose a few strategies to transfer knowledge from previous HPO runs and evaluate them on a series of simulated benchmarks. Results show the value added by transferring information from previous runs, as well as the surprising efficiency of simply reusing the best found hyperparameters from the previous run. 

Strong points:

- The framework is simple and clearly introduced. 
- Extensive experiments help bring to light the advantage of transferring across adjustments.
- The paper is very well written. 

Weak points:

- Not enough details on benchmarks, more on this below
- The use of simulated benchmarks with surrogate models introduces noise in the evaluation
- Comparisons with more baselines would be beneficial. RF and/or GP-based HPO methods are extremely popular and would have been easy to integrate with the best-first baseline. 

Recommendation:

The contributions are simple and incremental, and clearly rooted in machine learning engineering, however I still think they could be beneficial as a whole to the community given the extensive experiments realized. I have some issues with experiments, lack of details and baselines, but those issues are mostly fixable. I'll give the paper a weak accept for now.

Extra comments:

You do not specify which benchmarks are based on lookup tables and which ones are based on surrogate models. From looking at the search spaces, I would assume that the SVM and XGB benchmarks are modeled via surrogate benchmarks and the FCN and NAS benchmarks are lookup tables, but this should be explicited in the paper (or appendix). Parameters used for the benchmark surrogate model should also be given (if defaults of Eggensperger are used, simply mention this). It is also not clear what underlying datasets are used, this bears some importance and should be mentioned, even if only in the Appendix.

On surrogate model benchmarks: It can be seen in (Eggensperger et al. 2015., Figure 2) that ordering of methods can shift due to noise in the surrogate model (a random forest?). This is likely going to have a bigger impact when trying to measure the speedup, which is measured when a method reaches a certain threshold of performance. This threshold is likely to be met during the convergence phase of algorithms, and this phase appears noisier  (i.e. looking at how the phases of transition differ between the true benchmark and the RF surrogate benchmark differ in Eggensperger et al. 2015). Have you given this any thought? Have you compared experiments with a few runs on a real benchmark?

The method you end up recommending only has its detailed performance shown in the appendix. This feels counterintuitive to me. This result should be featured in the paper itself. This is perhaps due to the used of those split violin plots, which force you to display only two methods per plot. Maybe you should display a group of X single-sided violin plots where X is the number of methods you are trying to compare.

I think it is misleading to portray everything in terms of speedup or improvement over the "TPE solution with X iterations". A more strictly meaningful metric here is accuracy (assuming there is only one dataset per benchmark). Assuming the performance to beat by original TPE was an 11% error rate, there is a big difference between a method which was able to achieve a 10% error rate and a method which was able to achieve a 5% error rate, yet both will be assessed by how quickly they achieved x < 10% error rate. I can't seem to find such figures in the appendices.

Typos:

- Section 3.1 page 3, argmax g(x) / b(x) << you mean g(x) / l(x)?
- appendix G, you wrote TPE2 instead of T2PE
review_title: Valuable framework, precisions required

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper is addressing a problem which is quite relevant in practice, namely how to warmstart HP optimization after small changes have been done to the ML model. Such changes may modify the HP search space, both by adding/removing HPs, or by changing their value ranges. The paper is clearly written. It introduces 3 potential baselines, as well as a simple transfer strategy. All work is based on TPE, which is frequently used, but not SotA for HPO.

The paper does not elaborate on the motivations of these "developer changes". One could suspect many are attempts to modify/improve the HPO process itself, over the *same* model. And this is not really new, there is lots of prior work to help shaping search spaces, both by quantifying HP relevance or by learning search ranges. Say, a developer modifies the value range of an HP. What other motivation would there be than mistrust in the previous range, but no change of algorithm. Same for adding/removing an HP, which normally just means going from fixed default to HPO or back. In fact, the 8 benchmarks are all of that sort. My feeling is that by viewing the problem in this way (namely, just HP search space optimization), there is suddenly a lot more related work not taken into account here. More difficult problems, such as learning ensembles from a range of models, and then adding/removing model types, are not tackled here. These would call for more difficult transfer strategies.

This paper does not really propose new methodology, except maybe T2PE, which is a pretty basic heuristic. There is a lot of prior work on transfer HPO, some of which could cewrtainly be adopted. Given that the paper is mainly empirical, one would expect a more thorough and wider evaluation. On the positive side, the paper introduces 8 new benchmarks, even though they are pretty simple setups. Their empirical evaluations are a little thin. Only the best-first baseline works well, results for the others are not shown. It should be noted that best-first is standard in HPO practice, this is the first thing one does for transfer. Their T2PE essentially works just as well, and a combination of the two works slightly better. While the paper categorizes types of modification, the empirical evaluation does not differentiate among them anymore. Also, the restriction to TPE is questionable. Why not also use GP-BO? All baselines would work just the same.

My main recommendation for this work would be to be clear about the modification for such limited developer changes. If this is just about the developer trying to twist HPO in itself, this work would have to compare against previous work for optimizing search spaces. Otherwise, please address more complex scenarios, such as ensemble learning, where HPO transfer becomes really difficult.

review_title: Considers problem of warmstarting hyperparameter optimization after small changes to training algorithm and/or HP search space.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Major weaknesses of the paper:
- My understanding is that these are surrogate models that are meant to simulate a real-world task. However there is no description as to how these surrogates were created or trained, nor how their fidelity to the original task was vetted.
- The most simple and naive algorithm seems to provide similar speed-ups to the much more complicated proposed T2PE; especially when considering the much larger improvement of the naive method (see Fig. 11), none of the other proposed methods seem justified to me.
- Furthermore, I don't consider this naive approach (Best First) as being an HPO approach that leverages transfer, since it does exactly what anyone would do when faced with a slightly altered set of hyperparameters.
- This last point suggests that at least one of the following must be true:
  * non-trivial transfer is not as important as intuition would lead us to think;
  * this benchmark suite does not provide a good testbed for assessing an HPO method's ability to transfer; or
  * none of the non-trivial proposed algorithms do a good job transferring and can therefore not argue against the previous point.
  
Given this important contradiction, I must recommend a rejection. My recommendations would be to:
- focus on creating a good benchmark suite (perhaps focus on a single or two domains as introduce many variants, instead of four domains with only two variants);
- focus on vetting the surrogates in terms of their fidelity to the task they are meant to simulate; and
- focus on demonstrating that accounting for the slight modifications in the subsequent HPO does indeed provide a benefit over the naive thing to do.
For the record, I don't think this is an easy task.

Minor points:
- Justify geometric mean. I'm not saying it's the wrong way to compare these, I just think it requires at least a sentence of justification.
- Same for the violin plots. For such simple plots, simple boxes and whiskers, with perhaps data points to show the spread of measurements across seeds, would do just fine.
- Figure 4, and indeed any mention of the two methods therein, can be entirely removed from the paper; other than to perhaps mention that they were tried and failed---results in the appendix.
- A much more interesting replacement for that figure would be Figure 11.
- Not sure what is the point of comparing random search to TPE in the appendix unless this means Best-first then Random-search/TPE? If the latter is true, please clarify.
review_title: This manuscript introduces a benchmark suite of hyperparameter optimization tasks, that simulate slight developer modifications, with the goal of optimizing a slightly modified algorithm given knowledge of its previous form. The paper further studies the performance of some deliberately naive approaches as a performance benchmark to accompany the suite.

Meta-review: The paper has been actively discussed, both during and after the rebuttal phase. I enjoyed, and I am thankful for, the active communication that took place between the authors and the reviewers.

On the one hand, the reviewers agreed on several pros of the paper, e.g.,
* Clear, well presented manuscript
* The presentation of practically-relevant setting
* A work that fosters reproducible research (both BO data and algorithms are made available)
* Careful experiments

On the other hand, several important weaknesses were also outlined, e.g.,
* _Novelty_: While the authors claim they “introduce a practically relevant and fundamentally novel research problem”, existing commercial HPO solutions already mention, and propose solutions for, the very same problem, e.g., [AWS](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-warm-start.html) (section “Types of Warm Start Tuning Jobs”) and [Google cloud](https://cloud.google.com/blog/products/gcp/hyperparameter-tuning-on-google-cloud-platform-is-now-faster-and-smarter) (section “Learning from previous trials”). The reviewers all agreed on the fact that this down-weights the novelty aspect (claimed many times in the rebuttal and the manuscript): The paper formalizes an already existing framework rather than introducing it.
* In the light of the weakened "novelty" contribution (see above), the reviewers regretted the absence of a novel transfer method _tailored to HT-AA_, which would have certainly strengthened the submission.
* _“Dynamic range” of the benchmark_: It is difficult to evaluate the capacity of the benchmark to discriminate between different approaches (e.g., see new Fig. 3 showing the violin plot with all three methods for transfer, as suggested by Reviewer 1: the improvements over "best first" seem marginal at best). To better understand the benchmark, it would be nice to illustrate its “dynamic range” by exhibiting a more powerful method that would substantially improve over “best first”.

As illustrated by its scores, the paper is extremely borderline. Given the mixed perspectives of pros and cons, we decided with the reviewers to recommend the rejection of the paper. 
Decision: Reject