Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: You mention that sequential optimization methods are not guaranteed to be consistent. Couldn't they be made trivially consistent in practice by choosing a random point x% of the time? I believe that SMAC already does this.

With respect to Figure 2, Hyperband seems to involve sequentially running out each bracket in turn (unless I've missed something). Wouldn't that mean that it would take 5x as long as each bracket in Figure 2? How is the wall time the same as an individual bracket?
review_title: Consistency and a question about Figure 2

final_score: rating: null
review_content: I am surprised by the baselines you used in your experiments. Work on what you call "configuration evaluation" goes back (at least) 23 years to Maron & Moore, NIPS 1993: "Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation" (https://papers.nips.cc/paper/841-hoeffding-races-accelerating-model-selection-search-for-classification-and-function-approximation), so it is a well known problem. Since this is the problem you tackle, I would've expected a comparison to established baselines for that problem beyond your own work on successive halving. The comparison to Bayesian optimization without "configuration evaluation" seems like a red herring to me, particularly since there is work on Freeze-Thaw Bayesian optimization that includes this component.

I think a much more meaningful comparison would be against Krueger et al (JMLR 2015), who -- like your work -- also focused on "configuration evaluation", also followed a racing approach that starts out with many configurations & drops dominated configurations over time, and also derive theoretical guarantees. Your discussion states that their work relies on a used-defined "safety zone", but looking at their paper they are very clear about default choices for all parameters. Did you actually try to experimentally compare against their method (in this or in previous work)?
review_title: Lack of baselines for "configuration evaluation"

final_score: rating: null
review_content: Given the authors' discussion of CVST, I had expected Hyperband to do much better, but their experiment does not show that: 
- Despite Hyperband being an anytime algorithm, the authors ran it much shorter than CVST and got consistently worse mean results. Maybe not far worse, but no characteristics of the task are presented, so one might already get results within the error bars of CVST by picking a random configuration at no cost at all ... Why not run Hyperband as long as CVST and then compare apples with apples?

- Also, for this experiment, the authors ran Hyperband with a different \eta than for all other experiments. This begs the question: How much do you need to tune the method to work? What would be the result of using the same \eta=4 as elsewhere?

review_title: Inconclusive comparison to CVST 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This was an interesting paper. The algorithm seems clear, the problem well-recognized, and the results are both strong and plausible.

Approaches to hyperparameter optimization based on SMBO have struggled to make good use of convergence during training, and this paper presents a fresh look at a non-SMBO alternative (at least I thought it did, until one of the other reviewers pointed out how much overlap there is with the previously published successive halving algorithm - too bad!). Still, I'm excited to try it. I'm cautiously optimistic that this simple alternative to SMBO may be the first advance to model search for the skeptical practitioner since the case for random search > grid search (http://www.jmlr.org/papers/v13/bergstra12a.html, which this paper should probably cite in connection with their random search baseline.)

I would suggest that the authors remove the (incorrect?) claim that this algorithm is "embarrassingly parallel" as it seems that there are number of synchronization barriers at which state must be shared in order to make the go-no-go decisions on whatever training runs are still in progress.  As the authors themselves point out as future work, there are interesting questions around how to adapt this algorithm to make optimal use of a cluster (I'm optimistic that it should carry over, but it's not trivial).

For future work, the authors might be interested in Hutter et al's work on Bayesian Optimization With Censored Response Data (https://arxiv.org/abs/1310.1947) for some ideas about how to use the dis-continued runs.
review_title: interesting extension to successive halving, still looking forward to the parallel asynchronous version

final_score: rating: 7: Good paper, accept
review_content: This paper presents Hyperband, a method for hyperparameter optimization where the model is trained by gradient descent or some other iterative scheme. The paper builds on the successive halving + random search approach of Jamieson and Talwalkar and addresses the tradeoff between training fewer models for a longer amount of time, or many models for a shorter amount of time. Effectively, the idea is to perform multiple rounds of successive halving, starting from the most exploratory setting, and then in each round exponentially decreasing the number of experiments, but granting them exponentially more resources. In contrast to other recent papers on this topic, the approach here does not rely on any specific model of the underlying learning curves and therefore makes fewer assumptions about the nature of the model. The results seem to show that this approach can be highly effective, often providing several factors of speedup over sequential approaches.

Overall I think this paper is a good contribution to the hyperparameter optimization literature. It’s relatively simple to implement, and seems to be quite effective for many problems. It seems like a natural extension of the random search methodology to the case of early stopping. To me, it seems like Hyperband would be most useful on problems where a) random search itself is expected to perform well and b) the computational budget is sufficiently constrained so that squeezing out the absolute best performance is not feasible and near-optimal performance is sufficient. I would personally like to see the plots in Figure 3 run out far enough that the other methods have had time to converge in order to see what this gap between optimal and near-optimal really is (if there is one).

I’m not sure I agree with the use of random2x as a baseline. I can see why it’s a useful comparison because it demonstrates the benefit of parallelism over sequential methods, but virtually all of these other methods also have parallel extensions. I think if random2x is shown, then I would also like to see SMAC2x, Spearmint2x, TPE2x, etc. I also think it would be worth seeing 3x, 10x, and so forth and how Hyperband fares against these baselines.

review_title: Good extension of successive halving and random search

final_score: rating: 7: Good paper, accept
review_content: This paper discusses Hyperband, an extension of successive halving by Jamieson & Talwalkar (AISTATS 2016). Successive halving is a very nice algorithm that starts evaluating many configurations and repeatedly cuts off the current worst half to explore many configuration for a limited budget.

Having read the paper for the question period and just rereading it again, I am now not entirely sure what its contribution is meant to be: the only improvement of Hyperband vs. successive halving is in the theoretical worst case bounds (not more than 5x worse than random search), but you can (a) trivially obtain that bound by using a fifth of your time for running random configurations to completion and (b) the theoretical analysis to show this is said to be beyond the scope of the paper. That makes me wonder whether the theoretical results are the contribution of this paper, or whether they are the subject of a different paper and the current paper is mostly an empirical study of the method?
I hope to get a response by the authors and see this made clearer in an updated version of the paper.

In terms of experiments, the paper fails to show a case where Hyperband actually performs better than the authors' previous algorithm successive halving with its most agressive setting of bracket b=4. Literally, in every figure, bracket b=4 is at least as good (and sometimes substantially better) than Hyperband. That makes me think that in practice I would prefer successive halving with b=4 over Hyperband. (And if I really want Hyperband's guarantee of not being more than 5x worse than random search I can run random search on a fifth of my machines.) 
The experiments also compare to some Bayesian optimization methods, but not to the most relevant very closely related Multi-Task Bayesian Optimization methods that have been dominating effective methods for deep learning in that area in the last 3 years: "Multi-Task Bayesian Optimization" by Swersky, Snoek, and Adams (2013) already showed 5x speedups for deep learning by starting with smaller datasets, and there have been several follow-up papers showing even larger speedups. 

Given that this prominent work on multitask Bayesian optimization exists, I also think the introduction, which sells Hyperband as a very new approach to hyperparameter optimization is misleading. I would've much preferred a more down-to-earth pitch that says "configuration evaluation" has been becoming a very important feature in hyperparameter optimization, including Bayesian optimization, that sometimes yields very large speedups (this can be quantified by examples from existing papers) and this paper adds some much-needed theoretical understanding to this and demonstrates how important configuration evaluation is even in the simplest case of being used with random search. I think this could be done easily and locally by adding a paragraph to the intro.

As another point regarding novelty, I think the authors should make clear that approaches for adaptively deciding how many resources to use for which evaluation have been studied for (at least) 23 years in the ML community -- see Maron & Moore, NIPS 1993: "Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation" (https://papers.nips.cc/paper/841-hoeffding-races-accelerating-model-selection-search-for-classification-and-function-approximation). Again, this could be done by a paragraph in the intro. 

Overall, I think for this paper having the related work section at the end leads to many concepts appearing to be new in the paper that turn out not to be new in the end, which is a bit of a let-down. I encourage the authors to prominently discuss related work, including the recent trends in Bayesian optimization towards configuration evaluation, in the beginning, and then clearly state the contribution of this paper by positioning it in the context of that related work and saying what exactly is new. (I think the answer is "very simple method", "great empirical results for several deep learning tasks" and "much-needed new theoretical results", which is a very nice contribution.) I'm giving an accepting score trusting that the authors will follow this suggestion.


I have some responses to some of the author responses:

1) "In response to your question, we ran an experiment modeled after the empirical studies in Krueger et al tuning 2 hyperparameters of a kernel SVM to compare CVST (Krueger et al 2015) and Hyperband.  Hyperband is 3-4x faster than CVST on this experiment and the two achieve similar test performance.  Notably, CVST was only 50% faster than standard holdout.   For the experiments in our paper, we excluded CVST due to the aforementioned theoretical differences and because CVST is not an anytime algorithm, but as we perform more experiments, we will update the draft to reflect this comparison."

Great, I am looking forward to seeing the details on these experiments before the decision phase.

2) "Hyperband makes no assumptions on the shape or rate of convergence of the validation error, just that it eventually converges."

It's only the worst-case analysis that makes no assumption, but of course one would not be happy with that worst-case performance of being 5x worse than random search. (The 5x is what the authors call "modestly worse, by a log factor"; it's the logarithm of the dataset size or of the number of epochs, both of which tend to be large numbers). I think this number of 5x should be stated explicitly somewhere for the authors choice of Hyperband parameters. (E.g., at the beginning of the experiments, when Hyperband's parameters are stated.)

3) "Like random search, it is also embarrassingly parallel."

I think this is not quite correct. Let's say I want to tune hyperparameters on ImageNet and each hyperparameter evaluation takes 1 week, but I have 100 GPUs, then random search will give a decent solution (the best of 100 random configurations) after 1 week. However, Hyperband will require 5 weeks before it will give any solution. Again, the modest log factor is a factor of 5. To me, "embarassingly parallel" would mean making great predictions after a week if you throw enough resources at it.
review_title: A nice paper, just needs to relate to the existing literature better

final_score: rating: null
review_content: I can appreciate how this racing-style algorithm is faster than random search when early results (e.g. validation curves) give meaningful indication of future performance (e.g. bounds).

I'm not as clear on how hyperband deals with noisy or misleading early indicators, and nothing comes for free so it's helpful to understand what makes hyperband work. Suppose for example that the best configurations are in some region of the configuration space that accounts for 1% of the sampling distribution. For whatever reason, the validation performance of these configurations is terrible for the first few iterations (much worse than the average over the whole space), and only later on do these late-bloomers emerge as the best models. Is it true that in such a scenario random search would be faster than hyperband (at least for some values of S, R, nu, etc.) because random would spend at least a full 1% of time on these configurations, while hyperband would be distracted by promising leads elsewhere in the space?

Edit: training algorithms that are pipelines of trained components might exhibit this kind of behavior, where a lot of up-front computational investment is required before any kind of validation curve starts to go down.
review_title: intuition around worst-case

Meta-review: This paper presents a simple strategy for hyperparameter optimization that gives strong empirical results. The reviewers all agreed that the paper should be accepted and that it would be interesting and useful to the ICLR community. However, they did have strong reservations about the claims made in the paper and one reviewer stated that their accept decision was conditional on a better treatment of the related literature. 
 
 While it is natural for authors to argue for the advantages of their approach over existing methods, some of the claims made are unfounded. For example, the claim that the proposed method is guaranteed to converge while "methods that rely on these heuristics are not endowed with any
 theoretical consistency guarantees" is weak. Any optimization method can trivially add this guarantee by adopting a simple strategy of adding a random experiment 1/n of the time (in fact, SMAC does this I believe). This claim is true of random search compared to gradient based optimization on non-convex functions as well, yet no one optimizes their deep nets via random search. Also, the authors claim to compare to state-of-the-art in hyperparameter optimization but all the comparisons are to algorithms published in either 2011 or 2012. Four years of continued research on the subject are ignored (e.g. methods in Bayesian optimization for hyperparameter tuning have evolved considerably since 2012 - see e.g. the work of Miguel Hern‡ndez Lobato, Matthew Hoffman, Nando de Freitas, Ziyu Wang, etc.). It's understood that it is difficult to compare to the latest literature (and the authors state that they had trouble running recent algorithms), but one can't claim to compare to state-of-the-art without actually comparing to state-of-the-art.
 
 Please address the reviewers concerns and tone down the claims of the paper.
Decision: Accept (Poster)