Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents an extension to the commonly used Gaussian Process based HPO by incorporating the learning curve dynamics to decide the next HP configuration to be tried out. For this, the authors propose the use of a kernel that encodes the previous HP iterates using a neural network. The method is shown to reach lower regret values for the same computational budget compared to the baselines considered. 

main_review: ### Post rebuttal

Given the detailed rebuttals by the authors, the updated baselines, I'm confident about increasing my rating for this paper. It might aid their arguments, if the authors were to move some sections of the appendix to the main paper. 

---------------------------------------------------------------------------------------------------------------------------------------
## Pros: 

The paper is quite clear (see the subsequent comments), and easy to follow. The method proposed is simple, and is an intuitive extension to the methods in the literature. The paper is also well placed in the context of previous methods. The experiments section is quite strong in the experiments and baselines covered. 


## Cons:


1. The authors start with the motivation that the rank correlation of performance at various budgets is poor. However, this is seemingly contradicted in Fig 7 left where the exclusion of the learning curve in HPO doesn’t worsen the performance much over all the datasets. The authors show experiments where the inclusion of LC leads to better results for some datasets. The authors should either provide references for the poor correlation, or show statistics of how intermediate performance is a poor predictor of final rank. 

2. On training the deep convolutional kernel: The neural net used is a MLP with 128 and 256 hidden units. This is quite a large network. How do the authors reliably train this network with only a few (xi, j, Yi, j-1) tuples? In the initial phases, how reliable are the networks predictions to terminate a run? How are the hyperparameters of this training chosen? The authors should give details, and report ablations on network size, architecture, training parameters (lr, batch size etc). In the absence of these, it is hard to judge the merits of the proposed \phi, as I do not understand how these specifics were arrived at. Also, the authors should report how the additional time of training the deep kernel changes the wall clock time measurements.

3. Evaluations: The use of Epochs and Steps to describe the x-axis in various plots is a little confusing. Are these the same? Also, it is ideal that the authors include the true performance (say test accuracy on CIFAR/ImageNet exps), and the true wall-clock times somewhere in the paper, in addition to the regret plots presented. Also the proposed method’s rank fluctuates quite a bit in the initial steps in Fig 3 and 5; can the authors comment on this?
4. Minor: 
“Reversing the training update steps” in the Intro para 1 makes it sound like undoing the update steps. 
The authors might consider rewording Para 3 of Motivation to aid readability, as it took me a few reads to grasp the point. 
The authors say “gradient descent and Adam” on Page 5 last para, and “gradient ascent and Adam” in A.4. 

5. Additional comments: These are general comments the authors might consider discussing. 
Do the authors find that the trained deep conv kernel is transferable across tasks? This might have interesting implications, if it can be. 
The authors write at the end of Section 6 “the additional use of an explicit learning curve representation might not lead to a strong improvement in every scenario”. While experimental evidence has been provided, can the authors describe what factors determine if incorporating LC dynamics leads to better HPO?

summary_of_the_review: The presented work is interesting, barring a few points commented on above. The motivation for this work needs further clarification from the authors. The experimental evidence of the efficacy of the method is strong. However, the paper in its current form misses some important details, and ablations. If the authors can address these points adequately in their rebuttal, I’d be quite happy to raise my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A gray-box hyperparameter optimization framework has been developed based on a multi-fidelity acquisition function and a surrogate model incorporated with learning curve dynamics. The proposed method was built on top of deep kernel learning [Wilson et al., 2016] and multi-fidelity Bayesian optimization [Kandasamy et al., 2017]. Experimental results on three different settings were provided in optimizing hyperparameters for MLP, RNN, and CNN, respectively.
main_review: Pros:
- Incorporating learning curve dynamics into the surrogate model is well motivated and supported by the ablation study on the NAS-Bench 201 dataset. 
- Extensive experimental results have been provided in terms of tabular datasets, NLP tasks, and NAS.

Cons:
- Predicting learning curves is not new for HPO as it has been well explored by previous works like [1]. While the proposed method tries to involve the budget information for modeling curve dynamics, the technical novelty of this work is still somewhat limited since it seems like a direct combination between [Wilson et al., 2016] and [Kandasamy et al., 2017]. 
- The multi-fidelity acquisition function is not well supported by the ablation study. What is the comparison result between DYHPO and DYHPO w/o MF?
- Some necessary baselines are missing in the current experiment, such as [1] and [Wilson et al., 2016]. 

[1] Learning Curve Prediction with Bayesian Neural Networks, ICLR’17.

summary_of_the_review: Overall, the paper is easy to follow and well-motivated. While some ablated models and baselines are missing, the experimental results are comprehensive and seem to be solid. The main concern of this work is the lack of technical novelty compared with existing works.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper present a new Bayesian Optimization algorithm that integrates a Deep Kernel over both the hyperparameters x and the fidelity budget j (typically number of epochs). It also present a slightly modified version of the expected improvement acquisition function to for the fidelity budget. The paper shows on several benchmarks that the proposed algorithm, called DyHPO is highly competitive, better performing than BOHB and DEHB on most of the benchmarks, otherwise performing similarly.  
main_review: Strengths:
- Multi-fidelity is very important to obtain practical HPO algorithms for deep learning. 
- The proposed deep kernel accounts for the correlation between learning curves to avoid naively stopping trials to early like HB does.
- The modification proposed to account for the learning curve is fairly simple
- The experiments are quite convincing, with DyHPO outperforming other HPO algorithms almost systematically. The baselines are good, with Hyperband, BOHB and DEHB being serious multi-fidelity contenders.

Weaknesses:
- It is very unclear to me how we can guarantee that the algorithm will often resample x to make it continue. If most dimensions of the search space are real, it seems to me x will most likely be different than previous ones, leading the algorithm to never continue trials. The empirical results clearly show that this is not the case, however the explanations do not make clear why it would not be the case.
- Only figure 8 reports training time in seconds (I assume including HPO time to suggest new trials as well). With the frequent query on the algorithm (every 1 epoch) I assume there must be a significant amount of overhead, starting and stopping trials very often. Also the algorithm itself must be fairly slow compared to hyperband, with the deep kernel that requires training. It would be best to report more results on the running time of DyHPO.
- There are no clear experiments with learning curves that are best later on to synthetically show that DyHPO performs well in this case. This would be extremely valuable to support strongly that the reason why DyHPO works so well is that it indeed let the best trials train even though they do not perform well at the beginning.
summary_of_the_review: The algorithm presented in this paper is an appreciable improvement to multi-fidelity variants of Bayesian Optimization, especially because it accounts for the correlation between the learning curves and avoid relying to strongly on low fidelity to make hard decisions on trials to stop or continue training. The experiments are convincing, they are broad and compare good baselines. The paper lacks important details in my opinion with respect to the optimization of the expected improvement and how it ensures a good fraction of the trials continue training. It also lacks analysis of the execution time of the algorithm and more explicit experiments showing it can avoid stopping good trials that progress slowly in the first epochs. I consider the work good enough for publication, but it could benefit from some clarifications and additional analysis.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a gray-box optimization method for hyperparameter optimization of deep neural network models. In order to deal with the different budgets available for training NNs in the framework of multi-fidelity optimization, the proposed method uses a multi-task Gaussian process modeling that simultaneously measures the similarity between not only the inputs x but also the outputs y trained with different budgets. In particular, the multi-task Gaussian process model is constructed using a deep kernel with a feature extractor instead of the existing kernel function. The performance of the proposed method is evaluated by experiments on three types of neural nets: MLP, RNN, and CNN. MLP and RNN are treated as usual hyperparameter optimization in 7 and 8 dimensions, respectively, while CNN is treated as a rewrite of NAS as hyperparameter optimization.
main_review: major concerns

1. There are several parts of this paper that are not clearly related to similar studies. 
    e.g. 
	- multi-fidelity BO with deep models 
		- Li+ "Multi-Fidelity Bayesian Optimization via Deep Neural Networks" NeurISP2020
	- EI for multi-fidelity BO 
		- Picheny+ "Quantile-Based Optimization of Noisy Computer Experiments with Tunable Precision" Technometrics, 55(1):2-13
		- Lam+ "Multifidelity Optimization using Statistical Surrogate Modeling for Non-Hierarchical Information Sources" 56th AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference. 2015. 

2. There is a lack of explanation about the architecture of the deep kernel (why this architecture, what makes this architecture effective for multi-fidelity optimization, etc.). It looks to be just a presentation of a kernel architecture that happens to work well.

3. It is not appropriate to plot only the average riglet in the experimental results, so the variance should be plotted as well (if it is difficult to plot, it can be reported separately).

4. Although the effectiveness of the multitask kernel is evaluated in ablation study, modeling using multi-task kernels in multi-fidelity optimization has already become popular and has been evaluated in various studies (e.g., https://arxiv.org/abs/1406.3896, https://arxiv.org/abs/1605.07079, https://arxiv.org/abs/1903.04703). Rather, what we should consider is what parts of the deep kernel structure are effective and why.
summary_of_the_review: I cannot support the acceptance of this paper due to insufficient evaluation of the novelty and the effectiveness of the proposed method.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper is concerned with multi-fidelity HPO (the authors call this "grey-box", which is a non-standard term). They propose a surrogate model for learning curve data (e.g., metric values at each epoch) based on a deep GP kernel. Different to most previous work on synchronous multi-fidelity HPO, they decide for each running trial when it should be continued. Experiments are presented, where the method is compared to a range of synchronous HPO baselines. The experiment are fairly small-scale and do not use parallel evaluations.

main_review: A potential strength of this paper is the proposal of a novel surrogate model for learning-curve data which, despite involving a neural network, seems to be operational on just the data observed during a single HPO experiment. There is a lot of prior work proposing learning-curve surrogates (see below), some cited here, but most are either quite simple (multi-task GP) or require "warmstarting" on data from previous HPO experiments. Having said that, I could not find any mentioning of this point, and I am really curious about the authors explaining how their "deep GP" model can be trained just on the very limited data observed during a single HPO experiment. For an expensive tuning problem, you probably have 20-40 configurations, most of which do not run for many epochs. And even if some configurations run for many epochs, learning curve data is exceedingly noisy.

The details really matter here. With a standard BO surrogate, I just need to refit the GP hyperparameters now and then, which can easily be done even for little data. With DyHPO, you need to presumably update a deep kernel, i.e. re-train a neural network. The paper does not say how this is done in a fully automated fashion. Is training started from scratch, or from the last recent weights? The first is expensive, while the latter is prone to get stuck at the previous solution and ignores the new data. How long do you re-train? Do you re-train after getting each new observation? While using complex (deep neural) surrogate models in BO is an obvious idea, many previous trials have failed, because complex NN models are just not easy/fast to update as part of sequential decision making, and in any case cannot be fit robustly to very small datasets. This fact has been clearly spelled out, for example in [7] for the closely related problem of bandit optimization. I'd be personally really surprised if this work was different and solved these difficult issues, but would be willing to give benefit of doubt, if a lot more information was provided here how the authors pulled it off. As it stands, the authors do not even mention there could be issues here.

The most obvious weakness of this paper is that very relevant prior work is ignored, namely on asynchronous multi-fidelity. Most prominently, ASHA [1] is well known and implemented in Ray Tune [3] or AutoGluon [4]. The baselines compared to against in this paper (Hyperband, BOHB) are all synchronous (and quite dated by now), meaning that many trials need to run to a certain level until another decision is taken. If you force methods to be synchronous, this puts them at a disadvantage. They need to delay decisions until some rung is completely filled, which delays decisions and slows them down. This is explained in the ASHA paper [1]. It is well known that for large scale multi-fidelity HPO, asynchronous scheduling works much better than synchronous, see for example the comparisons in [2]. Algorithms like ASHA are behind commercial automated tuning services [5]. It is quite astonishing that part of the research community is still considering synchronous methods like Hyperband or BOHB the state of the art.

For example, the paper claims that it is a new idea that DyHPO "never discards a configuration". That is precisely what ASHA [1] does as well (known as pause-and-resume scheduling), and what Freeze-Thaw BO suggested long ago. I'd be surprised if a well configured ASHA method (available in Ray Tune [3]) would not be competitive or beat the approach suggested here, despite not requiring a complex surrogate. When doing such comparisons, it is important to also take decision time into account, because updating a surrogate model can be very expensive. In DyHPO, this likely means re-running MLP fitting, which is probably really expensive.

I also find the motivation as to why DyHPO works better than previous methods unconvincing. The authors claim that rank correlations between early evaluations (at few epochs) and late ones are poor. In my experience, this is just not the case, these correlations are in the majority pretty good, which is exactly why multi-fidelity methods work very well for DNN tuning. Sure, there are examples such as regularization, but the question is whether that matters. The authors should provide numerical evidence for such a claim. Now, even if these correlations are poor, it is not clear to me why DyHPO could do anything about that. Learning curve prediction is just hard, because by far most of the data is from early evaluations, but you are interested in late performance, so you need to extrapolate. Why would some vanilla deep kernel be good at that? The only way to really know about certain anti-correlations that can be exploited, is to either fit models to data from past HPO experiments (which DyHPO does not do), or to built the knowledge into the model (which they don't do either). Just because an NN is involved, does not mean it will do magic for you. The reason why DyHPO works better than competitors here is that it is asynchronous, but the others are synchronous, so at a disadvantage. Also, the reason why model-based HPO is better than random search based methods (like Hyperband) is mostly because the latter cannot exploit: they need to draw new configs always at random.

The experiments are pretty underwhelming. Apart from most relevant baselines missing (DyHPO is asynchronous, all competitors are synchronous), the curves are also not very meaningful, because the x axis is number of epochs instead of wall-clock time. DyHPO needs to update a complex surrogate model, including retraining a neural network, and the costs for doing that have to be taken into account. All experiments are also sequential, no parallel evaluations are used (this could easily be done by using Ray Tune [3]), again this falls far short of the current state of the art in automatic tuning of large neural models (e.g., methods like ASHA or PBT).

Finally, there are quite some works on using complex surrogates to model learning curves in the context of HPO, for example [5]. It is not clear why this was not compared against, as code is available. The work of Wistuba and Grabocka is cited, which proposed deep kernel surrogates before (so this paper, against their claim, is not the first to do this in the context of multi-fidelity), and in fact Perrone etal (2018), cited here, did this even earlier, just not in the context of learning curve data. The paper of Wistuba is quite careful in explaining why a complex surrogate cannot be trained robustly on the data from a single experiment, and proposes an algorithm to warmstart from past data. It is dismissed here (as competitor) for doing so, but as I said above, I am not sure how DyHPO solves the apparent issue that complex NNs cannot be trained on the small amount of data observed in HPO.

[1] ASHA: https://arxiv.org/abs/1810.05934
[2] MOBSTER: https://arxiv.org/abs/2003.10865
[3] Ray Tune: https://docs.ray.io/en/latest/tune/index.html
[4] AutoGluon: https://auto.gluon.ai/stable/index.html
[5] https://www.determined.ai/
[6] https://openreview.net/forum?id=S11KBYclx
[7] https://arxiv.org/abs/1802.09127

summary_of_the_review: The paper may have some merits in suggesting a "deep kernel" surrogate model which, although quite related to previous work, is stated to work even if just fitted on the small amount of data from a single experiment, in an online fashion. However, this has been tried several times before with little success, and details explaining why the current approach should work are missing.

The proposed method uses asynchronous scheduling (much like Freeze-Thaw), but is compared against synchronous scheduling baselines, which have a major disadvantage. Comparisons to SotA methods like ASHA (or PBT) are missing, these are not cited. There is also quite a range of prior work on learning curve modeling for HPO, which is not compared against. Open source code for doing a better comparison is publicly available (for example, Ray Tune).

Experiments are smallish scale, mostly on tabulated benchmarks, and again are not close to what is possible today with parallel computation. Compared to missing alternatives like ASHA, the proposed method is fairly complex and quite likely rather non-robust to handle. For example, it requires retraining a neural network model each time a bit of new data is obtained, which is very difficult to do.
review_title: null

Meta-review: This paper presents a new method for performing Bayesian optimization for hyperparameter tuning that uses learning curve trajectories to reason about how long to train a model for (thus "grey box" optimization) and whether to continue training a model.  The reviewers seem to find the paper clear, well-motivated and the presented methodology sensible.  However, the reviews were quite mixed and leaning towards reject with 3, 6, 5, 3, 6.  A challenge for the authors is that there is already significant related literature on the subject of multi-fidelity optimization and even specific formulations for hyperparameter optimization that reason about learning curves.  A common criticism raised by the reviewers is that while there are extensive experiments, they don't seem to be the right choice of experiments to help understand the advantages of this method (e.g. epochs instead of wall-clock on the x-axis, choice of baselines, demonstration that early results are used to forecast later success, etc.).  Unfortunately, because there is significant related literature, the bar is raised somewhat in terms of empirical evidence (although theoretical evidence of the performance of this method would also help).  It seems clear that some of the reviewers are not convinced by the experiments that were presented.  Thus the recommendation is to reject the paper but encourage the authors to submit to a future venue.  It looks like the authors have gone a long way to address these concerns in their author responses.  Incorporating these new results and the reviewer feedback would go a long way to improving the paper for a future submission.
Decision: Reject