Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors consider multi-fidelity hyperparameter optimization (HPO). They introduce a new method, DyHPO, which uses Bayesian optimization (BO) with a newly proposed deep kernel that takes as input the full learning curve. Their approach also uses a new multi-fidelity version of expected improvement. At every iteration, DyHPO decides which hyper parameter set to train for an additional fidelity (epoch). The authors run extensive experiments on 50 datasets spanning tabular, image, and NLP data.
strengths_and_weaknesses: ### Strengths
This is a clean method that performs well. Intuitively, methods such as Hyperband or BOHB could make use of learning curve extrapolation to improve performance, so it is great to see some progress in this direction.

Extensive experiments: the diversity and the quantity of the experiments is well-done. They chose seven other benchmarks, from baselines to state-of-the-art techniques.

They have evaluated three research questions by giving hypotheses and then designing experiments to test the hypotheses.

### Weaknesses
It is not particularly novel. All of the pieces are similar to existing work (e.g., multi-fidelity Bayesian optimization, deep kernel learning, freeze-thaw Bayesian optimization, and learning curve extrapolation work). However, it is still non-trivial to design this method.

The approach seems pretty similar to Swersky et al. 2014 (freeze-thaw Bayesian optimization). That approach also uses Bayesian optimization to decide whether to pause or continue training dynamically over all hyperparameter settings. (The difference is deep kernels, using the full learning curve, and the specific acquisition function.) Ideally, the authors should compare to this method in the experiments, given that it is so similar.

I would have liked to see more ablation studies. For example, the authors do not really mention some related works on extrapolating learning curves, such as the following:
- Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves
- Learning curve prediction with Bayesian neural networks
- Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings
- Accelerating neural architecture search using performance prediction

The authors could discuss and compare to these other techniques that implicitly or explicitly use the full learning curve dynamics. 

The authors could also run an ablation study on the acquisition function, and the deep kernel vs another surrogate.

Lines 241-242 seems strange because you can simply decrease the amount of fidelities. For example, instead of treating every epoch separately in DyHPO, you could only allow DyHPO to query epochs 5, 10, 15, 20, etc. 

No code. 
questions: The authors say the will release the code in the camera ready version. Can the authors release their code during the author response period? For example, by using https://anonymous.4open.science/?  This would improve the contribution.

Can the authors run more ablation studies, such as some of the ones mentioned in the weaknesses?

Can the authors comment on Freeze-thaw Bayesian optimization and ideally run empirical comparisons?
limitations: The authors do include limitations and societal impact, but this paper is technically over the page limit, since the societal impact section is supposed to fit within the first 9 pages. The authors should get the societal impact section into the first 9 pages.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In the paper "Supervising the Multi-Fidelity Race of Hyperparameter Configurations" an HPO method for neural networks is proposed based on the framework of Bayesian optimization. To this end, the authors propose a kernel function, which is mainly implemented via a neural network, that accounts for different fidelities and is thus able to make use of all the data in contrast to other methods that employ a distinct surrogate model for each fidelity level.
strengths_and_weaknesses: ## Strengths
The authors address a severe issue of current multi-fidelity methods which are prone to reject/cancel out hyperparameter configurations which perform bad on early budget stages but would outperform configurations promoted to the next multi-fidelity levels on higher budgets. As an example the authors explicitly mention neural networks which are trained with regularization and known to perform better in the long run but are dominated by configurations that are not regularizing for lower budgets. I agree with the authors that this problem is one of the main drawbacks of multi-fidelity methods. Another example would be to compare more complex networks, which require more data for training, to simpler networks, maybe even linear ones, which perform sufficiently well on lower budgets already. In this case, the configurations representing the simpler candidates would seem beneficial in the first place and be promoted to subsequent stages.

The proposed kernel function seems to me like a reasonable and original contribution. At least I am not aware of any such method which would pursue a very similar strategy which is not discussed in the paper.

In the results the proposed approach is found to yield state-of-the-art performance. More specifically, DyHPO shows outstanding performance.

Overall, the paper is clearly written and well structured. Especially the experimental part, I really enjoyed as it was well structured and the addressed research questions have been made very explicit.

## Weaknesses
The authors promise to release their code upon acceptance. However, for a thorough review process it would be more appropriate to also hand in the source code.

Figures and algrotihms are positioned inconveniently throughout the document. First, text floats around some of the figures, I assume to save some space. Honestly speaking, I do not know whether this is against the style guidelines. In a hurry, I could not find anything about this in the guidelines, so I assume it is okay, still I would consider it a bad practice and "cheating on the space limitations" to some extent. Figures should be aligned to the top to avoid outcomes as on page 7 where two lines of text are given above the figure. As a reader I did not expect that and first missed the two lines. Also the fonts in the figures are sometimes too small and thus hard to read as a printed version.
questions: * ll.16ff: What about SMAC? Which is a method particularly tweaked towards the configuration of algorithms and therewith also for HPO?
* What are the drawbacks and limitations of DyHPO?
limitations: Conceptual limitations are not discussed. However, the scope of the proposed HPO method is more or less clearly limited to the hyperparameter optimization for neural networks.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed a novel multi-fidelity grey-box Bayesian optimization method based on deep kernel GP.
strengths_and_weaknesses: Pros

The motivation is clear and well-defined. Good summary of related work.  The experiments are comprehensive. Limitations were also discussed.

questions: Could the authors discuss more the efficiency in terms of time?
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents DyHPO, a new multi-fidelity method for hyper parameter optimisation. They key novelty at the basis of DyHPO is its new deep kernel GP. By relying on a deep kernel (based on a CNN) DyHPO can leverage the learning curve to recommend the next configuration to explore. DyHPO relies on an expected improvement based acquisition function to decide which configuration to recommend and evaluates each configuration at different fidelities, incrementally. The evaluation based on well-known datasets and state-of-the-art baselines shows that DyHPO achieves state-of-the-art results on the considered benchmarks, demonstrating its superiority and the advantage of its deep kernel.

strengths_and_weaknesses: **Originality:** The paper presents a new method for for hyper parameter optimisation, which is a novel combination of known techniques. The main difference from existing work is the fact that it relies on a deep kernel. Related work is adequately cite, [1] and [2] may also be of interest to the authors. [1] presents a new HPO approach based on HyperBand, while the one presented in [2] is a multi-fidelity approach that creates one model that exploits all fidelities, evaluating a configuration on a specific fidelity as a single "entity".

**Quality:** The claims are well supported, the evaluation methods and baselines are appropriate. The limitations of the proposed work are also discussed.

**Clarity:** The paper is clearly written and well structured. It's easy to follow and mostly easy to understand. It could do a better job at explaining how j is maintained and updated for each configuration.

**Significance:** The results are important as hyper parameter optimisation is an active area of research and given the demonstrated results of the proposed method, it is likely to have a high impact on the community. This is especially true if the source code is made available (as the authors say they will do).

[1] Mendes, Pedro, Maria Casimiro, and Paolo Romano. "HyperJump: Accelerating HyperBand via Risk Modelling." arXiv preprint arXiv:2108.02479 (2021).

[2] Mendes, Pedro, et al. "Trimtuner: Efficient optimization of machine learning jobs in the cloud via sub-sampling." 2020 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS). IEEE, 2020.

questions: **a)** I would have liked to see a plot with the ablation line (DyHPO without CNN) and with the other baselines, because it is harder to evaluate whether DyHPO's gains are all due to the deep kernel (i.e., it is hard to compare DyHPO without the deep kernel with the baselines).

**b)** How does multi-fidelity EI contribute to the gains achieved by DyHPO? Did you try other acquisition functions? For example, [2] proposes an acquisition function based on entropy search (ES)

**c)** I assume "regret" is the same as "loss". If this is true perhaps you could clarify this in the text, as "loss" is a commonly used metric.

**d)** Why do you sometimes use "number of epochs" and others "number of steps" in the x-axis?

**e)** In some of the LCBench datasets (e.g., ppendix, figure 12, "Amazon_employee_access", "bank-marketing", "blood-transfusion-service-center", "credit-g") DyHPO is no longer as competitive with state-of-the-art methods. Do you have an intuition as to which special characteristics of these datasets may lead to a worse performance by DyHPO?

**f)** When you evaluate the precision (experiment 3, line 289), you explain your definition of precision (which is consistent with the "usual" definition of precision = TP / (TP + FP)) however in the plot the y-axis says true positives. Is this a typo or are you actually showing the true positives instead of the precision?

**g)** I'm assuming the budget is the number of epochs during which you evaluate a configuration (I don't think this is explicit in the text or I may have missed it) and also that you "maintain" a j for each configuration, and when that configuration is selected you increment it. Are these assumptions correct? This is not very clear in the text.

**h)** Given the previous assumptions, I wonder how many configurations are explored (and on how many budgets) when DyHPO has already found promising configurations. That is, once DyHPO finds the configuration it ends up recommending, how much more does it explore instead of exploit?

**i)** What is the total cost spent by DyHPO compared to the baselines?

**j)** The title of the submission is not very clear to me. Why "Supervising"?

**Comments:**
- maybe you could add some markers to each baseline to make it easier to distinguish each line. I had to zoom in quite a bit (e.g. on figure 3) to see the results easily.
- Figures 12 and 13: legend mentions steps, but x-axis has epochs
limitations: Given that the kernel is based on a CNN, which is more computationally expensive that "traditional" (non-deep) kernels, I would have liked to see a discussion on how relying on such a kernel may prevent DyHPO from being widely adopted (due to the compute power required to deploy it). Second, the problem of setting the parameters of the CNN is also not discussed. For example, if I were to use a shallower NN to reduce complexity, how would that impact performance? Similarly, what would the performance gains be if I made the network even deeper?
review_title: null

Meta-review: metareview: All reviewers consider this paper to make a good contribution to multi-fidelity BO, it is "a novel combination of known techniques", and achieves state-of-the-art performance. Multiple reviewers ask for releasing code which authors promise to do before camera-ready deadline. The authors' feedback also addresses most other concerns or questions from the reviewers.

After discussion, the remaining concerns from one reviewer are that (1) authors should discuss more on related works that conduct learning curve extrapolation (LCE) and make a comparison to one of prior works (2) lack of ablation studies to answer e.g., "why does DyHPO achieve strong performance: the deep kernel, or the mulit-fidelity EI acquisition function? For (1), it would be useful to include the discussion on LCE related work from Appendix A.6 and the reply to NJxE into the main body of the paper. For (2), another reviewer finds the enhanced Figure 7 in Section 5 and Figure 8 in Appendix B from rebuttal useful to showcase the impact of the learning curve. I would encourage the authors to take into consideration the remaining concerns from the reviewers and edit their final revision accordingly.
confidence: Certain
Decision: Accept