Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a hyperparameter optimization algorithm that leverages the power law phenomenon of the learning curves. In particular, the proposed method DPL assumes the surrogate $\hat f(\lambda)$  of the true learning curve $f(\lambda)$ obeys a power law function, which can be parameterized by a neural network. The ensemble of K diverse learned surrogates can be used to determine the next hyperparameter. Empirical studies show that 1) the power law assumption improves the quality of the learning curve prediction; 2) DPL achieves state-of-the-art results in HPO; 3) DPL explores the search space more efficiently than baselines.
strength_and_weaknesses: Strength:

1. Adopting the observation that the performance of machine learning methods as a function of budgets usually follows a power law to hyperparameter optimization is quite novel.
2. The performance of DPL seems to be better than most of the competitors in terms of average normalized regret in several benchmarks, e.g. LCBench. 

Weaknesses:

1. The paper is not very well written that some parts of it is hard to read.

    1.1. Notation issues: 1) It should be $\alpha_\lambda, \beta_\lambda, \gamma_\lambda\in \mathbb{R}$ in eqn (3); 2) Why is $b$ bolded in eqn (4)(5)(6)(7)(8) but not bolded before and after?

    1.2. The paper is not self-contained that several techniques are used without introduction. For example, what is deep ensemble strategy? What is expected improvement? They seem to play an important role in the algorithm, but readers do not know what they are without referring to literature.

2. The empirical studies are puzzling.

    2.1. What is average normalized regret? Why is it an indicator of the efficacy of the HPO algorithms?

    2.2. I think a more convincing setting would be the following: 1) Choose a task with randomly initialized hyperparameter; 2) Apply DPL and other HPO methods; 3) Compare the final results (e.g. accuracy) of different methods; 4) Repeat these experiments to many tasks (NLP, CV etc.). After all, what we really care is the ultimate performance on the tasks. 

clarity,_quality,_novelty_and_reproducibility: The clarity and quality of the paper is not good, as explained in the weakness section. 

The proposed method sounds pretty novel though, as it is the first to leverage the power law in learning curves.

Since quite a few important parts are missing, such as deep ensemble and expected improvement, it is hard to reproduce the results based on the description in the paper.
summary_of_the_review: The proposed method is novel in leveraging the power law in learning curves, but the clarity of the paper is a main concern. Besides, the evaluation is not convincing enough to show the efficacy of the proposed method.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a multi-fidelity Bayesian optimization approach to hyperparameter optimization called Deep Power Law (DPL) that exploits the power law assumption of learning curves to quickly identify strong hyperparameter settings.  Results on 3 benchmarks with tabular, image, and NLP datasets show DPL to outperform 7 state-of-the-art competitors.  
strength_and_weaknesses: Pros:
- The empirical results are strong on the 3 benchmarks studied.
- Paper is well written and easy to understand.

Cons:
- Empirical analysis limited to pretrained benchmarks with a finite number of trained configurations.  In truly continuous search spaces, it is unclear how often DPL will repeat the same hyperparameter setting to be trained for more batches without additional discretization.  
- From what I could tell, experiments do not contain search spaces with step wise learning rate schedules where learning rates decrease drastically at fixed intervals.  These types of LR schedules often result in learning curves that do now follow the power law and are used frequently in practice.  
- DPL is limited in novelty since power law based performance prediction methods have been studied before and the multi-fidelity aspect just allocates a small unit of work per decision step.
clarity,_quality,_novelty_and_reproducibility: ## Clarity
The paper is clearly written for the most part but some of the details I was looking for was in the appendix or in referenced papers (e.g. description of the surrogate models and search spaces studied in each of the benchmarks).  I suggest the authors include the missing search space specifications in the appendix so the paper is more self contained.
## Quality
The quality is sufficient overall.  I found the multi-fidelity aspect of the DPL algorithm very simple and straightforward, which is okay but this aspect of the algorithm was oversold in the abstract and introduction.
## Reproducibility
I believe sufficient information is provided to reproduce the results for DPL shown in the experiments.
summary_of_the_review: While DPL outperforms many leading HPO methods on the 3 studied benchmarks, the experiments do not sufficiently establish DPL as a viable general HPO method due to missing experiments on non-lookup based benchmarks and lack of search spaces with step LR schedules.  Additionally, the novelty of DPL is limited since it is similar to prior power law performance prediction methods and has a very simple fixed allocation multi-fidelity component.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work propose a Deep Power Law (DPL) which uses the scaling law property of learning curves for hyperparameter optimization, by improving fidelity of HOP for deep learning. DPL ensembles predicts the performance of hyperparameter configurations in low-budget regimes as a probabilistic surrogator for Bayesian optimization (BO). This work demonstrated the effectiveness of the proposed method by conducting extensive experiments on 62 datasets under various domain such as image, tabular, and NLP domains.
strength_and_weaknesses: - Strengths
  - The writing is clear and easy to follow.
  - Applying scaling law property of learning curves for HPO is novel.
  - This work demonstrated their method on extensive experiments under various domain such as image, tabular, and NLP domains.

- Weaknesses
  - In the experimental section, I think that the title 'Hypothesis' is not appropriate. For example, even if this work demonstrated the performance of their work empirically not theoretically, they regarded Hypothesis 'Our method DPL achieves state-of-the-art results in HPO' is achieved.
  - With the most experiments, this work focus on 'regret' metric. Could the authors show the experiments with the metric 'accuracy'? For example, by applying the proposed method and multiple HPO methods on ResNet and cifar10/imagenet, we can see the model performance. 
clarity,_quality,_novelty_and_reproducibility: The motivation is clear and the proposed method using scaling law property looks novel. They provided the code and implementational details.
summary_of_the_review: They uses scaling law property for improving the fidelity of HPO and demonstrated the proposed method on multiple HPO benchmarks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors address the very important topic of hyperparameter selection, by marrying scaling laws and the Bayesian optimization hyperparameter search --- under the assumption of learning curves taking a powelaw form. 

The central assumption in the paper is that training curves, and in particular validation error as function of number of epochs, are well approximated by powerlaws, with the coefficients dependent on the hyperparameter selection.

Based on this assumption the authors propose a bayesian optimization method for hyperparameter search which uses a neural network (NN) estimator of the expected coefficients value of unseen hyperparameters.
The NN is trained on a batch of hyperparameters pre-trained configurations (learning curves), multiple times, with seed and sample order permutations, to form an ensemble from which mean and variance are predicted.

The authors showcase a preliminary demonstration on a CIFAR10 task of predicting the performance while scanning dropout level and show consistent uncertainty behavior with respect to the distance (in dropout level, and in number of epochs) between measured and predicted points.

The authors then continue to assess the validity of the framework and its performance on 3 benchmarks, relative to several alternative hyperparameter optimization baselines.

First, the agreement of the powerlaw-base learning curve is assessed relative to other learning curve prediction methods. Superior relative correlation is demonstrated.

Second, the authors examine the averaged regret of the proposed method relative to the baselines. And finally the authors examine the exploration / exploitation of the method.

strength_and_weaknesses: $\bf{Strengths: }$

Scaling laws and specifically their power law characteristics have emerged as a powerful tool for both practical and theoretical investigation. Attempting to apply this very strong prior for improving in a principled baysian framework for the gains in the efficiency of hyperparameter search is an important and potentially very useful direction.

The preliminary results in the paper are encouraging relative to current methods examined in some of the cases presented in terms of performance (though not convincingly in all — e.g. no statistically significant difference between the random baseline and DPL as shown in figures 4).
Similarly, the claimed efficiency (and demonstrated relative to waltime — figure 5) is of paramount importance, as at the large scale hyperparameter tuning becomes practically prohibitive.

$\bf{Weaknesses: }$

However, there are several methodological improvements which merit attention:

1. The core assumption of power-law learning curves, is poorly qualified.

It is still very often the case, in practice, that multiple schedules are involved in the training configuration. These, in particular, are subject of (hyperparameter) tuning and materially violate the powerlaw assumption. To name several such very common schedules one may consider warmup and learning rate scheduling (stepped or otherwise as in this paper where cosine scheduling with no repetition is applied at least in part). 

The proof of concept provided on CIFAR10 suffers from several limitations, not contributing to the further the confidence in the power-law assumption:
The uncertainty of the estimation diverges very quickly outside the interpolation area (area within the range of measurements) — figure 1 left. Further, the learning curve itself *does not* agree with the prediction, even on the measured area, further indicating that it is probably not actually a powerlaw in this case (as, by construction, the prediction is a powerlaw).

2. The analysis conducted in figure 2, does show the relative superiority of the correlation, however correlation may be too blunt an instrument for the purposes of interest. High correlation may be present with significant (perhaps even rank non-conservation) deviations at the accuracies of interest (where small absolute error differences are large relative differences, in the low error regime).

These limitations are not merely of aesthetics, qualifying when the results of this method may be trusted is very important in order to substantiate it as a good hyperparameter optimization method. 

For example — from what point in the learning curve is the approximation reliable? (and consequently, what is the expected computational/budget gain?) . and what limitations of applicability does this method impose with respect to the optimization protocol? 

At the least a relative deviation (e.g. error in log scale correlation) should be added, and with a measure not only of the average over all datasets, but rather a breakdown per dataset and with both average (over learning curve agreement with different hyperparameters) and deviations.
As well as a discussion and examples of where this prior actually does not agree with the learning curves and fails.

3. Evaluation of performance:

The authors use existing benchmarks, but show average regret or average normalized regret over a multitude of datasets. It is not clear from such an analysis what is the individual dataset performance or whether the standard deviation is very large as to hide phenomena which may adversely affect the performance of the proposed method in different scenarios. For example, a behavior which makes one wonder as to its origin is figure 3, PD1 average normalized regret which is worse than the random baseline up until >0.5 of iterations. 

This same concern is mirrored in the rank statistical parity of the random baseline even at full budget (figure 4, right).

Can it be explained in these cases why the method does so poorly (even if others are as poor).

Again, this is of practical importance --- hyper parameter optimization is an exercise which one engages in the context of a specific, single task. A method should be shown superior separately on all, or called out for its failure modes when not universally applicable.

4. Finally, it is unclear to this reviewer if this method has strong predictive power in the sense that it can be effectively used with a low budget in order to reliably precede large scale (prohibitively expensive) experiments. The benchmarks used may answer this question, but it remains opaque in the body of the work.
Specifically, since the DPL requires training on multiple degrees of freedom (hyperparameter dimensions) in order to form a reliable prediction along these degrees of freedom, to what extent is this done at small fidelity? What is the expected computational gain relative to current methods which involve e.g. grid search at the small scale and then hyperparameter scaling?

It is crucial (especially if this work remains in the purely empirical domain, as is currently governed by the NN at the heart of the hyperparameters-loss estimation), that the computational gain be fleshed out for this venue to offer a path towards superior hyperparameter optimization.

clarity,_quality,_novelty_and_reproducibility: The paper is clear, although leaves more than desired for the reader to devine in terms of the experimental results (this reader found himself referring to the code for clarifications on the implementation of the EI, which is not ideal — as the paper should be self contained). 

For quality, as mentioned above, the experimental design and analysis need to be improved.

If improved such that a robust and practical significant gain in hyperparameter optimization may be demonstrated, this would be a sufficiently significant novel contribution.

Reproducibility given the code is adequately addressed by the code, and I commend the authors for providing it — while some more documentation (starting with a non-empty readme) and cleanliness would be warmly advised to improve clarity, even orthogonal to this review…  

Code bits like hamper clarity:
(excerpt from surrogate_models/power_law_surrogate.py:  )
{
if self.iterations_counter == 200:
            b = 6
}


Finally, the paper would do well to engage more deeply with prior work.

Pertinent to the assumption of powerlaw training curves the work of Pretrum et. al [1] comes to mind.

Further, optimization hyper-parameter scaling with model and data scale (a very important canonic tool in practice), should be mentioned — i.e. batch size scaling, Kaplan et. al. [2]

[1] https://arxiv.org/pdf/2010.08127.pdf

[2] https://arxiv.org/pdf/2001.08361.pdf

summary_of_the_review: Hyperparameter efficient optimization remains one of the practical/economical thorns in the field. The paper offers an interesting direction of capitalizing on predictable phenomena in the form of scaling laws for the leveraging in the context of bayesian inference for hyperparameter search.

While preliminary results are encouraging, there are numerous weaknesses pertaining to the robustness of the underlying assumptions, performance and practical (namely compute) expected utility.

Significant work is required for addressing these above mentioned issues, and I will be happy to upgrade my review and score if they are adequately resolved.

%%%%%% updated score following rebuttal clarifications %%%%%%
see reply for details --- core issue is concern with respect to actual applicability due to mixed results. Strongly encourage to resubmit with demonstration of the utilization of method claimed superiority to improve performance in the wild
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new method of hyperparameter optimization (HPO) called Deep Power Law (DPL) utilizing scaling law behavior of loss curve being power-law. As scaling law is becoming a more and more important toolkit for understanding neural networks, the authors utilize the power-law property for hyperparameter optimization. 

Authors demonstrate on 3 benchmarks (tabular, vision and NLP datasets) that DPL outperforms other 7 SoTA benchmarks. 

The strength of the paper acknowledge by the reviewers are:
- Motivation is clear.
- Novelty of utilizing scaling law property of learning curve for HPO.
- Provides code and implementation details
- Empirical results are strong over 3 benchmarks on various domains

Here are few weakness raised by the reviewers:

- Limitation to continuous search space
- Use of learning rate schedules often leads to a non power-law training curve which is in tension the main assumption of the paper. While authors argued that benchmark suites do contain tasks with learning rate schedules, reviewers did not find that tasks included in the benchmark to be sufficient to ease the concern. 
- While some reviewers did find the paper to be clear, others found issues with clarity and quality. They had a hard time finding details of the algorithm and implementation to the extent the main basis for rejection. 

justification_for_why_not_higher_score: The authors did a very good job of answering questions and concerns from the reviewers. Some reviewers did adjust their assessment based on the interaction, however the majority of the reviewer did not find strength outweigh weakness and evaluated the paper as still below acceptance threshold for ICLR. Reviewers suggested significantly improving papers readability and highlighting superiority of PDL better would be ways to improve the paper but would require significant change. 

Even with the author discussion and discussion among reviewers, no reviewer was in strong support for the paper acceptance.
Decision: Reject