Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ** Summary

The paper proposes an efficient framework to search for the optimal initial learning rate to train neural networks. The key idea is to introduce Knowledge Gain, a metric derived from the singular values of each layer, to indicate the convergency quality of training. Taking advantage of the metric, a logarithmic grid search algorithm (AutoHyper) is proposed to search for the optimal learning rate according to Eq 5 via short-time training (e.g. for 5 epochs), which is demonstrated to be very efficient and take effect to some extent. 


** Pros

1)	The paper studies a very important problem in neural network HPO: how to efficiently model and approximate the response function. The usage of Knowledge Gains as the proxy metric looks interesting and somewhat intuitive (however, I still have some concerns, see next). 
2)	I appreciate the authors present a lot of empirical results, on different models, datasets and optimizers. Detailed experimental settings and numerical data are also provided, which could be convenient to reproduce. 

** Cons

1)	The writing and presentation of the paper is not good. 
2)	From Table 5 to Table 8, it seems the automatically searched results do not outperform the suggested baselines significantly in many cases. 

** Technical concerns

In the definition of Knowledge Gain (Eq 3), though the distribution of singular values can be used as a metric to indicate the redundancy of a layer, to my knowledge it should be sensitive to the way of initialization; for example, an identity/orthogonal initialization may correspond to larger KG but random gaussian may result in smaller values. Furthermore, the evolving of KG during training may also vary from different layers, while in the formulation above Eq 4 it is just derived by averaging over all layers. So, I doubt whether the proposed metric is universally applicable for different situations. From the experiments, it seems the gain is not that significant. 

** Presentation

Another of my major concerns is that the paper is not well written. I can hardly understand what exactly the paper suggests, since many formulations and figures are not clear, for example:

1)	In Eq 2, what is the exact definition of \hat{W}_d? What does “low-rank factorized matrix” refer to? It seems the formulation here is slightly different from those in Hosseini & Plataniotis (2020).
2)	At the beginning of Sec 2.2, when \bar{\mathcal{G}}_\lambda equals to 0? (from Eq 3, it seems never to be zero). 
3)	In Eq 5, why to minimize (1 - \mathcal{Z})? Doesn’t smaller \mathcal{Z} correspond to better results? What is the meaning of the constraint in Eq 5?
4)	In Fig 1(b), how to calculate the histogram of \mathcal{Z} exactly?


================

Thanks for the rebuttal. I appreciate the authors' efforts on polishing the presentation. It clears some of my concerns. So, I raise my rating from 4 to 5. However, after reading the rebuttal and other reviewers' comments, I still feel that the contribution of the paper is not that significant,  since the search space just includes a scalar and the empirical improvements are not strong (Table 5 to Table 8). I think the authors may consider further applying their method to multi-dimension HPO problems (e.g. joint searching initial learning rate, momentum and weight decay) to verity the effectiveness on more challenging configurations. 

review_title: Interesting paper, but not well written

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors propose a new hyper-parameter optimization method based on a new response function defined on the low-rank factorization of 4D convolution weights. The presented approach appears to mathematically solid and interesting. Although I am not an expert in this hyper-parameter optimization, I think the approach has the potential to speed up and improve the hyper-parameter search or neural network architecture search.

In my opinion, the demonstrated experiments are less interesting because the focus is only on selecting a single initial learning rate. This is not that interesting because we normally have a learning rate scheduler that will change over time. The experiment somehow ignores different types of learning rate schedulers for comparisons. In addition, as the learning rate is just a scalar, we can simply do some grid search such as [0.01, 0.001, ..] to coarsely find a good learning rate and then refine.

Also, the presented work would be more interesting if it can demonstrate improvement in other hyper-paramter optimization such as weight decay and convolution filter size and channels. 

There are some questions about the presented approach:
- Are all the convolution layers in a CNN used in Eqn (4)? The definition of mode-3 and mode-4 is not clear to me.
- 
review_title: Review for Response Modeling of Hyper-Parameters for Deep Convolution Neural Network 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Overview:
Overall I find this an interesting algorithm, but have several serious concerns about the (lack of) experimental baselines.

My primary concern is that, given this paper is introducing a new hyperparameter tuning algorithm, there are no comparisons to baseline hyperparameter tuning setups. For example, in Figure 3, what is the best final validation and/or test accuracies achieved by autoHyper and by random search, for the same number of trials (one could even run autoHyper first to see how many trials it takes, and then see if random search could beat it in that many trials). I understand that BayesOpt algorithms can require more involved engineering to set up, but there do exist open source codebases that could be used (such as https://github.com/HIPS/Spearmint), and for a very solid paper I believe that comparing to existing SOTA methods would also be useful. For all of these methods, one could take the validation loss at 5 epochs (as is done in the proposed method) as the metric to tune on for a fair comparison.

Additionally, the suggested initial LRs seem problematic to compare to. It would useful to highlight how the suggested initial LRs were tuned. Using the information in Appendix D from Wilson 2017, I see they were tuned using less trials than the proposed algorithm, and using a grid search algorithm that may be worse than random search (Bergstra & Bengio (2012)). Furthermore, these suggested initial LRs were for a different model (this VGG model referenced from Wilson 2017 http://torch.ch/blog/2015/07/30/cifar.html) than the models considered in the experiments here, which means they could serve as a poor baseline. This baseline weakness is also shown when one considers that the ranges of values tuned over in Wilson 2017 seem to be quite close to several of the values proposed by autoHyper, meaning that perhaps repeating the same grid search (even with the same early stopping as in autoHyper) could be competitive (for Adam, Wilson 2017 considered {0.005, 0.001, 0.0005, 0.0003, 0.0001, 0.00005} and autoHyper proposed 0.000333, and for AdaGrad Wilson 2017 considered {0.1, 0.05, 0.01, 0.0075, 0.005} and autoHyper proposed 0.0049724). In the cases where the autoHyper values are not close, they are sometimes (although not always) outside the range considered by Wilson 2017, which could bias the results towards the proposed algorithm because the suggested LRs may not have had the chance to encounter the more successful values proposed by autoHyper (in the case of AdaGrad, Wilson 2017 considered {0.1, 0.05, 0.01, 0.0075, 0.005} while the proposed method tunes within [1e-4., 0.1] and selects 0.002861 for CIFAR10 ResNet34). Several recent works have shown that when tuning optimizers one needs to be careful to report the ranges used, as changing the hyperparameter ranges can drastically affect experimental results (https://arxiv.org/abs/2007.01547, https://arxiv.org/abs/1910.11758, https://arxiv.org/abs/1910.05446), and while they mainly consider the effect on optimizer comparisons the point still stands for comparisons between hyperparameter tuning algorithms (when comparing to random search, the range of LRs should be the same for each tuning algorithm).

Pros:
-It is useful to show that the tuning algorithm works across many optimizers, models, and datasets. The types of experiments seem sufficient, just missing baseline tuning algorithms.
-It is important to call out the negative results presented in experiments, and the authors did a good job of that (when applicable) in section 4.2

Concerns:
-Figure 1a would be much more informative if you showed the entire training trajectory, including past the first 5 epochs, to see if the selected learning rates actually generalize noticeably better.
-In your conclusion you discuss that you could extend your tuning algorithm to multiple hyperparameters, and I believe in order to truly demonstrate its capabilities this would be required, given that relatively simple baselines such as random search can continue to perform well in multi-dimensional tuning setups (although I would recommend against tuning the batch size given that it interacts so strongly with numerous other hyperparameters, and would instead suggest tuning momentum, weight decay, and/or label smoothing).

Writing:
-Is there an extra 2 in the denominator of Zt(λ)?
-It was initially confusing that the gradient notation was used for the constraint on Eq. 5, it would be clearer if the authors stated earlier that they did not compute the literal gradients (which would require backpropagating through unrolled updates).
-The “Observations on the generalization characteristics of optimizers.” subsection seems somewhat out of place

Additional feedback, comments, suggestions for improvement and questions for the authors:
-Instead of calculating Z(λ) with G¯λ(t, l), one could imagine making another baseline out of using validation accuracy at each epoch (perhaps normalized by some SOTA number if being in [0, 1] is desirable.)
-While very informative, Figures 5 and 10 seem cluttered, and in addition to them it may help for a future version to have the diff between the baseline and proposed method instead of two different lines
review_title: Interesting algorithm but lacking competitive baselines

final_score: rating: 5: Marginally below acceptance threshold
review_content: ###############################################################

Summary:

This paper used the knowledge gain to provide an algorithm to choose the initial learning rate. The paper explained the reason that choosing the specific response function and demonstrated the effectiveness of the algorithm by several experiments. 

##############################################################

Reason for Score:

The intuition is reasonable, but there are several points regarding the feasibility and correctness that need to be clarified.  

##############################################################

pros:

1, This paper proposed a dynamic tracking algorithm of low computational overhead on the order of minutes and hours to find a good initial learning rate.

2, The paper explained in detail about the reason to choose the specific response surface function, and demonstrated the reasonability of the algorithm.

##############################################################

cons:

1, The algorithm had an implicit assumption: the response function is monotonously decreasing (maybe under expectation). Otherwise, the algorithm can either not converge or go to a local minimum. Even though the author list some examples, I doubt about this. Imagine that we choose a very large learning rate (some unreasonable number that just for a counterexample), the training loss cannot be decreased, then I doubt that the response function cannot decrease either.  Did the author conduct some examples about this? If this is the case, then how to choose the range of the searching area matters a lot.

2, The algorithm induces some "hyperparameter" again. Such as the alpha, and the condition "rate of change plateaus". How to choose the alpha, and what is the "rate of change plateaus"? Does these "hyperparameters" influence the result?

3, The initial learning rate seems only influence the convergence rate for training loss, but why it influences the testing accuracy? I suggest the author at least give some reasonable explanation about this.

4, The paper compared the algorithm only with the baseline. Is there any other work about tuning the initial learning rate? Or this paper is the first?

5, minor problems:

(1) In equation 1, x is not showed in expectation function (I can only see X(train)), and also, it should be the minimum of expectation, not the expectation of minimum.

(2) In algorithm 1, there is nowhere that the index i is changed.

(3) I suggest the author change the color of the lines in graph. It is very difficult to find which line corresponds to which experiment.

Thanks for your rebuttal. It solves some of my concerns. However, combining with rebuttal and other reviewers' comment, I think only choosing the initial learning rate is not that reasonable. It may be more convincing to me that the whole parameters are chosen together. 

review_title: A new algorithm to choose the hyperparameter - initial learning rate, but has some feasibility and correctness problem

Meta-review: This paper studies the important problem of efficiently identifying good hyperparameters for convolutional neural networks. The proposed approach is based on using an SVD of unfolded weight tensors to build a response surface that can be optimized with a dynamic tracking algorithm. Reviewers raised a number of concerns which were not fully addressed in rebuttals and lead me to recommend rejecting this work. In particular: focus on single hyperparameter (learning rate) made it unclear whether the proposed approach could actually be used for other hyperparameters or to jointly optimize combination of hyperparameters, empirical improvements even for learning rate are not strong and baselines are weak, and concern that initial success early in training (5 epochs) may not lead to generalization late in training. Additionally, there were several concerns around the clarity of the presentation, which I also found hard to follow: how is KG related to information-theoretic metrics, why is the particular form of averaging across layers reasonable, and how is it related to other generalization / performance metrics? With additional experiments on other hyperparameters (for example L2 regularization), I think the work would be greatly strengthened. 
Decision: Reject