Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper concerns an adaptive learning rate algorithm based on the derivative of the objective with respect to the learning rate. He authors introduce the technique that they call “differentiable self-adaptive learning rate” (DSA) and present experimental results comparing their technique with conventional optimizers including Adam, AdaGrad, SGD with momentum in tasks including classification with MNIST, CIFAR10, CIFAR100, SVHN datasets.
main_review: The paper has serious shortcomings.

- The authors claim to introduce a novel method in gradient-based hyperparameter optimization (adjusting the learning rate by using the derivative of an objective with respect to the learning rate) when this has been done previously many times including by some highly cited papers such as 
 
Maclaurin, D., Duvenaud, D. and Adams, R. Gradient-based hyperparameter optimization through reversible learning. ICML 2015. https://arxiv.org/abs/1502.03492
 
Baydin, A.G., Cornish, R., Rubio, D.M., Schmidt, M. and Wood, F. Online learning rate adaptation with hypergradient descent. ICLR 2018. https://arxiv.org/abs/1703.04782
 
and a series of more recent papers covering similar approaches. The first paper deriving this method is actually from the 1990s (Almeida et al., 1998) and this is explained in the ICLR 2018 paper listed above.
 
- The method they claim to introduce and its derivation has the same analytical form as the method in the paper from ICLR 2018 listed above. So they are repeating the derivation of a method previously published at least twice without having any review or mention of related work which is quite easy to find.

- The submitted paper does not have any coverage whatsoever of related work on gradient-based hyperparameter optimization, and the related works section and introduction cover only very basic level adaptive learning rate algorithms such as AdaGrad or RMSProp. A good pointer for the authors to discover and survey related work would be to start with this paper by Bengio (2000):

Y. Bengio. Gradient-based optimization of hyperparameters. Neural Computation, 12(8):1889–1900, 2000. doi: 10.1162/089976600300015187.

and follow the trail of papers citing this paper up to our current decade.

In addition to these serious shortcomings, the paper is written in a way that makes vague and unsubstantiated statements such as:

- “we … realize an optimizer with truly self-adaptive learning rate”
- “our optimizer achieves fast and high qualified convergence in extremely short epochs”
- “Adam usually cannot converge well in the late stages of the training”
- “In a word, Adam is not sensitive enough”
- “None of the current optimizers are stable and sensitive enough to ensure a fast and high qualified convergence”

and many others throughout the paper.

summary_of_the_review: The paper makes the incorrect claim that the presented technique is novel and it crucially fails at finding and citing any instances of related work which are very easy to find with simple search terms like “gradient-based optimization of hyperparameters”.

I believe that this can be an honest case of an independent rediscovery of the technique by the authors combined with a lack of experience in finding related work and writing the paper in a more grounded way (e.g., avoiding vague statements). I believe that there is value in the experiments presented by the authors and I would like to encourage the authors to keep working in this problem space and improving their manuscript to get it eventually published after addressing the crucial shortcomings and finding a new angle to present their contributions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a method of adaptive learning rate in training deep neural networks and solving machine learning problems. The proposed method derives the adaptive learning rate by minimizing the loss function on the direction of $\\theta - \eta g$, with $\eta$ being treated as a parameter. 
main_review: This paper is well organized with an intuitive explanation and extensive experiments, and it can be considered as a good empirical study on investigating adaptive learning rates for solving machine learning problems. However, there are a few weaknesses of the paper, and I am listing them (with questions that could improve reviewers' understanding) here. 

1: First of all, the DSA method is in fact minimizing the loss function given current iterate $\theta$ and gradient $g$. Thus, if the modified loss function can be solved exactly, it would produce the optimal $\eta$ on the direction with current iterate and gradient. However, how could the author justify that it is better than a constant (or indeed any random) learning rate? Can you justify that in theory? 

2. Secondly, the novelty is weak as more literature review in optimization field would give rich content related to the DSA method. The classical way of deriving adaptive learning rate (mostly for the deterministic problem, but also stochastic problem in more recent literature) is line search. Line search finds appropriate learning rate along the same direction that DSA does. In addition, implicit SGD[1] was developed with implicit schema to derive a learning rate. Also, more recent work of deriving adaptive learning rate was proposed for variance reduced method [2].

3. Indeed, the modified loss function is a one-dimension optimization sub-problem. The way that DSA deals with the sub-problem is to either propose a gradient descent or use hyper-parameter to perform one-step update. Could you solve it approximately by using the newton method? In [2], a predefined sub-problem was solved approximately by newton method.

4. As authors define the terms in the paper, such as "stable", "sensitive", "grad loss" etc, this paper tries to explain the issues that was caused by non-adaptive method. However, the proposed DSA method is not solving the challenging optimization problem in the field. Could the weakness of convergence (as authors mention) for other optimizers be analyzed more? For example, optimizer often struggles to escape the saddle point, Adam with constant step-size only converges to neighborhood of local minimum, etc. I strongly feel that more in-depth understanding of issues with classical optimizers is needed for the proposed method to address the concerns in the field.

5. The empirical study is extensive. However, a few confusion that I am having are:

5.1 How DSA and other algorithms were configured? It seems like DSA also adopts batch training.

5.2 Does DSA incur additional cost? Could it be more fair to compare algorithms with respect to computational cost instead of epoch?

5.3 The plots for oscillation are specific for some problems, and would it cover general case?

5.4 Other algorithm requires fine-tuned learning rate, would it be fair to compare with fine-tuned version?


References:
[1] see https://sites.google.com/view/panos-toulis/implicit-sgd
[2] see https://arxiv.org/abs/2102.09700

summary_of_the_review: In summary, I'd love to see improvement in this paper. Based on my understanding of the field, I strongly recommend authors analyze the pro and cons of other algorithms and investigate in-depth the issues of not having a truly adaptive learning rate. with adaptive learning rate in developing optimization methods, it is often necessary to include theoretical analysis. And, theoretical analysis and empirical study can work together to understand the proposed method better.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a novel optimizer that makes the learning rate of a deep neural network to be differentiable in a self-adaptive manner, which overcomes the problems of existing problems DNN optimizers i.e., insensitive and unstable.
main_review: Authors propose an optimizer called DSA, which can increase or decrease very sensitively and significantly with the help of the learning rate’s gradient, which overcomes the disadvantages of the existing algorithms e.g., AdaGrad, RMSProp, and Adam. The proposed DSA is the first optimizer that has a clear instruction of directing the loss function, which applies to a wide range of DNN models as well as two classical machine learning problems. The paper has presented extensive experiments and proofed the values of DSAN as what I believe so far. 
summary_of_the_review: The paper is ready for acceptance.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The authors propose to tune the learning rate of a SGD, by training it by gradient descent.
main_review: The presented method is already known, the authors do not cite the preceding works in that line of research, and pretend that "[they] are the first to make learning rate in neural network differentiable with the goal of minimizing loss function".

Some non-cited preceding papers:
 * Seminal paper: *Learning to learn by gradient descent by gradient descent*, Andrychowicz 2016.
 * Paper with explicit optimization of $\eta$ by gradient descent in the context of feeforward neural networks: *Meta-SGD: Learning to Learn Quickly for Few-Shot Learning*, Li 2018.
 * Early preprint in the same in the same line of work: *Speed learning on the fly*, Massé 2016.

Besides, the notation lacks rigour: 
 * $\tilde{L}(\eta)$ is not truly a function of the learning rate $\eta$, it also depends on the parameters $\theta$.
 * The use of $\tilde{\theta}$ for the SGD update of $\theta$ is a bit clumsy: using $\theta_t$ and $\theta_{t + 1}$ would have been clearer. Besides, the use of $\tilde{\theta}$ without index $t$ to denote the training step shadows some freedom that we have when using the algorithm: we may want to train $\eta$ each $T$ steps (with arbitrary $T > 0$) instead of each $1$ step.
summary_of_the_review: This work is practically contained in existing papers, which are ignored by the authors. Strong reject.
review_title: null

Meta-review: The paper deals with the problem of adjusting the learning rate during gradient descent optimisation. Unfortunately the proposed approach is very similar to methods already presented in the literature and no significant contribution can be recognised. During the rebuttal, the author(s) have acknowledged their ignorance about the relevant literature and provided some further clarifications that did not turn into a revision of the reviewers’ initial assessment of the work.
Decision: Reject