Trainable Learning Rate

Abstract: Selecting an appropriate learning rate for efficiently training deep neural networks is a difficult process that can be affected by numerous parameters, such as the dataset, the model architecture or even the batch size. In this work, we propose an algorithm for automatically adjusting the learning rate during gradient descent. The rationale behind our approach is to train the learning rate along with the model weight, akin to line-search. Contrary to existing approaches, learning rate is optimized via a simple extra gradient descent step, justified by an analysis that takes into consideration the structure of a neural network loss function. We formulate first and second-order gradients with respect to learning rate as functions of consecutive weight gradients, leading to a cost-effective implementation. We also show that the scheme can be extended to accommodate for different learning rates per layer. Extensive experimental evaluation is conducted, validating the effectiveness of the proposed method for a plethora of different settings. The proposed method has proven to be robust to both the initial learning rate and the batch size, making it ideal for an off-the-shelf optimization scheme.

Reviews and author responses:

 We are thankful for the overall feedback, providing valuable insights about the presentation of the experimental results.  We respectfully disagree with the deduction "you spent some considerable effort to find a good configuration of your method on the considered baselines, as evident in the choices of p, a_0 and c". The comment "If we only care about optimizing a single workload's performance, it's very likely SGD with a well tuned schedule would match/outperform all considered methods you present" is in line with our main point. We do not state in any case that the proposed algorithm could outperform a carefully tuned optimizer+scheduler, rather that we propose a robust enough approach w.r.t. hyper-parameters that can swiftly provide well-performing solutions. Moreover, from our perspective, the advantages over Baydin et al. are straightforward: we impose a better, more stable behavior to the learning rate development through the use of second order information. Apart from the theoretical benefit of such a Newton formulation, the experimental results (along with the extra presented Figures) show exactly this acclaimed robustness, contrary to the simpler case of Baydin et al., where different settings lead to different behaviors. Of course, if we choose a specific fine-tuned meta learning-rate per initial hyper-parameters/datasets/models we will achieve similar behavior, similar to the case of a fine-tuned scheduling tactic.
 
As an extra observation, the reported bound has an entirely different motivation in our formulation compared to the case of the first-order derivative. Specifically, in our formulation the bound is used to prevent an exploding increase of the learning rate in the initial stage of the proposed algorithm due to the possibility of the denominator being close to zero.  To this end we selected a rather restricting bound in order to be sure that such exploding behavior is not met. We considered that such a bound is not optimal and a dynamic bound would further improve the behavior of our algorithm. However, this bound is not responsible for stopping the oscillations in our algorithm, since the decreasing learning rate stage has an entirely different expression, resembling an exponential decay compared to the behavior of Baydin et al. which may lead to a rapidly decreasing learning rate. Applying a similar bound to the first-order case (Baydin et al.) will cut off entirely the oscillations and simply stop the whole procedure since the learning rate is zero-ed when it becomes negative, but it will not be advantageous for the whole procedure apart from terminating oscillations (note that "small" oscillations may be helpful). This can be seen in the following figures where the same bound was used (anonymized links): https://drive.google.com/file/d/1zqBJqE3DpPHF05i-tL9a93d9lvF66gtt/view?usp=sharing ,  https://drive.google.com/file/d/16CXPXnMkfg-kimPHVd4Ia8q0EhA6akyr/view?usp=sharing .

 Thanks for the additional plots, but they still aren't fully addressing the reviewers concerns. You've tuned an update rule that (alpha_t < 1.25 alpha_{t+1}) which is independent of the second order term. A similar rule seems like it could fix Baydin et. al. particularly given the unstable Baydin runs have the learned lr jump quickly.

It is really easy, particularly when you are running all of your own baselines to tweak a method to make it look good (this is how our community has managed to publish 100 Adam variants that don't actually outperform Adam). What reviewers are asking for is a good faith effort to look for a single stable Baydin et. al. configuration that works on the considered baselines (you spent some considerable effort to find a good configuration of your method on the considered baselines, as evident in the choices of p, a_0 and c).


"Concerning our method, let us clarify that we did not select an "optimized" averaging hyperparameter. Figure 6 only reports the effect of this hyper-parameter to our algorithm, since we had stated ⅓ as our default value in the main text (e.g. the case of  has a higher end accuracy in the example of Fig. 6)." The fact that p=0.5 is the best on one workload isn't really relevant here, because you are arguing that your method works "ok" on diverse workloads without needed to be tuned for workloads individually. If we only care about optimizing a single workload's performance, it's very likely SGD with a well tuned schedule would match/outperform all considered methods you present.

"Concerning the idea of choosing a default meta-learning rate value, one can say that  works well enough for the scenarios presented in Fig.9 (at least for CIFAR10/100; in MNIST this isn’t optimal). However, this observation cannot be generalized because the reported behavior corresponds to a specific initial learning rate ()." This doesn't really answer the question on whether or not Baydin et. al. can be tuned to do well on most of the considered workloads. The fact that Figure 9 found a configuration that matches your method on 2/3 and is close on the third does beg the question.

 Thank you for the interesting question. Indeed, if we should assume that Baydin et al.+averaging (+single meta lr, which by definition is part of Baydin et al.) *could* be combined with a single workable hyperparameter configuration, that would hypothetically be optimal over any workload and constant regardless of the dataset at hand, then the proposed method (efficient SGD-TLR / sgd_tlr) would have little added-value over this setting. We argue that a) such a setting cannot exist, and that meta-learning rate is indeed a critical hyperparameter, b) the strength of our method lies in exactly that we do not need to bother fine-tuning meta-lr over the workload at hand. According to our method, the step we need to take for adjusting learning rate is automatically tuned in each iteration, theoretically justified, and following the updates shown in-text. 

We apologize and thank you for pointing out the erratum in Figure 9 concerning the y-axis; we have fixed this (see below). Figure 9 serves to support our claim that the required meta-learning rate is indeed a critical hyperparameter. We agree that despite showing this sensitivity, it is not clear how our method (sgd_tlr) compares to a “good” setting of this hyper-parameter. To better understand the performance of the fixed meta-lr variation (Baydin et al.+averaging) that we explore in Fig.9, we provide the curves of our method along with the reported ones and a scaled version of the axes in the following anonymized gdrive link: https://drive.google.com/file/d/1CHJNAt8c1PWIzgMc0fUQF3d8aiwKVx7m/view?usp=sharing \
Please note that we have added a new curve for sgd_tlr, comparing it to the variants that require a fixed meta-lr. Therein it is shown that our method has better overall behavior across all settings, in the sense that our method attains a better result faster and much more consistently than the other settings (for example, meta-lr=0.1 is oscillating wildly between good and very bad loss/acc values).

Concerning the idea of choosing a default meta-learning rate value, one can say that $\eta=0.01$ works well enough for the scenarios presented in Fig.9 (at least for CIFAR10/100; in MNIST this isn’t optimal). However, this observation cannot be generalized because the reported behavior corresponds to a specific initial learning rate ($a_0=0.1$). Changing the initial learning rate can lead to different $\eta$ requirements, as this extra figure shows (anonymized gdrive link) for $a_0=0.01$: https://drive.google.com/file/d/1cjsNfUl5P4z4GONob_-_Cvl21uSpjfd5/view?usp=sharing \
For example, in this setting of $a_0=0.01$, $\eta=0.01$ leads to an oscillating behavior and suboptimal performance. (A minor, yet interesting, observation is that large values of $\eta$, such as 0.1, may lead to intense oscillations that resemble a restarting tactic and can achieve “unstable” solutions of very good accuracy).

These results suggest (perhaps not surprisingly) that the choice of an optimal meta-lr is strongly correlated to other hyperparameters, as well as the characteristics of the dataset / problem at hand. We interpret this conclusion as an advantage of our method, which integrates a mechanism that in effect computes an optimal meta-step and furthermore fine-tunes it in each iteration for practically negligible cost.

Concerning our method, let us clarify that we did not select an "optimized" averaging hyperparameter. Figure 6 only reports the effect of this hyper-parameter to our algorithm, since we had stated $p=⅓$ as our default value in the main text (e.g. the case of $p=0.5$ has a higher end accuracy in the example of Fig. 6). 


 These are fantastic results, thank you so much for running them!! I hope that in a final version of the paper these "tunability" results can be highlighted in the main text. I was considering dropping my score to help have decisive reviewer scores, but these results make me optimistic of a paper that I believe can more clearly convey the message of ease of use instead of method superiority. However I now believe this paper is borderline, mainly due to the concerns raised by other reviewers, so I'll be leaving my score the same as is.

 A major question raised by some reviewers is whether or not Baydin et. al. + averaging + find a single meta lr that works across workloads would be competitive with the proposed algorithm. More generally, for the considered baselines try to find a single hparam configuration that works well across the considered workloads, my current understanding is the considered baselines don't really do this. E.g. your method works better for Adam default with no lr schedule, but no one has really tuned Adam to find a default configuration (say a fixed decay schedule) that works well across multiple workloads. 

Figure 9 is getting towards this, and shows that Baydin et. al. + averaging + meta lr=.01 does well on MNIST, CIFAR-10, CIFAR-100, but it's unclear whether or not this is competitive with the proposed method (e.g. the y-axis is removed from this plot making any comparison impossible). To further complicate this comparison, the averaging hparams you chose assume you are adding in the second order update (Figure 6), the optimal averaging hparams may differ without the second order update.

What baselines do the authors provide that are truly targeting the "no tuning" regime they consider? Adam default with constant lr is a fairly weak baseline in this space.

 We thank the reviewer for the effort and the time spent summarizing possible concerns. Even though we tried to address similar concerns raised by the reviewer in the respective thread, we will try to clarify possible misunderstandings based on the reviewer’s comments. 

**Relevant Work:** First, let us be clear about the case of the work of Amid et al., 2020. As we have stated already in the respective discussion thread, the brevity and density of this work of Amid et al. may have led us to a misjudgment of the existing similarities. Note that we fully understand that the suggested paper deals also with a meta-optimization approach, with a common starting point (meta-update of hyper-parameters based on first order gradient, an idea dated back to Almeida et al., 1999), but we consider our ideas and analysis much closer to Baydin et al., 2018. Specifically, Amid et al. tried to improve the idea of the first-order gradient update w.r.t. hyper-parameters by using exponential decaying scales as well as coordinate-wise momentum gains along with a base learning rate, whilst we focused on the second-order derivative w.r.t. the learning rate in order to assume a Newton approach. Note that closely related works that motivated us and share similar concepts and formulations, are already cited in our work and we do not claim their ideas as our own. Also please keep in mind that it would be very difficult for us to produce new experimental results in the short time of the rebuttal, when no code was provided for the work of Amid et al, 2020. Nonetheless, based on the reviewer’s viewpoint as an important relevant work, we would gladly add it in the respective section (Sec. 2: Related Work), as a recent development of this meta-optimizing concept of gradient descent over hyper-parameters.

**Contribution:** As the reviewer correctly points out, the main contribution of our work is a novel second-order update. All the other contributions are considered under this second-order update framework: cost-efficient implementation (analytical cost-effective update since no Hessian-derivation is required), robustness to hyper-parameters, such as $a_0$, and the per-layer adaptation. We did not try to present these contributions as novel new ideas, but as effects and refinements of our proposed algorithm. If the reviewer feels that our phrasing suggested otherwise, we should rephrase accordingly. 

**Experimental Results:** We expected a level of sensitivity to the selection of initial learning rate, but we do not consider it notable (standard deviations are: +-1.14% for MNIST, +-1.05 for CIFAR10 and +-1.37 for CIFAR100). Taking into account the large variability in both batch size and initial learning rate, we consider the reported behavior in Figure 1 as a strong indicator of robustness. Note that we used a stair-case decay for comparison (sgd_mstep) with the same initial rate as in Zagoruyko and Komodakis, 2016. The improvement over Baydin et al. can be found in Figs. 1 & 9, highlighting the sensitivity of the meta-learning rate which does not exist in our framework. The point of our work is to present a robust enough approach without the need to fine-tune our choices, as the ablation studies show.  We do *not* claim that the proposed approach has no hyper-parameters, rather we argue on the necessity to carefully set them in order to swiftly get a well-performing solution (not necessarily the best possible).

**Short-horizon bias:** We fully acknowledge the problem of short horizon bias. We refer to this problem as a limitation of our work and we have added the respective reference in the updated manuscript. We are thankful to both reviewers hkZ3 and RyYD for suggesting this paper. We did not want to leave the impression that we "solve" this problem with the second-order formulation. Our comment was an attempt to get a better insight to why our method seems to generalize good enough, at least compared to the SMD algorithm. Once again, why we did not make any claims of solving the short-horizon bias problem and we continue to consider the short horizon bias effect as a limitation of our method, as stated in the paper. We apologize if our response was misleading in the first place.

 One thing I particularly liked about the paper is fine-tuning was not attempted at each workload.
Having said that, based on some similarities between your approach and the earlier ones, my revised score stands at 8.

 - We indeed used the same configuration of hyperparameters. We have not invested in finding the best performing configuration of sgd_tlr for our comparisons; our goal was to support that our method converges easily enough to a well-performing solution. Ablation studies of our method show that we can attain better end accuracy by using different lrs for different batch sizes, or by tweaking the values of p,c, but fine-tuning our approach at each workload was not in line with the concept of the paper.    

- In the extra experiments of Figs. 10-12, we focused more on SGD-based approaches which intuitively can be affected in a similar manner by changing the lr. (Vanilla) Adam was already proven very good at minimizing the loss at the cost of generalization - at least for our experiments. SPS and SLS do not have an initial learning rate hyper-parameter.    

- Negative values in the second-order were indeed a rare case, almost never met and not bothered us in practice. However, a "problematic" case can occur when this second-order term gets very close to zero (this is not so common too, but can happen in the change between increasing/decreasing phases) and thus a very large increase in learning rate is requested which may lead to divergence. That's why we added the bound c.   


 The authors have mentioned that with the exception of Figure 5 (which forces a smaller initial lr for transfer learning task) all experiements with sgd_tlr use the same configuration (a0=.1). Am I correct then that this holds for all sgd_tlr hyperparameters (e.g. update frequency, the c term in eq(8) ect.)? So all sgd_tlr experiments involve no tuning across all workloads?

Why are adam, sls, sps not included in the Appendix Figure's 10-12?

Have the authors measured how often during training the second order term in eq 9 is negative? Is it a rare case?


 As a sidenote, even though we focused on widely-used datasets in order to be in line with relevant works, we agree that “using new models on existing datasets”, as reviewer rR3n suggests, or even using a dataset or datasets that would be “unknown” or less-known to the community, could be beneficial for an unbiased comparison between our method and scheduling tactics. To properly design such “unknown” settings, or “scenarios where learning rate hasn't been optimized” as Reviewer Q9e6 mentions, was not possible in the rebuttal period. Nevertheless, we consider the underlying idea as a promising direction for an unbiased evaluation of meta-optimization algorithms in general (e.g., consider a small set of problems/settings which require different scheduling setups and compare the cost of fine-tuning a scheduler against an adaptive optimizer).   

*update: Minor Clarifications*

We further want to clarify some points raised by the reviewer:
- Even though sgd_mstep is not carefully tuned in our experiment, we use the same initial learning rate (0.1) as in the original paper of Wide ResNets and a similar multistep tactic (we considered different overall epochs and thus not identical scheduling was possible). Different tuning per batch would be optimal, but for the considered range of lrs, Figs. 10, 11 and 12 of the appendix show an overall better performance for this value of lr. 
- We agree that Fig. 1 alone cannot lead to conclusions with respect to other works. It serves only as an indication of robustness. This acclaimed robustness is better understood in comparison to other algorithms, as shown in the aforementioned Figs. 10, 11 and 12 of Section D.5. Numerically, considering our method, the standard deviations corresponding to the reported results in Fig. 1 are: +-1.14% for MNIST, +-1.05 for CIFAR10 and +-1.37 for CIFAR100. As reference, we report also the results for SGD: +-22.08% for MNIST, +-4.27 for CIFAR10 and +-12.21 for CIFAR100, SGD with multistep scheduler: +-5.32% for MNIST, +-7.63 for CIFAR10 and +-19.34 for CIFAR100, and SGD with an exponential decay scheduler: +-5.92% for MNIST, +-15.54 for CIFAR10 and +-24.97 for CIFAR100. All the reported deviations refer to the same range of hyper-parameters (lr=0.001,0.01,0.1 & bs=64,128,256,512). 
- A minor comment on "considered adaptive techniques which can automatically adjust the LRs": adaptive methods indeed have hyper-parameters to be set (for example, we assumed the default parameter values for both SLS and SPS which can be potentially crucial - even if SPS does not requires an initial lr, other parameters exist). Our point was that from an adaptive optimization perspective, such algorithms can generate an underlying scheduling tactic on the fly, taking into account both the data and the model, while schedulers typically require to be manually re-tuned. Of course, despite this adaptation ability in theory, how well these algorithms will adapt to a new setting does depend on the selection of the hyper-parameters. 


 We greatly appreciate the feedback and we understand the reviewers’ concerns regarding comparisons of our method against schedulers. We are willing to update the manuscript in the camera-ready version with extra results comparing TLR vs schedulers, which we have performed as per the reviewers’ request. Before doing so, however, we feel that we must raise our concerns about comparisons to schedulers in more detail. We also report (the most crucial) part of our new numerical results.

The issue of designing an unbiased fair comparison experiment greatly puzzled us, since it could easily lead to a comparison that is *artificially unfair* either to the compared schedulers, or either to our method. In short, the problem is directly related to dataset schedules being “overfit by the community”, as reviewer rR3n points out. That is why we were restricted to a more theoretical discussion in Appendix C, at least thus far.

During the rebuttal period, we have indeed experimented with a warmup scheduler, as the reviewer suggested, and achieved better accuracy after performing a grid-search over basic hyper-parameters (base learning-rate, exponential decay) for the basic setting of Cifar100+WRNET_16_4 (with batch size=256). In the investigated settings (e.g. CIFAR) we more or less know what hyperparameter setup is favorable, exactly because the datasets we use are well-known to the community. Thus we can design appropriately the grid in order to quickly find a very well-performing solution, (at worst, only a small search will be truly necessary). In this case, our bias w.r.t. knowledge of “what works well” for the dataset leads to a comparison vs. an automatic method like ours, as even a grid search can target a small area and be relatively speedy. On the other hand, “pretending” that good scheduler hyperparameters are unknown to us, entails the opposite problem: if we assume no prior knowledge of the setting, the grid of possible hyperparameter values should be considerably extended. Such an extension would lead to a comparison that is more favorable to our method, especially when we expect that a range of values would be meaningless and they are reported for the sake of making a point on the superiority of our method.

Let us be more specific with a brief report on part of our results. Using a linear warm-up for the 10% of the overall epochs followed by an exponential decay, we have attained the following results as the *best* option from the conducted grid search (possible base LRs: 0.0125, 0.025, 0.05, 0.1, 0.2 / possible decay values: 0.90, 0.92, 0.94):\
SGD+scheduler: 73.69% (base_lr=0.2, exp_decay=0.94)\
Adam+scheduler: 71.13% (base_lr=0.1, exp_decay=0.94)\
However, we cannot be sure if a different setup would result in even better results -- an inherent problem in any grid search.
In comparison, our method achieves 72.68% (with a fixed initial learning rate $a_0=0.1$).

Interestingly enough, the SGD was rather sensitive to the choice of base_lr (most of the grid search results were underperforming), while Adam was very robust, achieving similar performance regardless of the base learning rate but has not provided competitive enough results. Numerically, we can report the sensitivity over the base learning rate choices of our grid search (base_lr=0.0125, 0.025, 0.05, 0.1, 0.2 / exp_decay=0.94), through the mean and standard deviation values:\
SGD+scheduler: 69.73 +- 3.19 % \
Adam+scheduler: 70.77 +- 0.45 %

Similarly, we can explore the sensitivity of our method for different values of $a_0$ ( 0.0125, 0.025, 0.05, 0.1, 0.2): \
SGD+TLR: 71.88 +- 1.10 % (we can achieve a performance of up to 73.51%)

As a final extra experimental result, SGD with cosine annealing and restarts achieves up to 74.79% accuracy, outperforming any other approach, indicating the importance of modern concepts such as cosine annealing/restarts. In fact, we consider combining our approach with such tactics, as we have stated in the Sec. C. 

Overall, we wanted to avoid painting a picture where schedulers or TLR (or any potential automatic method for that matter) is one worse than the other. Note that in our work we report the widely-used multistep approach in the main text, which most of the times results in slightly better end accuracy, highlighting the competence of schedulers (even if they assume a very specific predefined learning rate behavior). As our final remark states (and Sec. C), we consider our approach as a useful complementary model / orthogonal to (specific) scheduling tactics rather than a competitor. If the reviewers find any of these few quantitative results useful, we can add them in the corresponding analysis of Appendix C of the final manuscript.  


 I agree. Comparison with SGD/Adam with schedules is indeed useful and should be included. I also think that comparing the proposed method with a baseline LR schedule  with increasing epochs/hyperparameters would strengthen author's claims. I'm keeping the score as it iwas.

 For the time being I am keeping my score the same, but it is possible I could raise it based off further discussion.

Appendix D.6 results are very compelling, your method seems to converge quicker while maintaining competitive performance! Thank you very much for adding this analysis.

“we selected a good initial LR in order to achieve SoA performance by following empirical evidence of the existing literature” how exactly were these initial LRs chosen? If they were taken from existing literature, were they LRs that were picked to work with the same batch size and schedule used in the experiments in this paper? Also, when you say “considered adaptive techniques which can automatically adjust the LRs” oftentimes for algorithms that can adjust to the LRs, won’t tuning the initial LR can have a significant impact on performance?

I still think that it is useful to compare to SGD/Adam with schedules (even untuned or default hyperparameters!) in the main text, as this is what practitioners often do. The CIFAR/ImageNet experiments have very common schedules that are known to work much better than the currently plotted baselines, which to me seems accidentally unfair. Even if the goal is not to demonstrate that the proposed algorithm gets SoA, it is still useful to compare it to realistically competitive baselines (it’s hard for me to tell exactly what final values are achieved in Figures 3, 4 due to the vertical axis scales).

As I agreed with in my original review, it’s possible that this method can “auto-adjust” to new problems with less tuning than schedules with multiple hyperparameters, however I don’t believe that the main text experiments show this. First of all, there is still some tuning involved for the proposed method, and while there are for sure less hyperparameters than many LR schedules, I usually see many LR schedules that work much better than a constant LR with tuning of only 1-3 hyperparameters (also, there are some popular schedules such as cosine decay that often only use a single hyperparameter).

To show that the proposed method is better at adapting to new workloads with minimal tuning, I would expect to see experiments where both the proposed method and a baseline LR schedule are tuned with increasing numbers of trials/hyperparameters. Plotting the performance versus how much tuning is required to achieve good performance, this should then be able to show that the proposed method can achieve good performance with less tuning than an LR schedule. Of course, this introduces other comparison issues (how does one choose what to tune in the baseline?), but I believe that the paper could be significantly strengthened with even a simple comparison where the proposed method is tuned alongside a baseline where the LR schedule is a linear warmup+exp decay and baselines are an LR schedule where some of {peak LR, warmup steps, decay factor} are tuned. When some of these hyperparameters are not being tuned, I believe it is fair to set them to reasonable default values, similarly to how reasonable defaults are used for the proposed method.

You say “Note that typically even selecting the base learning rate can be bothersome for different models/datasets”, but don’t you also need to do this for the proposed method? In Figure 1 the argument is the proposed method can handle different base LRs, but you don’t show any results on how baselines can handle different base LRs which would be a much more compelling comparison than ablating over two different hyperparameters for just the proposed method.

To be fairer to the proposed method, it could be useful to train on new problems (which can be easily generated by using new models on existing datasets) that baseline LR schedules haven’t been overfit to by the community.


 I would like to raise serious concerns about the authors's lack of acknowledgment nor comparison to a highly relevant line of work (Amid et al. 2020: https://users.soe.ucsc.edu/~eamid/paps/funnel_ws.pdf, spotlight at ICML workshop on "Beyond first order methods in machine learning systems", video: https://icml.cc/virtual/2020/workshop/5737), as pointed out in detail in my review below. I sketched the similarity of the two approaches **line-by-line** and I hope the authors acknowledge this work and adjust the claims about the contribution of the current submission:  
- Trainable learning rate has been proposed many times before including in Baydin et al. 2018 and Amid et al. 2020, therefore not a contribution of this work.
- Per-layer learning rate has been proposed before in Amid et al. 2020 (e.g. see Figure 3 (a) for a result on MobileNetV1), as acknowledged by the authors in the comments, therefore not a contribution of this work.  
- Unlike Baydin et al. 2018 and Amid et al. 2020 which use first-order gradient descent and exponentiated gradient updates, respectively, for updating the learning rates (called “scales” in Amid et al. 2020), the paper uses a second-order Newton update. This is perhaps the main contribution of the paper.  
- Experimentally, the paper claims that the second-order update is more robust to choice of initial learning rate. This claim is not convincing: the performance in Figure 1 varies significantly when the initial learning rate is changed. There is no comparison to the performance of a well-tuned baseline with a proper learning rate schedule (such as stair-case or linear decay). Although I agree that there is improvement over Baydin et al. 2018 in their comparisons, but the authors refuse to acknowledge or compare to Amid et al. 2020 which uses EMA of past gradients and normalized updates and therefore expected to have improved results over Baydin et al. 2018.). Therefore, the method still seems to require tuning the initial learning rate, thus not providing any advantage over Baydin et al. 2018 and Amid et al. 2020 ito number of hyperparameters.   
- After raising a concern about the pitfall of short-horizon bias in meta-optimizers (Wu et al. 2018 https://arxiv.org/pdf/1803.02021.pdf), the authors seem to acknowledge this issue. However, their comment “In the framework set by Wu et al., we may interpret our approach as aiming to encode information about the loss curvature / surface in its update, in an effort to “negate” the short horizon-related problems.” is not convincing and it is unclear how the curvature information of a univariate function alleviates the short-horizon bias. I suggest comparing the results with the well-tuned baseline on the simple noisy-quadratic example (Figure 1 of Wu et al. 2018) and show that the proposed approach successfully fixes the problem.


 “Of course we do not claim that we are the first to introduce per-layer”:  
- the authors actually claim this in the paper and the earlier comments (from earlier comments: “adopt a NN-based viewpoint of the problem which effortlessly results to the per-layer variation under a mathematical framework which can produce “optimal” learning rate updates”, also “optimal” in what sense exactly?)

“As a sidenote, the importance of the proposed per-layer variant is two-fold: First, it is theoretically derived from Eq.2 without a heuristic rule and second we observe SoA performance for this variant in practice,”  
- Again, you refer to the per-layer learning rate as “proposed per-layer variant”. Later in the review, you also agree with my comment that Eq. (9) is still a heuristic when the denominator is negative “Eq. 9 has a theoretically-motivated robust heuristic bound”. Also, the results on the experiments are not as impressive, especially the robustness aspect that I pointed out earlier.

“We have explicitly stated (and multiple times across the paper) that the analysis up to the derivation of the first-order gradient w.r.t. the learning rate (Eq. 7) has been described before in a very similar manner.”  
- Yet, this was initially claimed as your contribution in the rebuttal.

Overall, I am not quite convinced with the reasoning that Amid et al. 2020 is “is not relevant enough to our line of research despite several superficial similarities”. I provided line-by-line similarity and posed contradiction between your claims in the paper and the comments throughout the rebuttal. To summarize:  
1) The method introduces per-layer adaptive learning rates which are treated as a trainable variables (done before: “Of course we do not claim that we are the first to introduce per-layer ”)  
2) Unlike Baydin et al. 2018 and Amid et al. 2020 which use first-order gradient descent and exponentiated gradient updates, respectively, for updating the learning rates (called “scales” in Amid et al. 2020), the paper uses a second-order Newton update (Main contribution of the paper: interesting idea, but as mentioned before, requires thorough ablations which I address next).  
3) Experimentally, the paper claims that the second-order update is more robust to choice of initial learning rate (Not convincing: as I mentioned, the performance in Figure 1 varies significantly when the initial learning rate is changed. For instance, how does the method compare to the performance of a well-tuned baseline with a proper learning rate schedule? I agree that there is improvement over Baydin et al. 2018 results, but the authors refuse to acknowledge or compare to Amid et al. 2020 which uses EMA of past gradients and normalized updates and therefore improves over Baydin et al. 2018.)  
4) Thank you for acknowledging the pitfall of short-horizon bias. Your comment “In the framework set by Wu et al., we may interpret our approach as aiming to encode information about the loss curvature / surface in its update, in an effort to “negate” the short horizon-related problems.” is not convincing and it is unclear how the curvature information of a univariate functions alleviates the short-horizon bias. I suggest comparing the results with the well-tuned baseline on the noisy-quadratic problem (Figure 1 of Wu et al. 2018) and show that the proposed approach successfully fixes the problem.



 The authors continue to ignore work of Amid et al 2020 (https://users.soe.ucsc.edu/~eamid/paps/funnel_ws.pdf) which contains line-by-line similarity up to Eq. (7) on devising a meta optimizer:  
1) Amid et al. 2020 use a SGD with momentum abstraction for the adaptive optimizer.
Eq (7) in the current submission is the expression after Eq. (5) in Amid et al. 2020 (per-layer learning rate (i.e. scale) gradient) when the meta-momentum parameter is set to zero (i.e. pure SGD, as is the case in the current submission).  
2) The expression before Eq. (4) is the per-coordinate learning rate (i.e. gain) gradient (which is not considered in the current submission).  
Amid et al. also use a per-layer learning rate (i.e. scale), but use a single hyper-learning rate for updating the per-layer learning rates. This is comparable in complexity to the number of parameters used in the current submission (alpha_0, c).  
3) Throughout my review, I have acknowledged the contribution of the current submission which is using a second-order update for the (per-layer) learning rates. However, in terms of number of hyperparameters, the current method and Amid et al. 2020 (with per-layer learning rates only) are comparable. Also, I raised concerns about how a second-order update (instead of a first-order exponentiated gradient update as in Amid et al. 2020) would magically solve the short-horizon bias problem. As I highlight later, the second-order update is still a heuristic (“Eq. 9 has a theoretically-motivated robust heuristic bound in order to provide valid Newton-Raphson step”).  
4) In terms of implementation, Amid et al. 2020 is extremely similar to your approach: it requires keeping track of the EMA of the previous gradients. I am sure you could easily add this to your current setup or ask the authors for code if needed.

Critically, Amid et al 2020 devised both per-coordinate as well per layer learning rate meta-optimization in contrast to what the authors claim (also done in several previous work, as the authors mention). This first must be acknowledged by the authors.

The talk from the ICML Beyond First Order optimization Workshop Spotlight from 2020 is available here: https://icml.cc/virtual/2020/workshop/5737

**Contributions:** First, let us consider the contributions claimed by the authors in the paper, despite their comments on the reviews where they acknowledge that these were done before (copied from latest revision at https://openreview.net/pdf?id=fHeK814NOMO):

"a) We show that we can construct a gradient-based method that requires neither specifying a fixed learning rate step, nor performing several search steps per line-search. Instead, the learning rate is itself cast as a trainable parameter. No manual tuning is required, and a simple update per iteration is used."  
- “nor performing several search steps” -> previous methods such as Baydin et al. 2018 and Amid et al. 2020 also do not require extra steps. They treat the learning rate as variables and apply gradient updates.  
- No manual tuning is required -> the method still requires setting the initial learning rate, despite the fact that the authors claim that the method is robust to such choice, which I address next.

"b) The proposed algorithm displays notable robustness to user-defined hyper-parameters of the training process, such as initial learning rate and batch size"  
- The accuracies in Figure 1 for choice of different initial alpha’s look quite different. How do the results compare to a well-tuned baseline with a proper learning rate schedule (e.g. linear decay)? I am not convinced that this approach is actually robust wrt the initial choice, therefore requires hyperparameter tuning.

"c) Implementation-wise the algorithm can be coded very simply, it is cost-efficient, and does not require any critical hyperparameter to be set."  
- Is there convincing evidence? How’s this different from Baydin et al. 2018 and Amid et al. 2020 - both of those are cost efficient. All that is needed for all these methods is keeping track of past gradient(s), which is also the case for the current approach. There is no obvious advantage ito memory and computation.

"d) We may naturally treat each layer separately, with a different trainable learning rate per layer."  
- This was already done in Amid et al 2020, as pointed out above. Please read the paper more carefully, e.g., Figure 3 (a) in Amid et al. 2020 shows the scales (i.e. per-layer learning rates) for different layers of a MobileNetV1 model. This is clearly stated in the paper and missed by the authors.

Additionally, your claims in the paper and throughout the author discussion period have been quite inconsistent. Sometimes, you claim that per-layer learning rate is your contribution, whereas elsewhere you agree that it was done before, e.g. in Amid et al. 2020. In the following, I highlight some of these cases:


 Now, let us address the reviewer’s comments to our initial answers:
* The reviewer claims “the layerwise-interpretation has already been used in Amid et al. 2020”. Of course we do not claim that we are the first to introduce per-layer and in Section D.2, where we comment on the per-layer adaptation, several layer-wise works are referred (most of them prior to Amid et al. 2020).  As a sidenote, the importance of the proposed per-layer variant is two-fold: First, it is theoretically derived from Eq.2 without a heuristic rule and second we observe SoA performance for this variant in practice, while we highlight a specific setting (detection task with pre-trained part) which serves as a practical use-case of such per-layer approaches. 
* We have explicitly stated (and multiple times across the paper) that the analysis up to the derivation of the first-order gradient w.r.t. the learning rate (Eq. 7) has been described before in a very similar manner.  Furthermore, we do not understand why the reviewer insists on referring to Amid et al., 2020 for these reported derivations, when similar expressions date to Almeida et al., 1998 and more recently to the closely-related work of Baydin et al., 2018 (all these are reported to our paper). Indeed, our major contribution is the introduction of an elegant second-order derivative expression.  Such higher-order information reveals the structure of the underlying LR function to be optimized and a Newton direction could be effectively selected. Following this Newton rationale, there is no need to decide for an extra hyper-parameter to control the meta gradient descent of the learning rate. Using such Newton approaches comes with a cost: they typically involve highly impractical Hessian derivations. The importance of the proposed Eq. 8 is that no such Hessian computation is involved and only constant-time operations between consecutive gradients are required, just like the first order gradient. The effect of this second-order derivative is mainly highlighted in Fig. 2, comparing to the previous method of Baydin et al. 2018. However, the reviewer does not acknowledge neither this theoretical analysis nor the comparison and discussion for Baydin et al., 2018. As we have already mentioned, we understand the importance of short-horizon bias, we have already hinted toward this problem and added this reference to the updated manuscript. Nonetheless, experimental results reveal a notable generalization ability of the proposed work in practice and this could be an interesting research direction for future work. Last, the reviewer states “If this is the main contribution of the paper, then it should be highlight of the work (theoretically and experimentally) by a clear comparison to these previous work and ablating across this choice” and we completely agree. That’s exactly why we have a thorough analysis and comparison with the very closely related work of Baydin et al., 2018 (see Fig. 2 and the corresponding text, as well as Sec D.4).
* The main point of the paper is to bypass the need of sensitive user-defined hyperparameters. The reviewer claims that “a tunable learning rate $\alpha_t$ involved for each layer”. We are not sure what the reviewer refers to exactly as tunable. The per-layer learning-rates are automatically “tuned” under the proposed optimization framework (“trainable learning rates”), similar to the global scheme, without the need of an external definition of a controlling hyperparameter. Naturally, an initial learning rate should be defined, as in the global case, and due to the robustness to such choices, supported by the experimental results, a common initial lr is set to every layer (as described in the manuscript). Indeed, Eq. 9 has a theoretically-motivated robust heuristic bound in order to provide valid Newton-Raphson steps, which can be intuitively set as our analysis and the experimental results of D.3 dictate.
* Finally, we do believe that the experimental results are further highlighting the contribution of this work, since we are comparing our approach to several very recent state-of-the-art adaptive optimizers with similar formulations and goals, such as Baydin et al., 2018 (ICLR), Vaswani et al., 2019 (NeurIPS)  Loizou et al., 2021(AISTATS). As we have already mentioned, we respectfully disagree with the claim that the proposed work “does not add anything to existing approaches” (e.g. see point 2). Hyperparameters of the proposed method and their intuition are explicitly stated in the manuscript (e.g. $p=0.33$ and $c=0.25$). We do not understand what the reviewer refers to as the “tuning process”. If this phrase hints to any underlying learning-rate tuning, we would like to bring to the attention of the reviewer that the learning rates (global or per-layer) are “trained” alongside weights with a SGD-like rationale.


 We can understand a possible source of confusion about the contribution of the paper and we will try to clarify in what follows. However, we are surprised by the continuous referral to the work of Amid et al., 2020. In fact, after having read this work following your initial kind suggestion, we believe this work is not relevant enough to our line of research despite several superficial similarities. 

Specifically, the all in all interesting work of Amid et al. follows a (loosely) similar initial mathematical formulation of the problem, as also done in several other related works (e.g., Baydin et al., 2018) which have been cited accordingly. After this, our analysis is much closer to the other cited works. Moreover, due to the length of the suggested paper neither an in-depth analysis exists nor convincing enough experimental results in order to consider it as a candidate for comparison (for example the proposed algorithms underperform compared to an SGD optimizer with a multistep scheduler / does not compare to other adaptive step-size algorithms). Please also note that Amid et al. 2020 contains a reference to available code that instead of code, leads to a “Hello world” page, https://users.soe.ucsc.edu/~eamid/funnel.html. It is also understandable that the workshop paper length restrictions perhaps are to blame for the brevity of the analysis of Amid et al.; a more refined extended version might have led us to different conclusions. 

Nonetheless, we acknowledge that this work of Amid et al. takes into consideration known restrictions of meta-optimization and applies several useful modules, such as exponentiated gradient updates/EMA (while using coordinate-wise learning rates), at the cost of introducing extra hyper-parameters. Such modules may be a handful in a future direction of our work and then this work of Amid et al. could be a more than suitable citation.

Note that in such a rapidly growing field, more and more works are published and we cannot cite them all. As expected, we selected and cited the most relevant works to ours, as well as groundbreaking papers in the field.


 Thank you for the suggestion, we added a relevant analysis in the appendix (Section D.6), where we report the number of required epochs for converging to a solution of a predefined target accuracy.

 We thank the reviewer for the observations/suggestions.

**Type of loss function:** The presented analysis focuses on a per-layer rationale and is therefore restricted to neural networks (conceptually) due to our interest in deep learning applications. This loose restriction can be easily overridden for a global learning rate analysis over arbitrary loss functions and weight parameters (e.g. as done in Baydin et al. for the first-order derivative).

**Fast convergence:** The end goal is to attain well-performing solutions regardless of the setting (initial learning rate, batch size etc.) and as fast as possible (fast convergence).  Therefore our goal is multifaceted and we consider that loss/accuracy curves contain all the necessary information about the behavior of the proposed scheme. To this end, we compare our approach to a set of different adaptive optimizers and validate this fast convergence property through these curves. However, according to the reviewer's suggestion, a more detailed analysis on the number of required epochs for obtaining a specific accuracy has added in Section D.6. We do not imply that our method attains a better solution in the long run, especially when compared to SoA finetuned scheduling approaches, but we propose a method which provides competitive performance and fast convergence without the need to carefully set optimization hyper-parameters.    

**Hyperparameter p:** The user-defined hyper-parameter $p$ is set throughout all the experiments, including vastly different datasets in complexity (MNIST/CIFAR10&100/ImageNet and Pascal VOC) along with a small transformer setting of Multi30k dataset in the appendix. Even though more diverse datasets/tasks could have been also used, the usage of $p$ is very intuitive: set the update frequency of learning rate. As we discussed in Section 4 (and shown in Fig. 6 of Appendix), setting small values may result in inaccurate predictions of learning rate (consecutive subgradients may correspond to very different data), while large values are unnecessary since NN weights are updated much more frequently than learning rate. Therefore, we do not consider this hyperparameter as critical and can be intuitively set to ⅓ (or a similar modest value) regardless of the dataset.

**Experimental Setup:** As the reviewer noted, a description of the experimental methodology exists in the appendix due to space limitations. Experiments were executed multiple runs (3) with random weight initialization. Results of Fig. 3 (and Figs. 13,14,15 of the appendix) correspond to the mean and std across 3 different runs.  All experiments but one, consider from scratch training. Only the object detection setting, shown in Fig. 5, has a part not trained from scratch and highlights the effectiveness of the per-layer alternative when a pre-trained part is considered hinting towards an efficient transfer learning capability. More experiments towards this should be conducted, but we consider it as out of scope of this paper. 

**Figure captions:** We thank the reviewer for the observation; we tried to improve captions wherever possible, however due to space limitation the main description is in the main text.

**Time requirements:** For the time requirements analysis, the extra computational overhead is rather straightforward, consisting of inner products for the first- and second-order derivatives. In the context of training a deep neural network, such overhead is trivial (as commented in the paper) and thus we considered only an indicative setting (as a proof of concept) to experimentally support this claim. 

**Small batch sizes:** Larger batch sizes do not consistently report better performance (e.g. for Cifar100 large batch sizes result in slightly better loss values and slightly worse accuracy). In fact, we have tried lower batch sizes (4, 8, 16 & 32) which resulted in similar behavior/performance, but we considered higher batch-sizes of interest due to their practicality in modern GPUs. Note, that due to the usage of hyper-parameter $p$, we expected stability of behavior across different batch sizes. 

**Small datasets and overfitting:** We do not have any experimental evidence to hint to such behavior (e.g. when using MNIST), nor do we have a clear theoretical motivation of such a case.

**Limitations of the proposed approach:** We have already highlighted possible shortcomings and limitations throughout the manuscript (extreme values of $p$, unbounded increase of learning rate etc.) and in the final remark at the end of Section 5, where we mention the possibility of divergence from imbalanced settings for the global learning rate approach (see Fig. 5) and possibility of convergence to suboptimal local minima. 
Moreover, in the updated manuscript we explicitly mention that the proposed method cannot outperform a meticulously crafted scheduler, adding a new appendix commenting on the connection of our method with learning rate schedulers.


 Regarding you answers:
- "adopt a NN-based viewpoint of the problem which effortlessly results to the per-layer variation": the layerwise-interpretation has already been used in Amid et al. 2020. I believe the authors should acknowledge this fact (the authors don't seem to even include a reference to this work in the  revision?).
- "second order gradient": second-order gradient for updating the step-size is the only contribution of this work (all the derivations up to Eq. (7) is a repeat of the derivations in Amid et al. 2020. Eq. (8) is the second-derivative, and Eq. (9) uses a heuristic to apply this equation to set the step-size). However, the authors still need to justify how going from a first-order to a second-order update (of a univariate function) would significantly improve previous methods (Baydin et al. 2018, Amid et al. 2020) as well as solve the problem with the short-horizon bias? (e.g., does this solve the noisy quadratic example in Wu et al. 2018?) If this is the main contribution of the paper, then it should be highlight of the work (theoretically and experimentally) by a clear comparison to these previous work and ablating across this choice.  
- "theoretically omits the necessity for setting a hyper-parameter": but in practice, there is still a tunable learning rate \alpha_t involved for each layer. Therefore, the approach provides no clear advantage ito tuning. Also, Eq. (9) is still a heuristic.
- Experimental results would only be valuable if the methods adds anything to the existing approaches. Also, without a clear description of the hyperparameter range, tuning procedure, and comparison to the more recent work Amid et a. 2020, these results would not conclusive. 

 We thank the reviewer for the feedback and the interesting suggestions (e.g. removing normalization or skip connections). In what follows, we address the concerns raised by the reviewer.

**Connection to schedulers:** Even though we agree with the reviewer about the evident similarities between schedulers and adaptive step size algorithms, the latter typically have a sense of optimality and are studied separately. That’s exactly why the comparisons in the main text are focused on adaptive approaches (along with the commonly-used multi-step scheduler as an indicative baseline). Acknowledging the similarities, we also added an extra section “Connection to Schedulers”  (Section C, Appendix), where we discuss the shared end goal of our algorithm and schedulers, along with their fundamental differences. We should highlight why we do not consider the comparison with schedulers fair: schedulers typically involve several hyper-parameters that are empirically set (initial/base learning rate, type of lr decay, milestones of decay, restart epochs, warm up epochs etc.), while adaptive step size algorithms automatically adjust learning rate in an “optimal” way according to a specific mathematical formulation. Note that typically even selecting the base learning rate can be bothersome for different models/datasets. This leads to our second point: we do not claim that TLR can outperform carefully tweaked schedulers, nor do we suggest replacing modern scheduling schemes. But, we can effortlessly provide an underlying scheduling policy without the need to tune anything. That’s exactly why in our final remark we consider our approach as “an ideal component for more complex heuristic schedulers”. For more details, please see the new Section C of the updated manuscript.

**Performance of Adam:** Adam is considered as an adaptive algorithm and thus compared as so without a scheduler (as also done in Vaswani et al., 2019 / Loizou et al., 2021). In other words, we are interested in reporting the performance of our method against other adaptive algorithms and even if Adam does not explicitly adjust learning rate, it has similar adaptive properties which assist in fast convergence. We do not claim that Adam along with a scheduler would underperform; on the contrary, it may outperform our method if chosen carefully, as stated in our previous comment, but this is not the main point of our paper. Last, while the reviewer claims that “CIFAR WRN Adam almost always outperforms other optimizers”, this is actually something that we have not encountered in practice. In fact, many recent works are using SGD variants for similar settings (e.g. Keskar & Socher, 2017). We would be thankful if the reviewer could provide any relevant literature to serve as a competitive baseline.

**Convergence performance:** Our aim is robustness and competitive performance regardless of the setting (initial learning rate, batch size etc.) and fast convergence to such well-performing solutions (of course, generalization ability is always a desired property of optimizers, even though we did not design our approach towards improving generalization). To this end, we compare our approach to a set of different adaptive optimizers and validate the desired fast convergence property through these curves, where essentially a per epoch comparison can be deduced (which methods perform better at x epoch). Similarly, we can deduce which method reaches a competitive performance first. Note that some of the adaptive optimizers (or baseline SGD) may not be able to achieve a pre-set competitive performance (e.g., only sgd_mstep and sgd_tlr attained an accuracy over 75.8% in the reported ImageNet setting of 80 total epochs). Nonetheless we understand the importance of such a setting and thus we added a relevant analysis in the appendix (Section D.6) with more quantitative results about the convergence on ImageNet.


 We are grateful to all the reviewers for the time they spent to read our paper as well as for their valuable comments.
We have added answers addressing the issues the reviewers raise, to each reviewer separately.

As a general remark, we would also like to note the following, mainly concerning requested changes over our manuscript:
* We have removed the former "Appendix B" which outlined the steps required to go from the "per-layer" derivation of the first-order gradient to the "global rate", and in its place we have included a simplified derivation of the same result. This result has been integrated in the main text. Also, we have clarified our notation for this part of the analysis.
* Concerning our "related work", we have restructured it by including numerous works that are relevant to the current work specifically concerning core ideas like using consecutive gradients and a meta-objective. We unfortunately had to choose to remove a more general reference to the literature that was included in the first draft, in order to have the text fit within the page limit.
* We have added a new appendix ("C"), where the relation between schedulers and adaptive learning rate techniques is discussed.
Also, we have rephrased / corrected and clarified / extended our text where it was deemed necessary by the reviewers (see specific answers for details).
* We have added a new appendix subsection (D.6) with a quantitative analysis of ImageNet convergence, reporting epochs of attained accuracy thresholds for different optimizers.
* Lastly, we have improved our legends and figures as much as was possible.

 I do want to emphasize on this point of generalizability of your method. Trainable LR is a good idea as long as it meets some Generalization threshold. I suggest you set some threshold (Acceptable one going by the current SOTA) and then demonstrate the robustness of your method by reporting the number of epochs to converge.

 Thank you very much for your insight on our work. We sincerely appreciate the reviewer’s take on the proposed contributions, which helped us improve the readability in some parts of the paper (including comments regarding previous works highlighted by the reviewer).

 We thank the reviewer for the feedback and for pointing out the strengths of our work. Next, we address the concerns of the reviewer.

**Related Work:** The reviewer correctly points out the existence of several closely related works, which (to the extent of our knowledge) are referred to in the manuscript but are not discussed adequately due to space limitations. However, we do understand the importance of this, and as a result, we have significantly rewritten a portion of the Related Work section, with more comments on works closely related to ours. Note that the closely related work of Baydin et al. and its practical considerations are already extensively discussed in the experimental section as an important comparison.

**Complex Architectures:** The presented analysis is on a per-layer basis and $\hat{y}$ is indeed an intermediate auxiliary result which can be easily extended in a whole complex network architecture. We have added a more concise analysis for the global learning rate. The two approaches are equivalent, and are simply two different realizations of the chain rule. Nonetheless, we understand a source of confusion in our presentation of the proposed analysis (before Eq. 4) by using a simple feedforward network as a simplification reference and not explicitly and clearly stating its extension to complex modules and thus we have carefully rewritten this part. Note that the majority of the experiments in the main manuscript involve ResNet-based architectures, providing evidence for the suitability of the proposed method for such complex architectures (appendix includes an extra variety of complex architectures, such as DenseNet and Transformers).  

**Figures:** We thank the reviewer for pointing out inconsistencies in legends and the main text. We tried to detect and correct any inconsistencies and improve the quality of the figures. Concerning Fig.1 in particular, we are aware that the information is convoluted and no safe conclusion can be drawn about which setting performs better. However, that is exactly the point for Fig. 1: the proposed method, under different settings, can provide similar behavior after a few steps. 

**Bias term on the footnote of page 4:** This is a matter of convention and both options are valid. We have selected this notation because it assumes both input (x) and output (y) vectors are augmented (as frequently used for expressing affine transformations on homogeneous coordinates). However, if the reviewer deems necessary this change for improved readability we will fulfill it.


 We thank the reviewer for correctly pointing out that the impact of the second order derivative is not highlighted in our paper (overcoming the necessity for the impractical computation of a Hessian over the weights). We explicitly state this in the introduction of the updated manuscript, as the reviewer suggested.

**On the usage of Transformers:** In our experimental validation of our method, we focused on classification (and detection) problems that are very popular for validating modern optimizer algorithms. Nevertheless, we have also tested a Seq2Seq transformer model for machine translation in the appendices. The motivation of our decision is that modern transformer architectures used in NLP problems are generally addressed by Adam-like optimizers which are well-behaved and designed for sparse updates. The proposed method seems to be compatible with an Adam formulation but this direction is not yet explored and thus a more NLP-focused experimentation would be favored in such a future work. 

**TLR reactiveness to initial learning rate:**  It is not that straightforward to infer that our method is reactive to low/high initial learning rates. Regardless of the initial value of the learning rate, experimental results  indicate that the learning rate increases (for fast convergence to the proximity of an optima) and then decreases (theoretically this cycle of increase/decrease can be repeated). Nevertheless, the duration and the intensity of the increase is indeed controlled by the initial learning rate. A theoretical analysis of a prospective approach bound could be an interesting direction for future work (especially if it can hint towards convergence guarantees).

**On “scenarios where learning rate hasn't been optimized”:** The main goal of our work is to provide such a method which does not require tuning and even though we focused on widely used settings, we assume no a priori knowledge about them when applying our method in order to support our claim. Also, the per-layer alternative indicates such a desired behavior in a more complex and imbalanced setting, when part of the model is pre-trained. 


 **Connection to related works and advantages:**  The reviewer correctly points out that the high-level concept of using consecutive gradients is not new, as we have already stated several times throughout the manuscript (as well as in appendix) with many references to prior works. In the updated manuscript, a more detailed description of this line of research is added in the Related Work.
Nonetheless, we respectfully disagree with the reviewer’s comments: 

*“So in a sense, the paper has nothing new to offer. The only minor change that the current submission considers is a second-order update on the learning rate, which I do not expect to have a major impact...”
 and 
“I believe the current submission misses multiple related work and has not much to offer in terms of novelty or addressing the issues associated with the gradient-based adaptive learning rate methods”.*

Specifically, we:
* adopt a NN-based viewpoint of the problem which effortlessly results to the per-layer variation under a mathematical framework which can produce “optimal” learning rate updates
* most importantly, derive an analytical formulation for the second order gradient w.r.t. learning rate, which cannot be treated as trivial, since it theoretically omits the necessity for setting a hyper-parameter which controls the learning rate whilst requiring trivial computational overhead (e.g. no Hessian computation w.r.t. the weights is required). 
* include extensive experimental results that support our theoretical results and claims over a variety of settings, where the proposed method displays a notable robustness to initial learning rate as well as to batch size, a property which can not be found to similar adaptive optimizers. Interestingly enough, the conceptually similar work of Baydin et al. 2018 (which shares a similar formulation up to the first derivative) seems to be sensitive to both the extra meta learning rate parameter and the batch size (see Fig. 2), highlighting the importance of the proposed method. 

**Short-horizon bias:**  Thank you for the reference to the very interesting work of Wu et al., also pointed out by Reviewer jSQ1. We included a reference to the issues it raises in the updated manuscript. Let us comment on a number of points concerning the relation of this work to ours. In Wu et al. 2018, we read 
*“because gradient-based meta-optimization can require thousands of updates, each of which unrolls the entire base-level optimization procedure, the meta-optimization is thousands of times more expensive than the base-level optimization”*. 
Our method is only more expensive than vanilla GD by a constant factor, and there is no "unrolling" procedure involved. Hence, according to their nomenclature, TLR should be considered a greedy schedule, and therefore susceptible to the short-horizon bias. We must agree that any meta-optimization scheme is susceptible to pitfalls akin to those involving vanilla optimization, and most notably those concerning getting stuck to local minima. Since computational resources are never infinite, any (practical) algorithm should then be closer to problems related to short-horizon, and our method is not excepted from this rule. Concerning the experimental results in Wu et al., we believe that the tested online Stochastic Meta-Descent algorithm is closest to the “Naive” TLR algorithm we proposed. That is in the sense that, the “metastep” of SMD is akin to step 5 of our naive algorithm, where a GD step is run on a meta-objective, over a meta-learning rate η. We acknowledge the differences between the two algorithms, and we understand that *this specific version* of SMD (cf. to Schraudolph, 1999) was chosen as a meta-optimization baseline. However, we argue that the proposed efficient variant does move away from this paradigm, in that learning rate is no longer a simple GD step, but updated using the two-point / consecutive first-order gradient and second-order gradient analysis that we present in our paper. Also, we made a case for the effective independence of the efficient algorithm to any hyperparameters, so η is not directly relatable to any variable in the efficient algorithm. 

In the framework set by Wu et al., we may interpret our approach as aiming to encode information about the loss curvature / surface in its update, in an effort to “negate” the short horizon-related problems. In our experiments, we believe that we make a good effort to show that the efficient GD-TLR is more efficient w.r.t. related methods (that should be affected by the same bias, up to a lesser or greater extent). In particular, our method shows better generalization performance compared to other methods and over various settings. Note also that we do not claim that the learning rate sequence we obtain with GD-TLR is globally optimal. It will practically only be locally optimal (cf. our note in paragraph “Final Remark” in section 5), a point that is directly relevant to the “short-horizon” problem.


 **Per-layer alternative and line-search:** Line-search algorithms, our original motivation for the proposed approach, cannot have a per-layer adaptive step size without significant computational overhead. The highlighted comment refers to this practical restriction of line-search algorithms, since a forward pass should be performed for every different layer learning rate according to their mathematical framework. We understand that without the appropriate context it may be misleading and we removed it from the main text (it was added in the Appendix section D.2 Granularity of Control). However, we do not claim that we are the first to introduce per-layer LRs. In fact, Sec. D.2 contains references to other per-layer approaches. Moreover, we are not sure what the reviewer means by “Can’t you always just rescale the updates for different layers?”. If the reviewer refers to setting a different learning rate for every layer as rescaling, deciding a policy for rescaling is not trivial, whilst our method provides an optimal way to achieve this under the proposed framework. 

**Storage Requirements:** the reviewer makes a valid point about storage requirements of our method (a common issue for several modern optimizers). The low-precision storage is a very interesting approach and we would like to investigate further into this in future work. We thank the reviewer for the suggestion. Nonetheless, we consider it as an out-of-scope consideration of the current version of our work. 

**Initial LR for Figure 3:** Figure 3 serves as a sensitivity indicator for the batch size selection and we consider it as an important aspect. In the SoA comparison figures, we either considered adaptive techniques which can automatically adjust the LRs, just like the proposed method, or we selected a good initial LR in order to achieve SoA performance by following empirical evidence of the existing literature. Therefore, by no means we have made poorly choices of initial LRs in the reported comparisons. 

**Short-horizon bias:**  Thank you for the reference to the very interesting work of Wu et al., also pointed out by Reviewer jSQ1. We have already hinted towards a similar problem and the reference is added in the updated manuscript. We further discuss this work in our answer to the comments to reviewer hkZ3.


The authors propose an adaptive learning rate strategy that relies on treating the learning rate as a learnable parameter. The learning rate is updated using first- and second-order updates wrt the loss, evaluated after applying a step on the weights. The results are validated on common datasets and architectures such as CIFAR-10/100 and ImageNet. The paper at the current stage is merely a repeat of a number of known approaches. Namely, Amid et al., 2020 already propose a very similar approach based on the inner-product of the current and past gradient to update the per-layer learning rate. Amid et al., 2020 extend this to coordinate-wise learning rates and use exponentiated gradient updates, which is better suited than gradient descent for updating non-negative values. They even consider adding momentum, EMA of the past gradients, normalized updates, etc. Methods that rely on updating the learning rate based on the interaction of the consecutive gradients date back to the Delta-Bar-Delta (DBD) method (Jacobs, 1988) and have been repeatedly proposed in different forms (Minai & Williams, 1990; Sutton, 1995; Schraudolph, 1999; Baydin et al., 2018). So in a sense, the paper has nothing new to offer. The only minor change that the current submission considers is a second-order update on the learning rate, which I do not expect to have a major impact, as discussed next.

Another important point missed by the authors is the problem of short-horizon bias associated with consecutive gradient-based adaptive learning rate methods. Specifically, Wu et al., 2018 show that all such approaches suffer from the effect of a short-time horizon where the method punishes for the variability in the loss surface (due to stochasticity and other factors) in the early stages of optimization. Thus, these methods usually do not allow a large enough learning rate initially to escape bad local minima. The paper does not acknowledge this problem nor provides any improvements over the earlier approaches to address this issue.



Minai, A. A. and Williams, R. D. Back-propagation heuristics: a study of the extended Delta-Bar-Delta algorithm. In 1990 IJCNN International Joint Conference on Neural
Networks, pp. 595–600 vol.1, 1990.  

Schraudolph, N. N. Local gain adaptation in stochastic gradient descent. In 1999 Ninth International Conference on Artificial Neural Networks ICANN 99. (Conf. Publ. No. 470), volume 2, pp. 569–574 vol.2, 1999.  

Baydin, A. G., Cornish, R., Martínez-Rubio, D., Schmidt, M., and Wood, F. Online learning rate adaptation with hypergradient descent. In 6th International Conference on Learning Representations, ICLR, 2018.  

Ehsan Amid, Rohan Anil, Christopher Fifty, Manfred K. Warmuth. Step-size Adaptation Using Exponentiated Gradient Updates. In Workshop on "Beyond first-order methods in ML systems" at the 37th International Conference on Machine Learning (ICML), 2020.  

Yuhuai Wu, Mengye Ren, Renjie Liao, Roger Grosse. Understanding Short-Horizon Bias in Stochastic Meta-Optimization. In ICLR, 2018.  


 Overall, I believe the current submission misses multiple related work and has not much to offer in terms of novelty or addressing the issues associated with the gradient-based adaptive learning rate methods. Therefore, I vote for a reject.  

Update: I forgot to add pointers to the missing references that I mentioned in my initial review. Apologies for that.

In this paper the authors derive updates to the learning rate (LR) that minimize the loss function that can be computed as functions of neural network gradients. These updates to the LR can be used to compute an adaptive LR schedule in an online, hyperparameter-free (or at least, less hyperparameter sensitive) fashion. They derive first and second order updates for the LR and then run experiments with it on various deep learning workloads. *Strengths:*
- Your Figures 1, 2 are great in that they actually showed what the computed LR schedule looked like.
- The time requirements in Section 5 are very useful; it’s great to know what the real-life overhead one should expect from different methods!
- Paper is very well written and formatted.

*Weaknesses:*

Overall I believe most of my issues stem from lack of comparisons to baselines with nontrivial LR schedules. The comparisons to other LR schedules in Appendix D are nice, and the authors should plot the baseline LR schedules next to the proposed LR schedules (such as in Figures 1, 2), and do so in the main text. The authors do not seem to include any LR schedules that include a warmup in their baselines, which is surprising given LR warmup is often common on the workloads considered in the experiments. Most of the computed TLR schedules look like a linear warmup followed by exponential or quadratic decay, so I would recommend trying this sort of schedule as a baseline, even just with default hyperparameter choices (warmup for 10% of training?). Figure 3 seems to imply that once the LR of the stepwise baseline schedule is dropped halfway through training, the baseline outperforms the proposed method. This to me indicates that by trying even a few more baselines LR schedules, the proposed method may not be competitive. While there is definitely appeal in the proposed method because it mostly does not require tuning, many heuristics used today do not actually require much guess/check work either beyond choosing the initial LR (which all proposed methods must do), that is they work “out of the box” (such as warmups or other popular decay schedules) or with very minimal tuning. As another example of this concern, it is surprising in Figure 3 that Adam does worse than SGD with momentum, given that on CIFAR WRN Adam almost always outperforms other optimizers; what LR schedule was used for Adam? How does Adam perform when using the same LR schedule as sgd_mstep? It seems unfair to compare to any optimizer with a knowingly poor LR schedule, especially given that commonly used ones (sgd_mstep) seem to outperform the proposed method.

Additionally, I struggle to determine which methods are better from many figures, because the subplots are too small/crowded. The authors could dramatically reduce the vertical axis on many such plots to paint a better picture. For example in Figure 4 it is unclear how the optimizers perform towards the end of training, because of the vertical axes (what were the final accuracies achieved for each optimizer?).

With any paper that introduces a training algorithm technique, I think it is critical to highlight if the goal of the proposed technique is to 1. improve training speed 2. improve generalization performance (or both). If any part of the goal is to improve training speed, then I believe that a better method of comparing performance is to instead report the number of steps (or total wallclock time) required for each technique to reach a competitive validation performance (for example, how many steps did it take for each optimizer to reach a competitive ImageNet validation accuracy (75.9%)?). If the proposed technique is supposed to improve generalization performance, then the baselines should probably include additional regularization techniques.

Some other concerns:
- “We should note that the per-layer rationale cannot be supported by backtracking
line-search algorithms, which use single-valued learning rates.” I may not understand the point of this text, but why can’t other algorithms use per-layer LRs? Can’t you always just rescale the updates for different layers?
- Storing an extra copy of gradients is actually nontrivial for a lot of larger models, making the storage cost of SGD with momentum similar to that of Adam. In these memory constrained cases, others have tried storing gradient statistics in lower precision to save space. While definitely not necessary, storing the older gradients in low precision could be an interesting additional experiment.
- In Figure 3, it would be much more interesting to see different LRs instead of batch sizes (you have plots of the losses on CIFAR100 in the appendix for different LRs, but as we saw in other figures the loss does not tell the whole story in the case of overfitting), because it seems like an unfair comparison to use an untuned LR for the baseline methods (each baseline only gets a single initial LR) when the proposed method can change its LR online to correct for a poor choice of initial LR.
- As a minor point, I believe citing this paper https://openreview.net/forum?id=H1MczcgR- would be useful, as it surfaces concerns with learning the LR. Overall, while the paper is very well written and formatted, I do not believe I can recommend this paper for acceptance without seeing better comparisons to more competitive baseline LR schedules (such as warm up followed by an exponential or quadratic decay, even with all the hyperparameters set to default values). The figures could use some cleanup as described above.

However, *I do believe that the proposed method could be of significant value, but I would want to see it demonstrated on workloads where common LR schedules/heuristics do not hold up*; could the authors find or design an experiment where the dataset/model is too difficult to optimize with traditional LR schedules, but the proposed method succeeds? Perhaps poorly conditioned neural nets (such as removing normalization or skip connections) could be of interest here.

In this paper, the authors propose a novel adaptive learning rate method where the learning rate is learned along with the weights through gradient descent. The main intuition is that the learning rate is part of the loss function (along with the weight gradients) and this combined loss is minimized through gradient descent (naive GD-TLR).  This method can be computationally expensive due to the fact 2 versions of the loss must be computed at each layer.  To mitigate this, the authors propose a variant of the method that updates the learning rate based upon a single loss (efficient GD-TLR). The efficient GD-TLR has the intuition that if 2 consecutive losses are in the same direction, the learning rate is increased (move to goal faster).  If the 2 consecutive losses are opposite direction, decrease the learning rate (bouncing around the well). In addition, this method can use different learning rates for different layers in the network. This paper provides a useful formulation of the problem. Posing the learning rate as a learnable parameter does appear to be well-founded theoretically, but the results are a bit lackluster for the networks that were tested.  Testing a transformer-based model would be an interesting data point. 
After the description of the technique, the authors do a good job of comparing it to existing techniques. However, I would like to a concise summary of how GD-TLR improves on the shortcomings of those techniques. The main advantage is decreased computational cost as there is no Hessian required and can be simplified into a constant-time operation on the normal backward pass.  A single sentence making this explicit would be helpful for the reader.
For the simplified variant, efficient GD-TLR, the authors indicate that the technique is reactive to either too high or too low of a learning rate. This should have been the first hint that it will at best perform as well as a prospective technique that considers a known gradient profile. The authors should relate this in the intuition under equation 6 to the expected performance.
The experimental portion was a good representation of networks for computer vision classification.  But GD-TLR is likely most applicable to other problems where there might be more of an advantage. The authors' final comments do indicate that they believe an important result is that GD-TLR can work in scenarios where learning rate hasn't been optimized much.  It would be nice for the authors to actually demonstrate this rather than simply stating it.  I suspect this is true, especially for large transformer models, but the paper provides no evidence to this end. Overall a good paper with an interesting technique. Showing that the technique really can work in scenarios where learning rate hasn't been optimized could make this a very important result.  As written, there is much work to prove this out, so the idea remains a curious hypothesis.

The paper introduces a new algorithm to automatically adjust the learning rate. More specifically, the learning rate is optimized via the extra gradient descent step during the training. The paper first discusses the role of learning rate and formalizes the problem within a gradient descent framework. Then based on that, the authors give an expression of both the first and second derivatives of the learning rate where the neural network structure is taken into consideration. That further leads to a cost-effective implementation to update the learning rate. They also show that such implementation can be extended to the case where a different learning rate is applied for each layer. Pros:
* The paper is well-written and easy to follow. The logic structure of the paper is clear and necessary explanation is given. e.g. the intuitive interpretation of derived gradients is friendly to a reader like me.
* The idea of incorporating the second-order gradient is novel and interesting. Most hypergradient descent methods require to introduce several additional hyper-paramters and carefully fine-tune them, but with a Newton-Raphson formulation, the update of learning rate become more concise.
* The derivation of the algorithm is clear and the method is easy to implement. Furthermore, the per-layer learning rate scheme seems a natural extension to the original algorithm where a global learning rate is applied to all layers.
* Both the experiment setup and the empirical results look good. The new algorithm outperforms the state-of-art methods in most cases.

Cons:
* The trainable learning rate is not a fancy idea. There are lots of papers about hypergradient descent which consider hyper-paramters (including the learning rate) to be trainable. However, I have not seen a decent discussion about the connection between the proposed method and existing work in that direction.
* The expression of derivatives of the learning rate with standard neural networks may not be applicable to more complex architectures (e.g. ResNet). Furthermore, I somehow feel $\hat{y}$ is used as more like an intermediate variable so I am not sure if it is necessary to formulate the problem in this way. [1] has a similar conclusion for calculating the gradient of the learning rate. Is there any specific reason not to cite their conclusion?
* I found the figures from the experiment sections extremely hard to read. For example, in Figure 1 the curves overlap with each other during training and I could not distinguish any one of them. If possible, please provide a table so the results will be much more readable. Also, some names of the legends are weird and not consistent with the ones in the main context. 

Comments:
* On page 4 there is a footnote explaining how to formulate the layer with bias term. I might miss something, but shouldn't $W = (W\ b)$ instead?

[1] Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. In International Conference on Learning Representations, 2018.  Overall this is a good paper and I tend to vote for acceptance. However, I do hope that the authors could provide some feedbacks on the reviews I made and polish the paper more in details.

The authors proposed an approach to optimize the learning rate of deep learning methods during gradient descent training. The approach uses first and second-order gradients concerning learning rate as functions of consecutive weight gradients. The gradient-based method suggests no manual tuning is required and considers the learning rate itself as a learnable parameter. The authors also investigate the use of the approach per layer. Finally, the authors studied the method’s performance considering variations in the initial learning rate and the batch size. The paper presents a set of experiments using MNIST, CIFAR10/CIFAR100, and ImageNet. Results consider MLP for MNIST, Wide-ResNet for CIFAR10/CIFAR100, and ResNet50 on Imagenet experiments.  The paper presents an interesting approach with exciting results, quantitative and analysis. The ideas of the paper are promising and have the potential to open discussion to a different direction of how to adjust the learning rate on iterative learning algorithms. I have the following considerations and suggestions on how this paper can be improved:
- The authors build the method in the context of gradient descent for optimizing a neural network loss function, and I’m wondering about the impact on different loss functions. 
- The paper suggests robustness to user-defined hyper-parameters such as batch size and initial learning rate, but no consideration is made about the number of epochs.  Considering the comparison with literature approaches in a long-term training regime the methods tend to be equivalent at some point of the optimization. The authors need to present some comments on this direction. If the authors consider fast convergence as a feature of the method and suggest ignoring the number of epochs, experiments and comments in this direction are needed.
- Besides the considerations about the impact of the user-defined hyperparameter p, which controls the frequency of learning rate updates. It is unclear if the suggested 0.33 value is valid for different datasets, considering small/complex/sparse or other datasets characteristics.
- Are the experiments considering one execution or different runs of training are performed? It is essential to introduce variance to confirm the experiment’s conclusions. Are all the methods starting from the same initial weights? Hyperparameters description? The experimental methodology description needs to be improved; some comments are made on the supplementary material but must be addressed in the main text.
- Are the models trained from scratch? What is the performance considering transfer learning?
- Improve the figure’s caption driving the reader to what needs to be understood from the figures. Please insert the axes description on the figure.
- When considering the Time Requirements experiments, why are only experiments with CIFAR100 and the WRNET_16_4 architecture presented?
- Did the authors identify any phase shift regime during the training process?
- Some experiments suggest the proposed method with better performance at larger batch sizes. What is the impact of using bs minor than 64?
- Could the method generate overfitting on small datasets?
- What is the statistical relevance of the presented results considering statistical tests?
- There has been little discussion on the limitations of the proposed approach. What are the possible technical limitations?
 The paper presents an interesting approach with promising results, quantitative and analysis. However, the authors need to provide more comments and address some concerns about the experimental methodology and experiments before publication.

An intuitive overview of the fundamental ideas:
• The fundamental problem that this work is concerned with is finding a suitable, “optimal”, way to adaptively set the learning rate for the
Neural Net setting (whose fundamental challenge, as the paper points out, is it’s ’over-parametrization’)
• The approach adopted in the paper is the proposal of a framework (WITHIN the existing GD-oriented ways in which we train our NN’s),
wherein we consider the Learning Rate as a learnable parameter 
• The real novelty of this approach shows itself in how elegantly the mathematical formalism that follows from this approach, works out,
 I try to briefly cover the interpretation of the obtained mathematical results in the paper below:
– Since we are trying to treat α as a learnable parameter, and want to treat it on the same footing as any other NN parameter, this
would mean that we ought to write a GD-update for it (which would involve the gradient of an augmented loss, say: Lα(αt,wt)
wrt the learning rate, which we would have to define), and would also involve an auxiliary parameter (say ηt) as a sort of meta
LR for α – this is where a straightforward pursuit of the LR as a learnable parameter would lead to. So far, the novelty has
been in the definition of the augmented loss, which was formed by combining the NN’s loss with the GD update for the weights that
it was a function of (namely: Lα(α;w) = L(w − α.∇L(w)).
– The real strength of the above approach, shines through when the relevant gradients are actually computed, namely the quantity:
∂Lα/∂α |α=αt , for which we end up with the really powerful expression: −⟨gt, gt−1⟩, which, in addition to being an extraordinarily
compact, for such a general gradient computation (probably giving us a nudge that our initial definition of the augmented loss
L was good) also affords to us a really intuitive (but robust and complete) qualitative interpretation of how we should update our
LR per iteration, the reason this expression is so significant, is because so far, we haven’t even considered the GD-styled update for
the learning rate, and already, the expression for this newly defined gradient (computing which is O(1), since the dimensionality
of our spaces are fixed when training) provides us with enough information to be able to make better LR-related decisions per iteration than using a constant LR/some weighting method to reduce our LR with progressing EPOCHS (either of which might
be schemes that do not accurately reflect the topology of the loss landscape)
– After the above gradient computation, we look towards the second order gradient – both in hopes of obtaining a similarly elegant
mathematical expression for the same and also with the aim that we might be able to leverage second order information to devise a
stronger update rule for the LR, and as it so happens, we end up getting a really elegant expression for the second-order derivative
as well, namely: ∂2L/∂2α |α=αt = 4/αt⟨gt, gt − gt−1⟩ = 4/αt(||gt||2 −⟨gt, gt−1⟩), which can also be computed along with the above
first order derivative in O(1) time.
– The fact that we have effectively O(1) access to relevant second order information for the framework that we have built for the
LR, is very significant, making available all the power associated with second order methods, for basically free!
– The next step would be to come up with a proper update rule utilizing the above information that we have derived about the
LR’s GD structure – since we have access to both first and second order information in this case, an obvious place to look for
motivation for the functional form of the update, would be the Newton direction, which essentially, allows for the incorporation
of second order information in an ’optimal’ manner (the Newton direction is a good direction move in when: 1) The function
is well-approximated by it’s second order Taylor expansion and 2) The Hessian for the function is positive-definite) – also observe,
that in a sense, using Line-search methods with the Newton direction, is on a conceptual level (differing in the underlying
details, like the requirement of positive-definiteness in LS, which may be relaxed in Trust regions) incredibly similar to using
Trust regions (where we use the quadratic approximation model: mk(p) = fk + pT + ∇fk + 1 2pTBkp and take Bk = ∇2fk).
The above strategy is precisely what is adopted in the paper, the Newton direction is given by: pNk
= −(∇2fk)−1∇fk, which may be re-created in the Naïve update rule: αt+1 = αt−η.∇Lα(αt;wt)  by setting η to −(∇2 αLα)−1 – in addition to this, the term is slightly altered to ensure numerical stability. 
A comparison with surrounding literature: The two specific papers that I am interested in talking about (wrt the current work), are: item 1 and item 3, since they are similar in spirit to the current work (in that, they also try to suggest ’optimal’, formalised methods for
adaptively setting the LR) and have both been published in good conferences (the former in NeuriPS,19 and the latter in ICLR,18)
2.1 Painless stochastic gradient: Interpolation, line-search, and convergence rates: This paper tries to solve fundamentally the same problem – i.e. it tries to find an ’optimal’ way to adaptively compute step size during training, though the approach adopted in their paper is fundamentally different from the current work, but with some interesting takeaways that could’ve influenced
the current work. I first summarize their approach:
The fundamental idea of the paper is to use line search on each iteration for computing the step-size, specifically, Armijo LS, which chooses it’s step size based off of the solution to the following optimization problem (the following is the expression suggested in the paper for the adaption of the problem to the stochastic setting). fik(wk − ηk∇fik(wk)) ≤ fik(wk) − c.||∇fik(wk)||2
What follow are proofs of convergence bounds for this method in the case of strongly convex and convex functions under the assumption of the validity of the interpolation condition for the loss function and data of our NN (namely, if f(w) = 1nPn i=1 fi(w), then in the case of an optimum of f, w∗, i.e. ∇f(w∗) = 0, we also have: ∇fi(w∗) = 0∀i ∈ [n])
• From the above, the major takeaway for the current work, was definitely the intuition to define our augmented loss in the general Line-Search setting (though of course, what the current work does is far from frame and solve a classical LS problem, since we straight off assume the direction of steepest descent, but clearly, the form of our augmented loss is the same as the general form of the LS optimization problem, namely: minα>0f(xk + αpk))
• It would be interesting to see to which extent the mathematical framework of the current work could’ve been adapted to agree with the
theory of item 1. Since the starting point of both agree to a certain extent.
• A clear strength of the current work over item 1 is the lack of requirement of the interpolation condition for results to hold good, and the
significantly easier implementation of the suggested methods 
2.2 Online learning rate adaptation with hypergradient descent: In terms of the fundamental structure of the work, the current paper is
incredibly similar to item 3. They also arrive at the same expression for the gradient, but that is where they decide to stop (and instead, focus their time towards the general formulation of the same update). Their way of formulating the update, where they didn’t explicitly define an augmented loss, instead they implicitly worked with ∂f(θt−1)/∂α , which at the end of the day is fundamentally the same thing as ∂Lα/∂α , but the more general formulation of the latter, (and a vested interest in second-order optimization!) is probably what led to the formulation of the second order update in the current work.
References:
1. Sharan Vaswani, Aaron Mishkin, Issam Laradji, Mark Schmidt, Gauthier Gidel, and Simon Lacoste-Julien. Painless stochastic gradient:
Interpolation, line-search, and convergence rates. arXiv preprint arXiv:1905.09997, 2019.
2. Rohan Mohapatra, Snehanshu Saha, Carlos A. Coello Coello, Anwesh Bhattacharya, Soma S. Dhavala and Sriparna Saha; AdaSwarm: Augmenting gradient-based optimizers in Deep Learning with Swarm Intelligence, the IEEE Transactions on Emerging Topics in Computational Intelligence; DOI: 10.1109/TETCI.2021.3083428.
3. Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. In International Conference on Learning Representations, 2018.
4. Pang Wei Koh and Percy Liang. Understanding Black-box Predictions via Influence Functions. In Internal Conference on Machine Learning, 2017.8
5. Jorge Nocedal and Stephen Wright, Numerical Optimization, 2nd edition, Springer Series in Operation Research and Financial Engineering, 2006 The strength of the framework put forward by the paper can be described in two points: 1) The ’optimality’ of the above suggested framework
is intuitively obvious to anyone who understands basic NN optimization (since we are treating the problem of optimizing the LR, just
as we would any other parameter of the NN) and 2) The mathematical objects required for the same (the gradient and the Hessian) turn out
to have extremely elegant closed form expressions which require zero computational overhead to implement. Now, notice that points 1 and
2 are talking about fundamentally distinct ideas, 2 is referencing the chain of mathematical argument, from the definition of the Augmented Loss (Lα), to the derivation of the expressions for the Gradient and the Hessian, while 1, references HOW we actually exploit the (first
and second order) information obtained in 2, in an ’optimal’ manner – hence, one front on which an extension may be made is in our method of choice for step 1(hence, that opens up us up to, in theory, choosing from all the rich literature on second order method) – but, sadly this extension won’t be as easy as swapping out using a Newton step with some other second order method, since we further observe that even though 1 and 2 are distinct ideas, they are intimately coupled together by the definition of the augmented loss, which in turn was directly motivated by the form of the standard GD update (and also by the standard form of the Line-Search optimization problem, namely: minα>0 f(xk+αpk), where we had already made the decision to fit this work to directions of steepest descent i.e. pk = ∇fk), and hence in a way, brings all of this full circle in this being an isolated framework.
There aren’t any ’obvious’ ways to extend the fundamental theoretical framework of the paper, and any kind of work that takes motivation from the structure of this one, would have to start with first choosing the broader framework to embed this optimization of the LR in (in this case, it was GD wrt the NN loss)

Meta-review: The work presents a modification to existing approaches of automatic learning rate adaptation (called TLR) via a second order approximation of the function mapping step size to the change in loss when optimized with SGD. This was easily the most controversial paper in the AC's stack, with 4 reviewers advocating accept and 2 reviewers strongly arguing for reject. The authors also went through considerable effort to address reviewer and AC concerns and uploaded multiple additional experiments and ablations to support the robustness and efficacy of the proposed method. Despite a long discussion and rebuttal period, reviewers were unable to reach a consensus. There were several different aspects of the work whose merits were thoroughly debated during the rebuttal period.

The first aspect regarded what the exact contribution of the work was. Initial reviewers who were very high on the work believed that the entire derivation from equation (1) to equation (9) was novel. However, as other reviewers correctly pointed out (1) to (7) is a standard derivation of adaptive learning rates and has appeared in several prior works. Instead it is primarily equation (9) that is the contribution. Given that multiple reviewers initially believed that (1 -> 7) was a novel contribution, I feel it is safe to say that the authors do not adequately discuss their contributions with respect to prior work. However, all reviewers in the end agreed that equation (9) is novel and potentially interesting (though some remain skeptical of it's utility).

The second topic of debate regarded the short horizon bias raised by reviewer hkZ3. The short horizon bias presents a fundamental barrier to meta optimization of the learning rate. To summarize, greedily selecting the step size to minimize the loss will result in the optimizer taking too small of steps in the flat directions of an ill conditioned loss surface. This results in faster training in the short term but slower training in the long run. The presented method seeks to greedily optimize the loss over short time scales and thus will be subject to the short horizon bias. The initial draft of the work did not include any discussion of this prior work. During the rebuttal, the authors initially argued that their method can help mitigate the short horizon bias before later concluding that it is a limitation of the method. There was debate between the AC and reviewers regarding whether or not existing methods of adaptive learning rate schedules were already at a fundamental barrier presented by the short horizon bias. One reviewer even mentioned that in their own research they have abandoned the general approach of adaptive learning rates because they cannot overcome this issue. This debate was never resolved, it's plausible to the AC that there is room for increasing the robustness of existing approaches while not addressing the short horizon issue. It is the AC's opinion however that the work would be significantly strengthened with experiments directly addressing the short horizon issue.

The final item of debate regarded the strength of the considered baselines. The authors claim that the second order term in (eq 9) largely removes the need for tuning relative to Baydin et. al. and that the method outperforms multiple baselines across multiple workloads, including Adam (Kingma et. al.), SLS (Vaswani et. al.), and SPS (Loizou et. al.). Indeed multiple plots are given showing that the authors have found a configuration of their method that consistently outperforms certain fixed configurations of the considered baselines. Furthermore, ablations are presented which suggest that indeed it is the addition of equation (9) that is responsible for this strong performance. Despite all of this presented evidence some reviewers remained skeptical, and believed that they could produce a different but fixed configuration of say Baydin et. al. (or even Adam) which matched the proposed method on all of the considered workloads. There are compelling reasons for reviewers to consider the presented experiments with skepticism. Indeed the deep learning optimization literature has for years struggled to make progress despite publishing hundreds of papers---see for example the results of [1] which perform an independent comparison of 100's of published methods and found that none convincingly outperform Adam. Given this, it is clear that the current standard for evaluating optimizers in the literature is inadequate if we are to reliably make progress.

To give a more relevant example of the difficulty of comparing optimization methods, suppose for the sake of argument that we were not evaluating the efficacy of TLR but instead the method of Vaswani et. al. (SLS). Vaswani et. al. makes many similar claims as the proposed method, namely the method consistently outperforms Adam across multiple workloads and enjoys a similar robustness to hyperparameters (e.g. their Figure 6). However, in Figure 3 and 4 of this work we see SLS no longer outperforms Adam on the considered workloads. If Vaswani et. al. had argued for acceptance based on the author's Figure 3 and 4, I don't think any reviewer would have recommended acceptance. This begs the obvious question, why does SLS consistently outperform Adam in the experiments run by Vaswani et. al., but not in the experiments run in the considered paper? There are at least two possible answers here, both of which are concerning. Either Vaswani et. al. is yet another method that generally doesn't outperform Adam or in comparing SLS with TLR in this work the authors did not properly tune SLS in their baselines. Furthermore, what is going to happen if future work tries to compare against TLR? Will TLR still look better than Adam or will independent review find that TLR is yet another method that on average performs about as well as Adam? As a reviewer trying to compare the two papers I see very similar evidence given supporting the two methods and thus I am left with an unresolved contradiction.

Given all of this, I am forced to conclude that there is insufficient evidence presented in this work that the proposed method generally outperforms related methods such as SLS and Adam. A natural question thus is, what would have been sufficient evidence? Indeed the presented experiments seem to be about as convincing as what is shown in previous published methods such as SLS. In a sense the AC is also arguing that Vaswani et. al. presented insufficient evidence that SLS generally outperforms Adam (looking at Figure 3 and 4 perhaps SLS in fact isn't as useful as Vaswani et. al. claim). In looking at the experiments presented in this work, related prior works, and the 100's of methods considered in [1] a common recurring theme is that when comparing with prior work authors consistently run their own implementation of baselines on workloads of their choosing, rather than directly comparing with published results. In doing so, this leaves open the question regarding whether or not the authors (perhaps inadvertently!) are only considering workloads and hyperparameter settings which favor their own method rather than giving a realistic assessment of the efficacy of their own methods relative to others. Thus, if the authors wish to argue that TLR generally outperforms SLS, a strong piece of evidence the authors could provide is to run TLR directly on the open sourced code provided by Vaswani et. al. Show the reviewer how TLR compares when added directly to (for example) Vaswani et. al. Figure 4. In doing so, the authors will have addressed any concerns reviewers may have about how well represented SLS is, as the authors will be comparing against SLS in a setting where there were actual incentives to make SLS look good.

1. Schmidt et. al. Descending in a crowded valley – Benchmarking Deep Learning Optimizers, https://arxiv.org/abs/2007.01547

Decision: Reject