Dynamic Learning Rate Scheduling based on Loss Changes Leads to Faster Convergence
Authors: Shreyas Subramanian, Bala Krishnamoorthy, Pranav Murthy


Reviewer paper summaries: This paper proposes a dynamic learning rate schedule GreedyLR that adaptively adjusts the step size based on the loss surface. Specifically, the step size is multiplied by a constant $1/F$ ($0 < F < 1$) if loss decreases, and reduced by the same constant if the loss increases. GreedyLR is designed under the principle that the "correct" loss-decreasing directions require larger step sizes to accelerate, while those "wrong" loss-increasing directions need smaller step sizes.

Extensive experimental results on CNNs, encoder-only models, and decoder-only models show that GreedyLR achieves better convergence behavior than common learning rate schedules, e.g. cosine schedules, in over 80% of experiments.

This paper introduces a novel method for dynamically update the learning rate during the training of deep neural networks.
Summary of strengths: The idea is extremely simple and the paper is presented in a way that is very easy for readers to follow. The algorithm is easy to implement in practice, and the experimental results (if applicable) look good.

It is known from the community that there is no single method that excels on every scenario (“no free lunch”). The authors attempted to break this rule by proposing a new method for update the learning rate that, in general, either perform very similarly of demonstrate superior performance when compared to other methods.
Summary of weaknesses: * **Quality**: Most of the experimental results are not available in this paper, where the authors refer readers to another paper (line 413-414), which may violate the anonymity policy.

* **Clarity**: The writing of the paper can be improved to make it more formal. And some missing citations and baselines can be added to compare with the proposed approach.

* **Significance**: The theoretical guarantee is weak, while the empirical significance is jeopardized by the introduction of too many hyperparameters. Also, there is a huge gap between the theory and the practical form of the algorithm.

* **Novelty**: The convergence proof is trivial, which contributes little technical novelty to the literature. The empirical idea is simple but similar to past works of Polyak step sizes [1]. I would recommend authors include that branch of literature and highlight the difference.

### Reference

[1] Hazan, Elad, and Sham Kakade. "Revisiting the Polyak step size." arXiv preprint arXiv:1905.00313 (2019).

1. Weak Novelty. It seems that most of the innovation described was already proposed in a previous work by the authors (Subramanian and Ganapathiraman, 2023). The only novelty of the current paper is the inclusion of more results (LLMs) and an added proof of convergence.
2. Weak Assumptions. The provided proof of convergence and the analytical form for defining F are based on assumptions that do not seem to hold true in general, as demonstrated by the results and acknowledged by the authors. Hence, it seems the whole work is based on a very fragile foundation.
3. Weak Results. For the “Small Model Results,” the results show that in the majority of cases (122), the proposed method does not make a significant difference (the yes* and no*). Only in 48 cases was there a clear improvement over previous methods. It would be better to define the cases where the proposed method presents some improvements.
Comments, suggestions and typos: ### Clarity
* [Clarity] line 73, typo: ee -> we
* [Clarity] line 76: redundant period "." at the end of the sentence.
* [Clarity] line 197: We understand "no free lunch", and the fact that coming up with..." -> We understand that there is "no free lunch", and a scheduler ...
* [Clarity] line 200-202, 241-243: the language is too informal
* [Clarity] line 210: the reference to the algorithm is missing
* [Clarity] line 264: $\bar{x}T$ -> $\bar{x}_T$
* [Clarity] line 265: better specify the Appendix section number.
* [Clarity] line 302-303: once again better specify the Appendix section number ;D
* [Clarity] line 385: better specify the GPU type: NVIDIA T4 (16G) x 16.
* [Clarity] Table 1: better specify dataset sizes.
* [Clarity] Appendix A.2: better improve the layout of figures here.
* [Clarity] Appendix A.4, Table 9: can replace "better/worse" with "+/-" ahead of the metric value.
* [Question, Clarity] Algorithm 3: "GreedyLR functions contd...". I am wondering what is "contd...".

### Quality
* [Quality] some missing citations:
  - MiniCPM [1], whose learning rate stays constant most of the time
  - Eigencurve [2], which determines learning rates based on architecture information
  - Literature related to Polyak step sizes, e.g. [3][4].

* [Quality] Missing citations for AdamP and LAMB.
* **[Quality] line 413-414: I am wondering if this violates the anonymity policy. Why would the authors refer readers to another paper for detailed experimental results? Putting those results in the appendix instead would be a much better choice.**
* [Quality] Table 2, 3, 4: The metric here is strange. I recommend authors compare their performance in terms of the standard metrics on those benchmarks, such as accuracy in CIFAR10/100. Also, add the detailed loss results for different baselines.
* [Quality] Table 1: better compare with other adaptive learning rate methods instead of fixed learning rate schedules only.
* [Quality] line 262, Theorem 4.1: inaccurate, the "bias" term converges in $O(1/T)$, but the "variance" term is non-convergent, making the overall bound non-convergent.
* [Quality] Table 5: More models and datasets are highly encouraged, such as LLaMA-3-8B/Qwen2-7B/Mixtral-8x7B with Alpaca/C4/Pile.
* [Question, Quality] line 227-240, 278-284: the step size is adjusted based on the batch-level loss, where the loss decrease/increase may be caused by the sample difference. I am wondering how robust the algorithm is under high-level noises, such as small batch-size settings.

### Significance
* [Significance] Algorithm 1, Theorem 4.1: There is a huge gap between the theory and practice. Algorithm 1 is based on SGD, where in common practices of LLMs, AdamW is normally adopted. Tricks in line 339-363 are also not considered in the proof. Since the authors have provided a reasonable explanation for this matter, I consider this to be acceptable and will focus more on the practical performance of the proposed optimizer.
* [Significance] Theorem 4.1: The theoretical guarantees here is weak. For smooth convex objectives, the convergence for SGD is normally $O(1/\sqrt{T})$ with decaying step sizes. I understand this is caused by the nature of Algorithm 1, where the step size is not guaranteed to always decrease. I am wondering if the authors are considering a variant of this algorithm to ensure better theoretical bounds.
* [Significance] line 349-363: Too many tunable hyperparameters make the optimizer less practical. The goal is to avoid learning schedule tuning, but introducing more hyperparameters just leads to the opposite of it.
* [Significance] Figure 1: The gain of the proposed method is not significant when compared with cosine schedules. I would suggest the authors adopt settings that can highlight the advantage of GreedyLR, such as pretraining settings, and focus on a specific practical gain like acceleration.

### Novelty
* [Novelty] The technical contribution in Theorem 4.1's proof is not very high, which is a trivial extension of SGD in smooth convex objectives.
* [Novelty] Better highlight the difference between GreedyLR and Polyak step sizes [3].

### Reference
[1] Hu, Shengding, et al. "Minicpm: Unveiling the potential of small language models with scalable training strategies." arXiv preprint arXiv:2404.06395 (2024).

[2] Pan, Rui, Haishan Ye, and Tong Zhang. "Eigencurve: Optimal learning rate schedule for sgd on quadratic objectives with skewed hessian spectrums." arXiv preprint arXiv:2110.14109 (2021).

[3] Hazan, Elad, and Sham Kakade. "Revisiting the Polyak step size." arXiv preprint arXiv:1905.00313 (2019).

[4] Loizou, Nicolas, et al. "Stochastic polyak step-size for sgd: An adaptive learning rate for fast convergence." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

There is a typo in the Introduction: "In our recent work, ee." In the same paragraph, the base work is cited as also belonging to the authors. This might be a violation of the anonymous submission guidelines.
In Section 3, first paragraph, Algorithm 1 is not correctly referenced.
The term DOE is not properly defined in the text, only in Table 1.
In Section 6.1, what was the criterion for selecting the chosen 132 experiments out of the 4800?
In the last paragraph of Section 6.1, it says the detailed results are available in another paper. Why not include those results in the Appendix?
In Section 6.2, it seems the proposed method exhibits better performance, with proportionally more counts in the “yes” column (Table 6). However, only a very restricted set was evaluated, so it is not clear if those good results are related to the size of the networks or because fewer competing techniques were tried.
Figure 1 (b) seems to have a wrong Y label. I guess it should be Loss instead of Learning Rate. Besides, using a log-scale would give a better comparison.
In the first paragraph of Sec. 7 there is a typo in “limitaitons”.
Overall assessment scores: 1

2