How to decay your learning rate
Authors: Aitor Lewkowycz
Venue: ICLR 2022 Submitted (2022)
Keywords: learning rates
hyperparameter tuning
schedules

Abstract: Complex learning rate schedules have become an integral part of deep learning. We find empirically that common fine-tuned schedules decay the learning rate after the weight norm bounces. This leads to the proposal of ABEL: an automatic scheduler which decays the learning rate by keeping track of the weight norm. ABEL's performance matches that of tuned schedules, is more robust with respect to its parameters and does not depend on the time budget. Through extensive experiments in vision, NLP, and RL, we show that if the weight norm does not bounce, we can simplify schedules even further with no loss in performance. In such cases, a complex schedule has similar performance to a constant learning rate with a decay at the end of training. 

Under review as a conference paper at ICLR 2022
How TO DECAY YOUR LEARNING RATE
Anonymous authors Paper under double-blind review
ABSTRACT
Complex learning rate schedules have become an integral part of deep learning. We find empirically that common fine-tuned schedules decay the learning rate after the weight norm bounces. This leads to the proposal of ABEL: an automatic scheduler which decays the learning rate by keeping track of the weight norm. ABEL's performance matches that of tuned schedules, more robust with respect to its parameters and does not depend on the time budget. Through extensive experiments in vision, NLP, and RL, we show that if the weight norm does not bounce, we can simplify schedules even further with no loss in performance. In such cases, a complex schedule has similar performance to a constant learning rate with decay at the end of training.
INTRODUCTION
Learning rate schedules play a crucial role in modern deep learning. They were originally proposed with the goal of reducing noise to ensure the convergence of SGD convex optimization (Bottou, 1998). A variety of tuned schedules are often used, some of the most common being step-wise, linear or cosine decay. Each schedule has its own advantages and disadvantages and they all require hyperparameter tuning. Given this heterogeneity, it would be desirable to have a coherent picture of when schedules are useful and come up with good schedules with minimal tuning.
While we do not expect dependence on the initial learning rate in convex optimization, large learning rates behave quite different from small learning rates deep learning (Li al., 2020a; Lewkowyez et al., 2020). We expect the situation to be similar for learning rate schedules: the non-convex landscape makes it desirable to reduce the learning rate as we evolve our models. The goal this paper is to study empirically (a) in which situations schedules are beneficial and (b) when during training one should decay the learning rate. Because of the use of stochastic gradients, the loss will have a positive definite contribution due to noise. While this noise is key explore the landscape, we want to minimize this contribution at the end of training and we will use a simple schedule as our baseline: constant learning rate with decay close the end of training. Training with this smaller learning rate for a short time is expected to reduce the noise without letting the model explore the landscape too much. This is corroborated by the fact that the minimum test error often occurs almost immediately after decaying the learning rate. Part of the paper focuses on comparing the simple schedule with standard complex schedules used in the literature, studying the situations in which these complex schedules are advantageous. We find that complex schedules are considerably helpful whenever the weight norm bounces, which happens often in the usual, optimal setups. We observe that the weight norm (sum over the squared L2-norm of the weight in each layer) often presents a bouncing behaviour: it decreases, hits a minimum after and continues increasing. In the presence of bouncing weight norm, we propose automatic scheduler which performs well as fine-tuned schedules.
1.1 OUR CONTRIBUTION
The goal of the paper is study the benefits of learning rate schedules and when the learning rate should be decayed. We focus on the dynamics of the weight norm (sum over layers of the square of weights' To-norm)).

---

Under review as a conference paper at ICLR 2022
4000
25000
Weight Norm
Weight. Norn
3000
20000
15 15000
2000
10000
1000
20
40
60
80
100
150
200
Epochs
Epochs
(a) Resnet-50 on ImageNet
(b) WRN28-10 10 on CIFAR-100
Figure I: Evolution of the weight norm when training with step-wise decay (decay times marked by black dashed lines). The learning rate is decayed when the weight norm converges after bouncing. Models were evolved in optimal settings whose tuning did not use the weight norm as input.
The paper is divided into two parts and their main conclusions are:
1. Decaying the learning rate using complex schedules is only beneficial in the presence of weight bouncing. L2 regularization seems crucial for the weight norm to bounce and in its absence (which common in NL.P and RI.) we don't see a benefit from complex schedules. This is explored in detail in section 3 and the results are summarized in table 2.
2.
In the presence of bouncing, one should use the weight norm to inform when to decay the learning rate. We observe that tuned step-wise schedules decay the learning rate when the weight norm converges after bouncing'. Towards the end of training. last decay decreases the noise. See figure 1. We propose an Automatic, Bouncing into Equilibration Learning rate scheduler (ABEL). ABEL is competitive with fine-tuned schedules and needs less tuning (see table and discussion in section 2).
While these two points are somewhat independent, together they provide a very general picture of when to decay the learning rate. For pedagogical purposes, present the two points in inverse order: first reverse engineer what makes fine-tuned learning rate schedules work and then study to what extent these helpful across tasks and domains.
The origin of weight bouncing. In section 4, we explain why the weight norm often bounces.
Weight bouncing and performance. Generally speaking, weight bouncing occurs when we have non-zero L2 regularization and large enough learning rates. While L2 regularization is crucial in vision tasks, it is not found to be that beneficial in NLP or Reinforcement Learning tasks (for example Vaswani et al. (2017) does not use L2). If the weight norm does not bounce, ABEL yields the "simple" learning rate schedule that expect naively from the noise reduction picture: decay the learning rate once towards the end of training. We confirm that, in the absence of bouncing, such a simple schedule is competitive with
complicated ones across variety of tasks and architectures, table 2. We also see that the well known advantage of momentum compared with Adam for image classification in ImageNet (see Agarwal et al. (2020) for example) seems disappear in the absence of bouncing, when we turn off L2 regularization. Weight norm bouncing thus seems empirically necessary condition for non-trivial schedules to provide a benefit, but it that when the datasets are easy enough that simple schedules can get zero training error, schedules do make a difference for the generalization performance (models can hit zero training error and still improve the test error see Lewkowyc7 & Gur-Ari (2020) for example).
sufficient: we observe
1.2 RELATED WORKS
We do not know of any explicit discussion of weight bouncing in the literature. The dynamics of deep networks with T regularization has drawn recent attention, see for example van Laarhoven
We define bouncing by the monotonic decrease of the weight norm followed by monotonic increase which occurs for fixed learning
he in figure
2

---

Under review as a conference paper at ICLR 2022
(2017); Lewkowyez & Gur-Ari (2020); Li et al. (2020b); Wan et al. (2020); Kumin et al. (2020). The recent paper Wan et al. (2020) observes that the weight norm equilibration is a dynamical process (the weight norm still changes even if the equilibrium conditions are approximately satisfied) which happens soon after the bounce.
The classic justification for schedules comes from reducing the noise in quadratic potential (Bottou, 1998). Different schedules do not provide an advantage in convex optimization unless there is a substantial mismatch between train and test landscape (Nakkiran, 2020), however this is not the effect that we are observing in our setup: when schedules are beneficial, their training performance is substantially different, see for example figure S4. Of course, this is not too surprising because convex optimization does not apply to deep networks. The work of Li et al. (2020a) could be helpful to understand better the theory behind our phenomena, although it not clear to us how their mechanism can generalize to multiple decays (or other complex schedules). There has been lots of empirical work trying to learn schedules/optimizers see for example Maclaurin et al. (2015): Li & Malik (2016); Li et al. (2017); Wichrowska et al. (2017); Rolinek & Martius (2018); Qi al. (2020); You et al. (2017, 2020). Our approach does not have an outer loop: the learning rate is decayed depending on the weight norm, which is conceptually similar to the Reduce LROnP lat scheduler, where the learning rate is decayed when the loss plateaus which is present in most deep learning libraries. However, Reduce LROmP a aateuu does not perform well across tasks. couple of papers which thoroughly compare the performance of learning rate schedules are Shallue et al. (2019); Kaplan et al. (2020).
2
AN AUTOMATIC LEARNING RATE SCHEDULE BASED ON THE WEIGHT NORM
ABEL AND ITS MOTIVATION
From the two setups in figure 1 seems that optimal schedules tend decay the learning rate after bouncing, when the weight norm growth slows down". We can use this observation to propose ABEL (Automatic Bouncing into Equilibration Learning rate scheduler): a schedule which implements this behaviour automatically, see algorithm 1. In words, we keep track of the changes in weight norm between subsequent epochs (denoted by 1), A |uy,12= = |w, 12 |10 When the sign of ury/ flips, it necessarily means that it has gone through a local minimum: because A wyy 12 < 0 initially if A /we+1l2 12_ A w < 0, A w < 0, then [we] is a minimum: |wc-1] > |we| < 11. After this, the weight norm grows and slows down until at some point |/ is noise dominated. In this regime, A |2e12 will become negative, which we will take as our decaying condition. In order to reduce SGD noise, near the end of training we decay it one last time. In practice 85% of the total training time and as we can see in the SM B. this particular value does not really matter as long it is few epochs. This schedule does not require a fixed number of epochs (as opposed to cosine decay which is strict in this regard): one can decide to train models for longer times by loading the checkpoint before this last noise reducing decay, or stop models early (and do the corresponding last decay for a few epochs).
do this last decay around
Algorithm is an implementation of the idea with the base learning rate and the decay factor as the main hyperparameters. While alternative implementations could be more explicit about the weight norm slowing down after reaching the minimum, they would likely require more hyperparameters. The definition 1 does not guarantee that the learning rate will be decayed after the weight norm reaches a minimum, particular if the weight norm was noisy, 1 could trigger a learning rate decay without bouncing. In practice, in all the experiments that we ran (which span variations across 4 datasets, 6 learning rates, decay factors, 4 architectures). ABEL successfully decayed the learning rate after the weight norm reached equilibrium.
We have decided to focus on the total weight norm, but one might ask what happens with the layer- wise weight norm. In SM B.2, we study the evolution of the weight norm in different layers. We focus on the 10 layers which contribute the most to the weight norm (these layers account for 50% of the weight norm). We see that most layers exhibit the same bouncing plus slowing down pattern as the total weight norm and this happens at roughly the same time scale.
The decay rate and hase learning rates are not the same between experiments, but here we focus on when the learning rate decay occurs.
3

---

Under review as a conference paper at ICLR 2022
Algorithm 1 ABEL Scheduler
# Repeat every epoch f. reached manimum-Falleeee initially.
if (|w 2 - 1 (1w [ ) then
if reeaced_minnnneee minimum then
learning_rate = factor learning_rate reached_mes minimum=Falsee
else
reached minimum=True end if end if
If no fixed budget, decay for a few epochs when training is done instead, see end of sec.2.
if == last_ last__dccay_epoeeee_ee_aaaaaaaae then
learning rate decay factor learning rate end if
Setup
Test error
Dataset
Architecture
Step-wise
ABEL
Cosine
ImageNet
Resnet-50
24.0
23.8
23.2
CIFAR-10
WRN 28-10
3.7
3.8
3.5
CIFAR-10
VGG- 16
7.1
6.9
CIFAR-100
WRN 28-10 10
18.5
18.7
18.4
CIFAR-100 100
PyramidNet
10.8
10.8
SVHN
WRN 16-8
1.77
1.79
1.89
Table I: Comparison of test error at the end of training for different setups and learning rate schedules. We see that ABEL has very similar performance to the fine-tuned step-wise schedule without the need to tune when to decay. ABEL uses the baseline values of learning rates and decay factors and we have not fine-tuned these. The cells denoted by refer to setups for which we do not have reference step-wiee decays. Step-wise and cosine schedules use the optimal hyperparameters from our baselines and we plug these values for ABEL without further tuning, see SM for more experimental details.
PERFORMANCE COMPARISON ACROSS SETUPS
We have run variety of experiments comparing learning rate schedules with ABEL, see table for a summary and figure 2 for some selected training curves (rest of the training curves in SM D.1). We use ABEL without hyperparameter tuning: we are plugging the base learning rate and the decay factor of the reference step-wise schedule (these reference decay factors 0.2 for CIFAR and 0.1 for other datasets). We see that ABEL is competitive with existing fine-tuned learning rate schedules and slightly outperforms step-wise decay on ImageNet. Cosine often beats step-wise schedules, however as we will discuss shortly, such decay has several drawbacks.
ROBUSTNESS ABEL
ABEL is quite robust with respect to the learning rate and the decay factor. Since depends implicitly on the natural time scales of the system, it will adapt to when to decay the learning rate. We can illustrate this by repeating the ImageNet experiment with different base learning rates or decay factors. The results are shown in figure 3. Note: when the decay factor is 0.5 we evolved both models for 120 epochs (denote with star in figure) to allow more time to bounce. We observe more bounces for larger decay rates and shallower bounces the smaller the learning rate.
We would like to highlight the mild dependence of performance on the learning rate: the learning rate is too high, the weight norm will bounce faster and ABEL will adapt to this by quickly decaying the learning rate. This can be seen quite clearly in the learning rate = 16 training curves, see SM D.4.
ABEL also has the 'last decay_epoch' hyperparameter, which determines when to perform the last decay in order to reduce noise. Performance depends very weakly on this hyperparameter (see SM more) and for all setups in table we have chosen be 85% of the total training time. The most
4

---

Under review as a conference paper at ICLR 2022
0.40
0.8
ABEL.
AIEL
Weight Norm
25000
Leaming Rate
Coetse
Test Error
0.35
Stegwtne
0.6
Stepeene
20000
0.30
0.4
15000
0.25
Cheire
10000
0.2
Stepetse
0.0
0.20
20
40
60
80
20
60
80
20
40
60
80
Epochs
Epochs
Epochs
(a) Resnet-50 on ImageNet
(b) Resnet-50 on ImageNet
(c) Resnet-50 on ImageNet
0.30
8000
0.100
Test Error
0.25
Weight Norm
6000
Contre
Seepeior
ADEL
Learning Rate
0.075
Costae
Stepostoo
ABEL
4000
0.050
0.20
0.15
Chetre
ADEL
2000
0.025
Stepetse
0.000
50
100
150
200
0
50
100
150
200
0
50
100
150
200
Epochs
Epodas
Epuchs
(d) WRN28-10
CIFAR-100
(e) WRN28-10 10 on CIFAR-100
(I) WRN28-10 CIFAR-I00
Figure 2: Training curves of two experiments from table 1.
natural way to think about this would be to run ABEL for fixed amount of time and after decay the learning rate for a small number of epochs in order to get the performance with less SGD noise.
COMPARISON OF ABEL WITH OTHER SCHEDULES
It is very natural to compare ABEL with step- beep-wise decay. Step-wise decay is complicated to use in new settings because on top of the base learning rate and the decay factor, one has to determine when to decay the learning rate. ABEL,, takes care of the 'when' automatically without hurting performance. Because when to decay depends strongly on the system and its current hyperparameters, ABEL much more robust to the choices of base learning rate and decay factor.
second class of schedules are those which depend explicitly on the number of training epochs (T), like cosine or linear decay. This strongly determines the decay profile: with cosine decay, the learning rate will not decay by a factor of 10 with respect to its initial value until 93% of training! Having T as determining hyperparameter is problematic: it takes long time for these schedules to have comparable error rates to step-wiss wise decays, as can be seen in figures 2, S8. This implies that until very late in training one can not tell whether T is too short, in which case there is straightforward way to resume training (if we want to evolve the model with the same decay for a longer time, we have start training from the beginning). This is part of the reason why large models NLP and vision use schedules which can be easily resumed like rsqrt decay (Vaswani et al., 2017), "clipped" cosine decay (Kaplan et al., 2020; Brown et al., 2020) or exponential decay (Tan & Le, 2020). In contrast, for ABEL the learning rate at any given time is independent of the total training budget ( while there is the last fisst__ecy____eeee decay_epoch parameter, it can easily be evolved for longer if we load the model before the last decay).
We have decided to compare ABEL with the previous two schedules because they are the most commonly used ones. There are a lot of automatic/learnt learning rate schedules (or optimizers), see Maclaurin et al. (2015); Li & Malik (2016); Li et al. (2017); Wichrowska et al. (2017); Yaida (2018); Rolinek & Martius (2018); Qi et al. (2020) and to our knowledge most of them require either significant change in the code (like the addition of non- trivial measurements) or outer loops and also add hyperparameter of their own, so these never completely hyperparameter free. Compared with these algorithms ABEL is simple, interpretable (it can be easily compared with fine-tuned step-wise decays) and performs well as tuned schedules. It is also quite robust compared with other automatic
5

---

Under review as a conference paper at ICLR 2022
methods because it relies in the weight norm which is mostly noise free through training compared with other batched quantities like gradients or losses.
An algorithm similar in simplicity and interpretability is Reduce www.RRoonnattee lateau which is one of the basic optimizers of Py PyTorch or TensorFlow and decays the learning rate whenever the loss cquilibrates. We train Resnet-50 ImageNet model and WRN 28- 10 CIFAR-10 model with this algorithm. SM B.3 for details. We use the default hyperparameters and for the ImageNet experiment, the learning rate does decay at all, yielding a test error of 47.4. For CIFAR-IO, ReduceLROnP lateeau does fairly well, test error of 3.9, however the learning rate decays without bound rather fast. These two experiments suggest that ReduceLROn Mampllaeau eau can not really compete with the schedules described above: it seems rather finicky with respect to hyperparameters and we do not want to have to precisely tune different hyperparameters for a given setup (see robustness discussion).
0.40
0.40
Suppiis
Stepwinn
ABEL
ADEL
Test. Earor
0.35
Cotnn
Test Error
0.35
0.30
0.30
0.25
0.25
10'
10
0.2
0.3
0.4
0.5
Learning Rate (a)
Docay Factor (b)
Figure 3: ResNet-50 trained on ImageNet for different learning rates and decay factors. (a) ABEL beats others schedules when using non-optimal learning rates. At learning rate 40, only ABEL converges. (b) ABEL is robust with respect to changes in the decay factor, its performance does not depend too much on the decay factor because it adjusts the number of decays accordingly.
ABEL DOES NOT REQUIRE A FIXED TRAIN BUDGET
From the empirical studies, the drop in the test error after decaying the learning rate is upper bounded by the previous drops in the test eπrоr, reason for this is that this drop can be attributed to a reduction of the SGD noise and smaller learning rates have less SGD noise. This provides an automatic way of prescribing the train budget: if the improvement of accuracy after decay is smaller than some threshold, exit training after a small number of epochs (to process the last decay). This approach _eccch'eeeehhe hyperparameter. Such training setup would not be possible for
does not have cosine/linear decay by construction since they depend on the training budget. This seems hard for step- wise decay since there is no way to predetermine how decay the learning rate automatically. In order to compare ABEL with the standard baselines, all the experiments in the paper have been trained with fixed budget.
3 SCHEDULES AND PERFORMANCE IN THE ABSENCE OF A BOUNCING WEIGHT NORM
In this section, we study settings where the weight norm does not bounce understand the impact of learning rate schedules on performance. Setups without L2 regularization are the most common situation with no bouncing, these setups often present a monotonically increasing weight norm. is not clear to us what characteristics of task make 122 regularization beneficial but as shown in Table 2 it seems that Vision benefits considerably more from than NLP or RI.
We conduct an extensive set of experiments in the realms of vision, NLP and RL and the results are summarized in table 2. In these experiments, we compare complex learning rate schedules with a simple schedule where the model is evolved with a constant learning rate and decayed once towards the end of training. This simple schedule mainly reduces noise: the error decreases considerably immediately after decaying and it does not change much afterwards (it often increases). Across these experiments, we observe that complicated learning rate schedules are not significantly better than the simple ones. For couple of tasks (like ALBERT fine-tuning or WRN on CIFAR-100),

---

Under review as a conference paper at ICLR 2022
complex schedules are slightly better (~ 0.3%) than the simple decay but this small advantage is nothing compared with the substantial advantage that schedules have in vision tasks with L2. Another situation where there is no bouncing weight norm is for small learning rates, for example VGG-16 with learning rate 0.01, in such case there is also no benefit from using complex schedules, see SM B.5 for more details. Note that in this paper are using T22 regularization and weight decay interchangeably: what matters is that there is explicit weight regularization. These experiments also show that the well known advantage of momentum versus Adam for vision tasks is only significant in the presence of L2. In the absence of a benefit from L2 regularization / weight decay it seems like Adam is a better optimizer, Agarwal et al. (2020) suggested that this is because it can adjusts the learning rate of each layer appropriately and it would be interesting to understand whether there is any connection between that and bouncing.
These experiments have growing weight norm as can be seen in SM D.3. While the weight norm does not have to be always increasing in the absence of L22 regularization, this is a function of the learning rate (see section 4.1). and the learning rates used in practice exhibit this property. Homogeneous networks with cross entropy loss will have an increasing weight norm at late times, see Lyu & Li (2020). Even if simple schedule is competitive this does not imply that other features of convex optimization like the independence of performance in the learning rate carry over. We repeat the CIFAR- 100 experiments for a fixed small learning rate of 0.02 (the same as the final learning rate for the simple schedule) and the error with L2- = O is 23.8 while with L22 * O is 29.7, we see that while there a performance gap between a small and large learning rates, this gap is much smaller if there is no bouncing (difference in error rate of 1.2% for L2 0 vs 7.5% for L2 + 0). For fair comparison with small learning rates, we evolved these experiments for 5 times longer than the large learning rates, but this did not give any benefit.
While the experiments presented in table 2 do not have La regularization, some NLP architectures like Devlin et al. (2019); Brown et al. (2020) have weight decays of 0.01, 0.1 respectively. We tried adding weight decay to our translation models and while performance did not change substantially, we were not able to get a bouncing weight norm.
The effect of different learning rate schedules in NLP was also studied thoroughly in appendix D.6 of Kaplan et al. (2020) with the similar conclusion that as long as the learning rate is not small and is decayed near the end of training, performance stays roughly the same.
The presence of a bouncing weight norm does not guarantee that schedules are beneficial. From this section, a bouncing weight norm seems to be a necessary condition for learning rate sched- ules to matter, but is not a sufficient condition. Learning rate schedules seem only advantageous if the training task is hard enough. In our experience, if the training data can be memorized with a simple learning rate schedule before the weight norm has bounced, then more complex schedules are not useful. This can be seen by removing data augmentation in our Wide Resnet CIFAR experiments, see figure 4. In the presence of data augmentation, simple schedules can not reach training error 0 even when evolved for 200 epochs, see SM.
0.3
0.3
Costne
Pest, Stuple
Errar Rate
Traio Cosian
Error Rate
Itaii. Simpie
0.2
Sampie TDaip Simple (1
0.2
Test,
Irail. Coiine
0.1
0.1
0.0
0.0
50
100
150
200
10
15
Epochs
Epochs
(a) WRN on 10, 200 epochs
(b) WRN on 10, 15 epachs
Figure 4: Wide Resnet CIFAR- without data augmentation evolved for 200 epochs (left) and 15 epochs (right). In this setup, the weight norm bounces at around 15 epochs. a) Both schedules reach training error 0 and their performance is the same (error of 7.3). b) If we evolve the model for only 15 epochs, both schedules can still get training error 0 without a weight norm bounce, we think this is the reason why there is no performance difference in a).
7

---

Under review as a conference paper at ICLR 2022
Setup
Performance for different schedules
Type
Task and metric
Architecture
Complex Decay
Simple Decay
EN-DE, BLEU
Transformer
29.0
28.
NLP
EN-FR, BLEU
Transformer
43.0
43.0
GLUE, Average score
ALBERT fine-tuning
83.1
82.9
Qbert, Score
PPO
1750
1850
RL
Seaquest, Score
PPO
21.0
20.7
Pong, Score
PPO
22300
23000
ImageNet, Test Accuracy
Resnet-50
71.9
72.2
ImageNet, Test Accuracy
Resnet-50 + Adam
71.8
71.1
Vision L2 =
CIFAR-10, Test Accuracy
Wide Resnet 28-10
95.0
95.0
CIFAR-100, Test Accuracy
Wide Resnet 28-10
78.2
77.9
ImageNet, Test Accuracy ImageNet, Test Accuracy CIFAR-10. 10, Test Accuracy CIFAR-100, Test Accuracy
Resnet-50
76.8
71.5
Vision L2 + 0
Resnet-50 + Adam
75.3
74.0
(has bounce)
Wide Resnet 28-10
96.5
95.1
Wide Resnet 28-10
82.2
77.8
Table 2: Comparison of performance between simple learning rate decay and "complex" decay among tasks: "complex" means cosine decay for vision tasks and linear decay for NLP and RL. For NLP and RL tasks higher metrics imply better performance, while for vision tasks, lower error denotes better performance. None of these tasks (except for the vision task with L2 used as a reference) have weight norm bouncing nor advantage from non-simple schedules. We have averaged the RL tasks over 3 runs and their difference is compatible with noise. See S1 for the individual GLUE scores, as it is common we have omitted the problematic WNLI. We use test accuracy 100- error, so that for all metrics in these table, higher is better.
UNDERSTANDING WEIGHT NORM BOUNCING
In this section, we will pursue some first steps towards understanding the mechanism behind the phenomena that found empirically in the previous sections.
4.1 INTUITION BEHIND BOUNCING BEHAVIOUR
We can build intuition about the dynamics of the weight norm by studying its dynamics under SGD updates. Given the loss L, in the absence of L2 regularization, let dLa be its gradient and À dws
are the learning rate and Lg regularization coefficient, one can easily derive:
Awa+1 Wr++ Wt = -119t n/wr
A W 2
n7² gg 2 --n)n>> w 2(1 m/)ge wt
n² 1lgl 2n///wal2 2nge we O()
where we have used that empirically nd < 1. This equation holds layer by layer and we include patch-norm weights if present. In the absence of L2 regularization, for large enough learning rates
1g.1² this suggests that the weight norm will be increasing.
(1)
2ge
Equation can be further simplified for scale invariant networks, which satisfy Sr wy O. see for example van Laarhoven (2017)* S. In the absence of such term, we see that the updates of the weight norm are determined by the relative values of the gradient and weight norm. If X 0 or 1 is very small, the weight norm updates will have fixed sign and thus there will not be bouncing. More generally, we expect that in the initial stages of training. the weight norm is large and its dynamics are dominated by the decay term. As it shrinks, the relative value of the gradient norm term becomes larger and it seems natural that at some point, will dominate, making the weight norm bounce. This is also studied in Wan et al. (2020), where it is shown that after the bounce, the two terms in equation I are the same order and the weight norm "dynamically equilibrates" (although it can not stay constant
Scale invariant networks are defined by network functions which independent of the weight norm: f(aw) f(n) The weight of these functions still affects its dynamics (Li & Arora, 2019).
8

---

Under review as a conference paper at ICLR 2022
because the gradient norm changes with time). While we expect the St . we to be non-zero in our setups, only layers which are not scale invariant would contribute to this term and roughly any layer before BatchNorm layer is scale invariant
expect this term to be smaller than the other two.
The only necessary condition for a model to have a bouncing weight norm seems to be that it has Le regularization and large learning rate. Empirically, we have seen that different optimizers, losses and batch sizes can have bouncing weight norm.
4.2 TOWARDS UNDERSTANDING THE BENEFITS OF BOUNCING AND SCHEDULES
To better understand this phenomenon, finding the simplest model that captures it would be desirable: we expect this to be generic as long as we have L2 regularization and the learning rate is big enough. We believe that learning rate schedules being only advantageous for hard tasks (as we discussed in section 3) is the principal roadblock to find theoretically tractable models of these phenomena.
For bouncing setups, decaying the learning rate when the weight norm is equilibrating allows the weight decay term in equation 1 to dominate, causing the weight norm to bounce again. However, from equation 1, in the absence of T.2 decaying the learning rate can only slow down the weight norm equilibration process and this implies that the weights change more slowly, see SM. It seems like the combination weight bouncing and decaying the learning rate might be beneficial because allows the model to explore larger portion of the landscape. Exploring this direction further might yield better insights to this phenomenon, perhaps building on the results of Wan et al. (2020); Kunin et al. (2020).
In the SM C, we do some extra experiments which explore properties of bouncing weight norm models.
The disadvantage of decaying too early or too late. Waiting for the weight norm to bounce seems key to get good performance. Decaying too late might be harmful because the weight norm does not have enough time to bounce again, but it is not clear if that is bad. We run simple VGG-5 experiment on CIFAR-100 and see that decaying too early significantly hurts performance and it is best to decay is after the weight norm has started slowing down its growth, before it is fully equilibrated.
Dependence on initialization scale. One could wonder if the bounce would disappear if we change the initialization of the weights so that the initial weight norm is smaller than the minimum of the bounce with the original normalization. We studied this and conclude that even for very small initialization scales, there is a bouncing weight norm. If the initialization scale is too small, the bouncing weight norm disappears and the performance gets significantly degraded.
CONCLUSIONS AND LIMITATIONS
In this work we have studied the connections between learning rate schedules and the weight norm. We have made the empirical observation that a bouncing weight norm is a necessary condition for complex learning rate schedules to be heneficial, and we have observed that step-wise schedules tend to decay the learning rates when the weight norm equilibrates after bouncing. We have checked these observations across architectures and datasets and have proposed ABEL: a learning rate scheduler which automatically decays the learning rate depending on the weight norm, performs as well as fine-tuned schedules and is more robust than standard schedules with respect to its initial learning rate and the decay factor. In the absence of weight bouncing, complex schedules do not seem to matter.
We have studied a diverse yet fixed number of setups. For vision tasks where the weight norm bounces, ABEL provides an automatic schedule which has significant advantages in terms of hyperparameter tuning. The other main application is that without bouncing (common in NLP and RI.,, ABEL is constant learning rate with decay at the end of training and this matches the performance of complicated schedules. This might have been expected from the fact that different groups use different schedules in NLP (linear, rsqrt, clipped cosine) without significant impact in performance. In this context, we would like to argue that there is no need to optimize for the precise form of the schedules. Weight bouncing does not seem related with warmup and we have kept if the baselines had it. ABEL does not improve the results of common schedules but it is quite robust and does not depend on the total number of epochs, making it more efficient when confronting setups.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang. Disentangling adaptive gradient methods from learning rates, 2020.
Léon Bottou. Online learning and stochastic approximations, 1998.
James Bradbury, Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, and Skye Wanderman-Milne. JAX: composable transformations of Python+NumPy programs. 2018. URL / com/ coom/googleeee jax.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. 2019.
Ming-Wei
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Daniel Kunin, Javier Bagastuy-Brena Surya Ganguli, Daniel L. K. Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics, 2020.
Aitor Lewkowycz and Guy Gur-Ari. On the training dynamics deep networks with l2 regularization, 2020.
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism, 2020.
Ke Li and Jitendra Malik. Learning to optimize, 2016.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks, 2020a.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning, 2017.
Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning, 2019.
Zhiyuan Li, Kaifeng Lyu. and Sanjeev Arora. Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate, 2020b.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks, 2020.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimiza- tion through reversible learning, 2015.
Preetum Nakkiran. Learning rate annealing can provably help generalization, even for convex problems, 2020.
Xiaoman Qi, PanPan Zhu, Yuebin Wang, Liqiang Zhang, Junhuan Peng. Mengfan Wu, Jialong Chen, Xudong Zhao, Ning Zang. and P. Takis Mathiopoulos. Mirsnet: A multi-label high spatial resolution remote sensing dataset for semantic scene understanding, 2020.
Michal Rolinek and Georg Martius. L4: Practical loss-based stepsize adaptation for deep learning, 2018.
10

---

Under review as a conference paper at ICLR 2022
Christopher J. Shallue, Jachoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. Measuring the effects of data parallelism on neural network training, 2019.
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks, 2020.
Twan van Laarhoven. L.2 regularization versus batch and weight normalization, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
Ruosi Wan, Zhanxing Zhu, Xiangyu Zhang, and Jian Sun. Spherical motion dynamics: Learning dynamics of neural network with normalization, weight decay, and sgd, 2020.
Olga Wichrowska, Niru Maheswaranathan, Matthew W. Hoffmann Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize, 2017.
Sho Yaida. Fluctuation-diisiiation relations for stochastic gradient descent, 2018.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks, 2017.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes, 2020.


Reviews:
[{"review_id": "t31EAG_PoPT", "reviewer": "Reviewer_y2w2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Step-wise schedules have been proved effective in a number of contexts (especially in computer vision). The proposed scheduler is a very simple modification that allows to automatically determine the decay epoch without adding much complexity to the resulting pipeline. \n- Linking the weight norm to the learning rate is a novel and interesting take on the topic of automatically tuning learning rate schedules... \n\nWeaknesses\n- ... however, at the moment the work seems almost entirely heuristic driven. The paper needs more work before being ready for publication, as I will detail below.\n1. Following on what I mentioned in the summary, while I can understand that, empirically, changes of sign in the weight norm happen in concomitance with what the authors say, I do not think that it is proper to present the method as it is done at the moment. From what I can read and from the pseudocode of the algorithm, ABEL decays the learning rate after registering two changes in the direction in which the weight norm is progressing. This does not necessarily mean that the first change happens at a (local) minimum while the second is because of noise. Just to be fully clear, what I mean is that, in principle, the first change could also be a local maximum. Then, empirically you can observe what the authors say, but, in order to strengthen the position of the paper, I think it would be necessary to show some more organized results about this phenomenon. For instance, reporting something on this line: \"over N experiments changing Y and Z conditions (e.g. varying the random seed, the initial learning rate, dataset, network architecture, etc), we observe that pair of changes in weight progression are in X% of the cases linked to [....]\". In these experiments, particular care must be taken not to confuse the causal direction of the interplay between weight norm and learning rate.\n2. About the point above, one of the phenomena that worries me the most is that changes could be linked to noise. In fact, I noticed that in the code there is also a parameter called `meas_freq` that counteracts the presence of noise in some settings. This is a hyperparameter of the method which seems not to be mentioned in the main paper, which is probably used for some of the experiments (e.g. looking at the last row of Figure S8). Could the authors explain this? How sensible is ABEL w.r.t. this hyperparamter?\n3. I can agree that ABEL is not as linked to the training budget as the cosine scheduler is, but I substantially disagree with the claim that ABEL does not require (at all) a fixed training budget. The training budget is even a parameter of ABEL in the code. This is not a major concern in my eyes, but I think the claim must be played down from what it is currently written.\n4. There is ample space to improve the clarity of the paper. Here below are some points about this:\n  a) For starters, I do not think that the authors mention explicitly what $t$ denotes (I assume it to be epochs, but could be e.g. also iterations...). This is quite important, especially when paired with observation 2. \n  b) The introductory discussion about the learning rate decay depicts a rather partial picture (e.g. decay comes up also in the convergence proofs of subgradent methods for non-smooth optimization). \n  c) The the sentence  *Given that stochastic gradients are used in deep learning, we will use a simple schedule as our baseline: a constant learning rate with one decay close to the end of training.* is quite obscure to me. Why using stochastic gradients should imply that a simple schedule is a good baseline?\n  d) The overall organization makes it difficult to pinpoint the actual contribution of the paper, as observations and experimental results are mixed with comments and parenthesis. \n  e) Equation (1) lacks context. How is the loss defined? Are the authors considering networks that use batch norm or not (or both)? If yes, are the parameters of the batch norm layers included in w? Generally, the paper would greatly benefit from a clearer framing of the context. What is the typical setting that the authors have in mind and that ABEL is designed for?\n5. The utility of (automatic) learning rate schedulers can be twofold: 1) improve on the state of the art, possibly starting from standard hyperparameter settings 2) ease the burden of hyperparameter tuning, especially when dealing with novel learning settings and datasets. As ABEL seems not to excel at 1) I think the authors should focus more on experiments to support 2). These could be done e.g. setting up time-controlled (repeated) experiments as in [1] .\n6. I do not think that section 4 adds much value to the paper. Could the authors comment and summarize their findings reported in sec 4. and what do they add to e.g. Wan et al. (2020)?\n\nTypos:\n- which depend explicitly *in* (on) the number of training \n- to have a bouncing weight norm is that *is* (it?) has L2 regularization\n\n[1] Donini, Michele, et al. \"MARTHE: Scheduling the Learning Rate Via Online Hypergradients.\" IJCAI 2020.\n\n--------------------------------\n**Post-rebuttal:** I thank the authors for their reply and modifications to the text that addressed some of my points. However, I remain of the opinion that this paper needs more work on several fronts before recommending acceptance. Among these, I would remark on: better treatment of the background material, clearer identification on when the weight norm behaviour happens beside $L^2$ norm (possibly looking also for counter-examples!), rethinking section 6, and a more convincing set of experiments (for showing convincing evidence about e.g. 5.2). \nRegarding this last point, I want to clarify that in my review I mentioned [1] not for the grid search, but rather for the time-controlled experiments. If you go with random search for selecting the hyperparameters of the learning rate adaptation methods. I personally think that a recipe to make the comparison fair enough is to choose a prior distribution (e.g. uniform/log uniform) that covers reasonable values (e.g. as used for different datasets) with mean equal/close to the known well-performing (\"optimal\") value. \n\n\nI think the main idea of the paper - i.e. linking weight norm to learning rate - is a quite interesting, novel and practical take. I do not raise my score because I think that the work would benefit greatly from a thorough review which could lead to a stronger submission to a later conference. \n\nFor 4.b, see e.g. Bertsekas, Dimitri P. (2015). Convex Optimization Algorithms (Second ed.). Belmont, MA.: Athena Scientific.", "summary_of_the_paper": "This paper presents an empirical study of the dynamics of the total norm of the weights of neural network models in relation to the learning rate scheduling, focusing on what the authors call the \"bouncing effect\".  \nBased on empirical findings, they propose a method called ABEL to automatically decay the learning rate in a step-wise fashion. Essentially ABEL decays the learning rate after registering two changes in the direction in which the weight norm is progressing. The authors assume that the first change occurs at a local minimum, while the subsequent one indicates that the noise prevails the signal. \nABEL further decays the learning rate one last time at ~85% of the total training epochs.\nThey show that the cosine annealing and (manually) tuned step-wise schedules are beneficial when the bouncing effect happens, while in the absence of it, they claim that using a constant learning rate for most of the training epochs with a final decay is sufficient to obtain good performances.\nThe paper concludes with a section dedicated to understanding the bouncing effect which recalls some recent works that study the weight dynamics of neural nets.", "summary_of_the_review": "Linking the weight norm to the learning rate is a novel and interesting take on the topic of automatically tuning learning rate schedules. However, at the moment the work seems almost entirely heuristic driven. The paper needs more work before being ready for publication, both in terms of writing and exposing ideas and in terms of experiments, especially regarding the main assumptions of ABEL.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "g5Z_6ejhCZb", "reviewer": "Reviewer_iadJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* The proposed idea seems simple and effective. ABEL does not require lots of hyperparamters to tune and it is straightforward to implement.\n* The empirical results looks adequate. To compare ABEL with other scheduling methods, the authors not only provide the final performance of model but also study the roustness of ABEL. ABEL gets close to baselines on several architectures and data sets, and it seems to be slightly more robust to the choices of initial learning rate and decay factor.\n* Besides the ABEL algorithm itself, the authors also study bouncing weight norm in depth. They also show how to deal with the case where the weight norm is not bouncing and how it is connect to $l_2$ regularization. They also indicate that the presence of weight norm bouncing does not guarantee the schedule is beneficial but it depends on the complexity of the training task. \n\nCons:\n* Some parts of the paper are not well-described or I could not find the related context. e.g. Algorithm 1 could have been more complete. Even the initialization of reached_minimum is not provided. The current one is readable but could be more detailed. Also, in section 4.1 equation (1) pops up from nowhere. As a reader, I find it hard to understand where this update of SGD comes from, so the authors might consider moving the deduction of that equation from supplements as it has only a few lines. Besides, the mathematics notations look weird to me and it would be helpful if the authors could give them more clear definitions.\n* I haven't noticed any theoretical guarantee about such learning rate decay scheme. The results provided in the paper are most empirical. However, there are some parts not that obvious to me. For example, even if the authors say that learning rate schedules have advantages only when the training task is hard enough, I do not find any concrete evidence to support such view. Data augmentation might influence the results but it might be reason that the generalization ability of the model changes but not the training process gets affected. It would be helpful if the authors could give more insights about the reason for such observations. \n* The experiments seem well designed, but ABEL is actually losing to Cosine for most cases. That somehow hurts the practicability of ABEL.", "summary_of_the_paper": "The paper proposes a simple method to decay the learning rate automatically by tracking the weight norm of deep neural networks. A key observation from this work is that: there is a connection between weight norm bouncing, weight decay and optimizer convergence. The authors indicate that complex learning only benefits the training when weight norm bouncing exists, which further relies the presence of $l_2$ regularization on the weight. They also indicate that for those experiments without weight norm bouncing, like experiments in NLP and RL, the learning rate schedules could be simplified with no loss in performance.\n\nBased on that, the paper proposes a new learning rate scheduler named ABEL. Basically, ABEL will drop the learning rate when the weight norm starts to bouncing. More specifically, when the weight norm growth slows down, the learning rate will be dropped by a pre-defined dropping factor. ", "summary_of_the_review": "The proposed idea is concise and interesting. However, it needs stronger justification to show that ABEL has advantage over other learning rate scheduling methods and the authors could provide a deeper study about the behavior of weight norm bouncing from a perspective of SGD dynamics.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Vsf40uIPTzJ", "reviewer": "Reviewer_oLgq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Finding better learning rate schedulers, either in the metric of performance or robustness, is an important aspect of training deep networks. The suggestion by the authors to autotune the learning rate schedule by an internal property of the DNN (namely the bouncing feature of the weight norms) is interesting as well as the conjecture that such oscillations of the weight norm are a necessary condition for complex learning schedules. In addition, the equation governing this bouncing behaviour is simple enough to attract further theoretical interest. The scope of the experiments is also reasonable in my mind. \n\nIt would have been more exciting to show performances that exceed state of the art in some models, however, I accept that this is quite a high bar due to the mature state of the field. It remains to be seen whether this algorithm will be taken up by practitioners and whether it would push the state of the art in the field. \n\nThe work compares the ABEL scheduler with only a single adaptive learning rate scheduler (reduceLRonPlateau) and shows superiority. Other than that they compare with non-adaptive methods. What about algorithms such as AdaGrad or other adaptive schedulers on the market? More fundamentally, one can argue that their scheduling based on weight norm will be similar to scheduling based on any clear \"equilibration\" feature of the training loss. True the authors show that a plateau is not a robust feature. But still, I could imagine that, for instance, waiting for the average derivative of the training loss or the average gradient to decay to 10% of its initial value would be just as good. To summarize it'll be nice to disentangle whether it is the adaptive nature of the algorithm that is mainly useful here, or whether adapting it specifically based on the weight norms is central to its success.  \n\nMore minor comments are -\n\n1. In addition in their algorithm, I didn't see a precise treatment of a sticky point - what happens if one initializes the weights to a smaller value thereby reaching the maximum before the minimum? Put differently how to initialize the reached_minima variable, by measuring the slope of the weight_norm at the first epoch?\n\n2. The paragraph which starts with \"The origin of weight bouncing.\" tries both to explain what was the contribution and derive it in a hand-wavy manner. I'd personally be happier with just citing the contribution and deferring the arguments to where there's more room to go over them properly. \n\n3. The authors comment that cosine annealing takes 93% of the training budget to reduce LR by a factor of 10 and finish the sentence with an exclamation mark. I don't see why that's necessarily an obvious drawback. \n\n4. In the motivation section, the authors say \"From the two setups in figure 1 it seems that optimal schedules tend to decay the learning rate after bouncing when the weight norm growth slows down.\" which can be taken to think that these two schedules really match. However, there's often a factor of 2 between them and occasionally (for Wide Resnet 16-8 on SVHN) a factor of 5. \n\n\n\n ", "summary_of_the_paper": "This work deals with the optimization of learning rate schedules for deep learning. It offers three main contributions \n\n1. An equation relating the change in weight norm to the gradient norm, learning rate, weight decay, and the weight norm--- for weights that go into batch-norm. This equation provides intuition for the behaviour of this weight norm. \n\n2. An algorithm and code for adjusting the learning rate by reducing it by a factor once the norm of the weights reach a maximum, then reducing it again by the same factor after about 85% of the total training time. The algorithm is benchmarked on about 14 learning tasks, within vision, RL, and NLP and shows similar results when compared to other complex learning rate schedulers algorithms. Being adaptive, their algorithm has at least one less hyperparameter compared to the competition and appears more robust. \n\n3. Empirical tests of a conjecture that complex learning rates are only beneficial in the presence of extremums in the overall weight norms.  \n\n", "summary_of_the_review": "Finding better learning rate schedulers, either in the metric of performance or robustness, is an important aspect of training deep networks. The suggestion by the authors to autotune the learning rate schedule by an internal property of the DNN (namely the bouncing feature of the weight norms) is interesting as well as the conjecture that such oscillations of the weight norm are a necessary condition for complex learning schedules. In addition, the equation governing this bouncing behaviour is simple enough to attract further theoretical interest. The scope of the experiments is also reasonable in my mind, although further comparison with other adaptive algorithms could be useful. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DZzdUK2h3O0", "reviewer": "Reviewer_rqFL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n1. The observation of weight norm bounce and its correlation with the effect of LR decay is novel (as far as I know) and interesting. The proposed technique is also simple enough to be practically useful. \n2. I also appreciate the careful investigation on the generality of this finding, including experiments on different benchmarks as well as the importance of l2 weight decay. The results and analyses are pretty thought provoking. \n3. Writing is pretty clear and easy to follow.\n\nCons & Questions\n1. The title seems a bit too general (and bold), and can probably be made more specific about the findings. \n2. As a paper about SGD, I'm a bit disappointed that training loss curves are not shown in the main text, along side test errors. Having both will help a lot on disentangling the benefit from optimization and generalization, which I think the paper should explicitly discuss and clarify. \n3. Normalization layers (BN, LN) and others are also important confounding factors to the training dynamics, which I don't think the paper explicitly discuss. It appears to me that all models experimented are equipped with some normalization layers (correct me if I'm wrong), does removing them change the conclusion?\n4. Other than the weight norm, I'm also curious how the gradient norm evolves during training and if it's useful in a similar way for guiding the LR decay.\n\n***post rebuttal***\nI'd like to thank the authors for addressing my questions. Overall, I decide to stand with my original ratings. I still believe that this paper presents interesting empirical findings regarding the empirical correlation between weight norm increase and the effectiveness of LR decay. And I think this observation can potentially help us better understand the training dynamics of deep models. However, I also share similar concerns with other reviewers, where the proposed method is ad hoc and does not strictly outperform standard LR decay schedules. ", "summary_of_the_paper": "This paper presents a novel method for automatically decaying the learning rate when training deep neural networks. The core idea relies on the observation that the weight norms often times bounce up during training, and that decaying the learning rate after the bouncing is beneficial. The authors conducted extensive experiments in different settings, highlighting the effectiveness of the proposed LR decay schedule, as well as the special cases where weight bouncing doesn't happen and simple decay schedules work competitively.  Overall, I think this paper shows intriguing findings, and opens potential new directions for further understanding the training and generalization of deep neural networks. However, there are also questions that I hope the authors can answer in the rebuttal period, in order to further improve this paper. ", "summary_of_the_review": "Overall I think this is an interesting work, and its quality can be further improved by answering the questions listed above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors provide an investigation into tuning learning rate schedules. The problem is certainly of great practical importance. After discussion, the reviewers felt the main idea of the paper is worth pursuing, but could use significant refinement. One reviewer suggests: "
"...better treatment of the background material, clearer identification on when the weight norm behaviour happens beside norm (possibly looking also for counter-examples!), rethinking section 6, and a more convincing set of experiments (for showing convincing evidence about e.g. 5.2). Regarding this last point, I want to clarify that in my review I mentioned [1] not for the grid search, but rather for the time-controlled experiments. If you go with random search for selecting the hyperparameters of the learning rate adaptation methods. I personally think that a recipe to make the comparison fair enough is to choose a prior distribution (e.g. uniform/log uniform) that covers reasonable values (e.g. as used for different datasets) with mean equal/close to the known well-performing ("optimal") value." Other reviewers were generally of a similar opinion. The authors are encouraged to continue with the work, taking reviewer comments into account for updated versions.
Decision: Reject