How Does Learning Rate Decay Help Modern Neural Networks?
Authors: Kaichao You
Mingsheng Long
Jianmin Wang
Michael I. Jordan
Venue:  (2020)
Keywords: Learning rate decay
Optimization
Explainability
Deep learning
Transfer learning

Abstract: Learning rate decay (lrDecay) is a \emph{de facto} technique for training modern neural networks. It starts with a large learning rate and then decays it multiple times. It is empirically observed to help both optimization and generalization. Common beliefs in how lrDecay works come from the optimization analysis of (Stochastic) Gradient Descent: 1) an initially large learning rate accelerates training or helps the network escape spurious local minima; 2) decaying the learning rate helps the network converge to a local minimum and avoid oscillation. Despite the popularity of these common beliefs, experiments suggest that they are insufficient in explaining the general effectiveness of lrDecay in training modern neural networks that are deep, wide, and nonconvex. We provide another novel explanation: an initially large learning rate suppresses the network from memorizing noisy data while decaying the learning rate improves the learning of complex patterns. The proposed explanation is validated on a carefully-constructed dataset with tractable pattern complexity. And its implication, that additional patterns learned in later stages of lrDecay are more complex and thus less transferable, is justified in real-world datasets. We believe that this alternative explanation will shed light into the design of better training strategies for modern neural networks.

Under review as a conference paper at ICLR 2020
How DOES LEARNING RATE DECAY HELP MODERN NEURAL NETWORKS?
Anonymous authors Paper under double-blind review
ABSTRACT
Learning rate decay (IrDecay) is de facto technique for training modern neural networks. It starts with a large learning rate and then decays it multiple times. It is empirically observed to help both optimization and generalization. Common beliefs in how IrDecay works come from the optimization analysis of (Stochas- tic) Gradient Descent: an initially large learning rate accelerates training or helps the network escape spurious local minima; 2) decaying the learning rate helps the network converge a local minimum and avoid oscillation. Despite the popularity of these common beliefs, experiments suggest that they are insufficient explaining the general effectiveness of IrDecay in training modern neural net- works that are deep, wide, and nonconvex. We provide another novel explanation: an initially large learning rate suppresses the network from memorizing noisy data while decaying the learning rate improves the learning of complex patterns. The proposed explanation is validated on a carefully- construtecd dataset with tractable pattern complexity. And its implication, that additional patterns learned in later stages of IrDecay are more complex and thus less transferable, is justified in real- world datasets. We believe that this alternative explanation will shed light into the design of better training strategies for modern neural networks.
INTRODUCTION
Modern neural networks are deep, wide, and nonconvex. They are powerful tools for representation learning and serve core components of deep learning systems. They are top-perforning models in language translation (Sutskever et al., 2014), visual recognition (He et al., 2016), and decision making (Silver et al., 2018). However, the understanding of modern neural networks is way behind their broad applications. A series of pioncering works (Zhang et al., 2017: Belkin et al., 2019; Locatello et al., 2019) reveal the difficulty of applying conventional machine learning wisdom to deep learning. A better understanding of deep learning is a major mission in the AI field.
One obstacle in the way of understanding deep learning is the existence of magic modules in modern neural networks and magic tricks to train them. Take batch normalization module (loffe & Szegedy, 2015) for example, its pervasiveness in both academia and industry is undoubted. The exact reason why it expedites training and helps generalization, however, remains mysterious and is actively studied recent years (Bjorck et al., 2018; Santurkar al., 2018: Kohler al., 2019). Only when we clearly understand these magical practices can we promote the theoretical understanding of modern neural networks.
Learning rate is "the single most important hyper- parameter" (Bengio, 2012) in training neural net- works. Learning rate decay (IrDecay) a de facto technique for training modern neural networks, where we adopt an initially large learning rate and then decay it by certain factor after pre-defined epochs. Popular deep networks such as ResNet (He al., 2016), DenseNet (Huang et al., 2017b) all trained by Stochastic Gradient Descent (SGD) with IrDecay. Figure 1(a) is an example of IrDecay, with the learning rate decayed by 10 every 30 epochs. The training is divided into several stages by the moments of decay. These stages be easily identified in learning curves (such as Figure ((b),, where the performance boosts sharply shortly after the learning rate is decayed. The IrDecay enjoys great popularity due its simplicity and general effectiveness.
Common beliefs in how IrDecay works are derived from the optimization analysis in (Stochastic) Gradient Descent (LeCun et al., 1991; Kleinberg al., 2018). They attribute the effect of an initially

---

Under review as a conference paper at ICLR 2020
60
10
50
Learning Rate
caor (%)
10
40
18-layer
30
10-3
ResNet-18 18
ResNet-34
34-layer
20
50
30
40
50
epoch
iter. (1e4)
(a) Learning rate decay strategy
(b) Figure taken from He et al. (2016)
Figure Training error in (b) is shown by thin curves, while test error in (b) by bold curves.
explanation
perspective
initially large
supported
decay
supported
LeCun et (1991)
optimization optimization
accelerates training
avnids oscillation
Kleinberg et al. (2018)
escapes bad local minima
converges to local minimum more complex patterns
Proposed
pattern complexity
litting noisy data
Table I: Comparison of explanations on why IrDecay helps training neural networks. The column "supported" means whether the explanation supported by the empirical experiments in this paper.
large learning rate to escaping spurious local minima accelerating training and attribute the effect of decaying the learning rate avoiding oscillation around local minima. However, these common beliefs are insufficient to explain our empirical observations from a series of carefully-designed experiments in Section 4.
In this paper, we provide alternative view: the magnitude of the learning rate is closely related to the complexity of learnable patterns. From this perspective, we propose novel explanation for the efficacy of IrDecay: an initially large learning rate suppresses the memorization of noisy data while decaying the learning rate improves the learning of complex patterns. This validated on trrefully-conssructee dataset with tractable pattern complexity. The pattern complexity in real- world datasets is often intractable. We thus validate the explanation by testing its implication on real-world datasets. The implication that additional patterns learned in later stages of IrDecay more complex and thus less transferable across different datasets, is also justified empirically. A comparison between the proposed explanation and the common beliefs summarized in Table 1. Our explanation is supported by carcfully-desieee experiments and provides a new perspective on analyzing learning rate decay.
The contribution of this paper is two-fold:
We demonstrate by experiments that existing explanations of how IrDecay works are insuf- ficient in explaining the training behaviors in modern neural networks.
We propose novel explanation based on pattern complexity, which is validated on a dataset with tractable pattern complexity, and its implication is validated on real-world datasets.
The explanation also suggests that complex patterns are only learnable after learning rate decay. Thus, when the model learns all simple patterns, but the epoch to decay has not reached, immediately decaying the learning rate will not hurt the performance. This implication is validated in Section A.
2 RELATED WORK
2.1
UNDERSTANDING THE BEHAVIOR OF SGD
Recently, researchers reveal the behavior of SGD from multiple perspectives (Li et al., 2019; Man- galam & Prabhu, 2019; Nakkiran al., 2019). They respect the difference among data items rather
2

---

Under review as a conference paper at ICLR 2020
than treat them as identical samples from a distribution. They study the behavior of SGD in given dataset. In Mangalam & Prabhu (2019), they show that deep models first learn easy examples classifiable by shallow methods. The mutual information between deep models and linear models is measured in Nakkiran et al. (2019), which suggests deep models first learn data explainable by linear models. Note that they are not relevant to learning rates. Li et al. (2019) analyze toy problem to uncover the regularization effect of an initially large learning rate. Their theoretical explanation is, however, based on a specific two-layer neural network they design. Different from these works, Section 5 studies the behavior of SGD induced by IrDecay in modern WideResNet (Zagoruyko & Komodakis, 2016), finding that learning rate decay improves learning of complex pattern.. We for- mally define pattern complexity by expected class conditional entropy, while the measure of pattern complexity in Mangalam & Prabhu (2019); Nakkiran et al. (2019) relies on an auxiliary model.
2.2 ADAPTIVE LEARNING RATE METHOSS
Adaptive learning rate methods such as AdaGrad (Duchi et al., 2011), AdaDelta (Zeiler, 2012), and ADAM (Kingma & Ba, 2015) are sophisticated optimization algorithms for training modern neural networks. It remains an active research field to study their behaviors and underlying mech- anisms (Reddi et al., 2018; Luo al., 2019). However, we focus on learning rate decay SGD) rather than on the adaptive learning rate methods. On the one hand, SGD is the de facto training algorithm for popular models (He et al., 2016; Huang et al., 2017b) while IrDecay is not common in the adaptive methods; On the other hand, many adaptive methods are not simple as SGD and even degenerate in convergence in some scenarios (Wilson et al., 2017; Liu et al., 2019). We choose to study SGD with IrDecay, without introducing adaptive learning rate to keep away from its confounding factors.
2.3 OTHER LEARNING RATE STRATEGIES
Besides the commonly used IrDecay, there are other learning rate strategies. Smith (2017) proposes a cyclic strategy, claiming to dismiss the need for tuning learning rates. Warm restart of learning rate is explored in Loshchilov & Hutter (2017). They achieve better results when combined with Snapshot Ensemble (Huang et al., 20) These learning rate strategies often yield better results at the cost of additional hyperparameters that are not intuitive. Consequently, it is still the de facto to decay the learning rate after pre-defined epochs as in Figure (a). We stick our analysis to IrDecay rather than other fancy ones because of its simplicity and general effectiveness.
2.4 TRANSFERABILITY OF DEEP MODELS
Training a model on one dataset that can be transferred to other datasets has long been a goal of AI researches. The exploration of model transfcrability has attracted extensive attention. In Oquab et al. (2014), deep features trained for classification are transferred to improve object detection successfully. Yosinski al. (2014) study the transferability of different modules in pre-trained net- works, indicating that higher layers are more task- ssecifpciiiic and less transferable across datasets. By varying network architectures, Kornblith al. (2019) show architectures with better ImageNet ac- curacy generally transfer better. Raghu et al. (2019) explore transfer learning in the field of medical imaging to address domain-specifer difficulties. Different from these works that only consider the transferability of models after training, we investigate another dimension of model transferability in Section 6: the evolution of transfcrability during training with IrDecay.
3 COMMON BELIEFS IN EXPLAINING LRDECAY
3.1 GRADIENT DESCENT EXPLANATION
The practice of Decaa in training neural networks dates back to LeCun et al. (1998). The most popular belief in the effect of IrDecay comes from the optimization analysis of Gradient Descent (GD) (LeCun et al., 1991). Although SGD is more practical in deep learning, researchers are usually satisfied with the analysis of GD considering that SGD is stochastic variant of GD.
3

---

Under review as a conference paper at ICLR 2020
Figure 2:
Gradient Descent explanation. From left to right: 1) learning rate is small enough to converge around a minimum, 2) moderate so that it bounces among minima, 3) too large to converge.
Specifically, LeCun et al. (1991) analyze the property of a quadratic loss surface which can be seen second-order approximation around a local minimum in nonconvex optimization. Learning rates are characterized by the relationship with eigenvalues of the Hessian at local minimum. Denote the learning rate, the Hessian, an eigenvalue of H, and v an eigenvector of The behavior of the network along the direction v can be characterized as (1 ny)"v,, with & the iteration number. Convergence in the direction of v requires 0 < 1) < 2/X, while 7j > 2/> leads to divergence in the direction of v. If < < 2/ X holds for every eigenvalue of the Hessian, the network will converge holds for some directions but not for directions, the network will diverge in some directions and thus jump into the neighborhood of another local minimum (Figure middle). the learning rate too large, the network will not converge (Figure right). particular, when oscillation happens, it means the learning rate is too large and should be decayed. The effect of IrDecay hence to avoid oscillation and to obtain faster convergence. Note LeCun et al. (1991) only analyze simple one-layer network. It may not hold for modern neural networks (see Section 4.1).
quickly (Figure left).
3.2 STOCHASTIC GRADIENT DESCENT EXPLANATION
Another common belief is the Stochastic Gradient Descent explanation, arguing that "with high learning rate, the system unable to settle down into deeper, but narrower parts of the loss func- tion." Although it is common, this argument has not been formally analyzed until very recently.
15.0
15.0
50-
12.5
12.5
40
5.-
5.0-
2.5-
10
M
-2
Figure 3: SGD explanation (taken from Kleinberg et al. (2018)). The first plot: an initially large learning rate helps escape spurious local minima. From the second to the fourth plots: after more rounds of learning rate decay, the probability reaching the minimum becomes larger.
Under some assumptions, Kleinberg et al. (2018) prove SGD is equivalent to the convolution of loss surface, with the learning rate serving as the conceptual kernel size of the convolution. With an appropriate learning rate, spurious local minima can be smoothed out, thus helping neural networks escape bad local minima. The decay of learning rate later helps the network converge around the minimum. Figure 3 is intuitive one-dimensional example. The first plot shows that a large learning rate helps escape bad local minima in both sides. The IrDecay subsequent plots increases the probability of reaching the global minimum. Although intuitive, the explanation requires some assumptions that may not hold for modern neural networks (see Section 4.2).
hhttp://c233nn githuu.. 1o/neural networks-33 anneal

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS AGAINST EXISTING EXPLANATIONS
Although the (Stochastic) Gradient Descent explanations in Section 3 account for the effect of IrDe- cay to some extent, in this section, we show by carefully-designe experiments that they are insuffi cient to explain the efficacy of IrDecay in modern neural networks. In all the experiments except for Section 6, we use a modern neural network named WideResNet (Zagoruyko Komodakis, 2016). It is deep, wide, nonconvex, and suitable for datasets like CIFARIO (Krizhevsky & Hinton, 2009).
4.1 EXPERIMENTS AGAINST THE GRADIENT DESCENT EXPLANATION
We train a WideResNet on CIFARIO dataset with GD, decay the learning rate at different epochs, and report the training loss (optimization) as well as the test accuracy (generalization) Figure 4. WideResNet and CIFARIO 10 are commonly used for studying deep learning (Zhang et al., 2017). CIFARIO is not too large so that we can feed the whole dataset as single batch using distributed training. computing the exact gradient rather than estimating it in mini-batches. Experiments show that IrDec brings negligible benefit to either optimization or generalization. No matter when the learning rate is decayed, the final performances almost the same. The instability in the beginning is related to the high loss wall described in Pascanu et al. (2013), which is not the focus of this paper.
70-
70
70
70
70
Test Accuracy
60
dacay
50-
50-
50-
60
40
40
40
40
200
20
200
20
20
10-
10
10
2.5
2.5
2.5
2.5
2.5
Training Loss
2.0
2.0
2.0
2.0
2.04
1.5
1.5
1.5
1.0
1.0
0.3
0.5
Jacay
J Jaccy
dvay
1
Ewoy
no deea
8.0
0.0
Figure 4: Training of WideResNet CIFARIO with Gradient Descent. X-axis indicates the number of epochs (in 10*), Arrows indicate the epoch with learning rate decay.
The above observation contradicts directly with the GD explanation in Section 3.1. The contradic- tion arises from the fact that LeCum et al. (1991) only analyze simple linear networks, and no wonder the explanation fails in modern non-linear deep networks. Recent studies (Keskar al., 2017; Yao et al., 2018) 8) reveal that large-batch training of modern networks can lead to very sharp local minima. Gradient Descent (the extreme of large batch training) can lead even sharper local minima. In Fig- ure 5, we calculate the largest ten eigenvalues" of the Hessian as well as the convergence interval (0 7) < 2/A) for each eigenvalue for trained WideResNet. The top eigenvalues reach the order of = 200. By contrast, cigenvalues of simple networks in LeCun et al. (1991) often lie in j0, 10] (Figure in their original paper). The spectrum of eigenvalues in modern networks is very different from that in simple networks analyzed by LeCun et al. (1991): the Hessian of modern networks has a much larger spectral norm.
The GD explanation in Section 3.1 attributes the effect of IrDecay to avoiding oscillation. Oscillation means there is a small divergence in some directions of the landscape so that the network bounces among nearby minima. However, the divergence factor - ιyλ for the largest eigenvalue (s 200) is too large even for a small growth of learning rate. Thus, the learning rate is either small enough to converge in local minimum large enough diverge. It is hardly possible to observe the oscillation in learning curves (Figure 2 middle), and diverging learning curves (Figure 2 right) can be discarded during hyperparameter tuning. Therefore, only stable solutions are observable where is small enough (Figure 2 left), leaving no necessity for learning rate decay. Indeed, when the
Thaans to the advances of Xu al. (2018); Yao al. (2018), we can compute the eigenvalues directly.
5

---

Under review as a conference paper at ICLR 2020
learning rate is increased mildly, immediately observe diverging learning curves (Section A.2). In short, the GD explanation cannot explain the effect of IrDecay in training modern neural networks.
90 65
I7c
0.00
interva
Test Accuracy
23 38
Training Loss Sanimengg
Eigenvalue
too
0.66
20
decaa
0.0
Conver
Jdecay
Eigenvalue Index
Figure 5: The largest ten eigenvalues À (blue curve) and converge intervals (0, 2 (bar) for WideResNet trained with Gradient Descent.
Figure 6: Expected behavior (but not observed) induced by the SGD explanation: best perfor- mances before and after decay are comparable.
4.2 EXPERIMENTS AGAINST THE STOCHASTIC GRADIENT DESCENT EXPLANATION
We follow the experiment setups in Section 4.1, but replace GD with SGD in Figure 7. According to the SGD explanation in Section 3.2, the effect of learning rate decay is to increase the probability of reaching a good minimum. If it is true, the model trained before decay can also reach minima, only by smaller probability compared to the model after decay. In other words, the SGD explanation indicates the best performances before and after decay are the same. predicts learning curves like Figure 6. However, Figure 7 does not comply with the SGD explanation: the best performances be- fore and after IrDecay are different by a noticeable margin. Without IrDecay (the rightmost column in Figure 7), the performance plateaus and oscillates, with no chance reaching the performance of the other columns after decay. The performance boost after learning rate decay is widely observed (Figure ((b) for example). However, possibly due to the violation of its assumptions (Kleinberg et al., 2018), the SGD explanation cannot explain the underlying effect of IrDecay.
Test Accuracy
85
85
85
85
85
80-
MW
80:
23
20
70
70
70
70
60
60
55-
35-
55-
55
55+
8 PP
1.4
1.4-
1.4
1.4
1.4
Training Loss
1.2
1.0
1.0
1.0
1.0
1.0
80.8
0.5
0.8
0.5
0.8
0.6
0.2
0.4
ddoe
0.4
0.2
4kay
014
0.2
1 Aeny
0.4
0.2
dase
0.4
0.2
doay
Figure 7: Training WideResNet on CIFARIO with SGD. X-axis indicates the number epochs. Arrows show the moment of learning rate decay. The rightmost plots show results without decay.
5 AN EXPLANATION FROM THE VIEW OF PATTERN COMPLEXITY
Section uncovers the insufficiency of common beliefs in explaining IrDecay. We thus set off to find a better explanation. Mangalam & Prabhu (2019); Nakkiran et al. (2019) reveal that SGD (without learning rate decay) learns from easy to complex. As learning rates often change from large small in typical learning rate strategies, we hypothesize that the complexity of learned patterns related to the magnitude of learning rates. Based this, provide novel explanation from the view of pattern complexity: the effect of learning rate decay to improve the learning of complex patterns while the effect of initially large learning rate is to avoid memorization of noisy

---

Under review as a conference paper at ICLR 2020
data. To justify this explanation, carefully construct dataset with tractable pattern complexity, and record model accuracies in simple and complex patterns separately with and without IrDecay.
5.1 PATTERN SEPARATION 10 (PS10) DATASET WITH TRACTABLE PATTERN COMPLEXITY
The explanation we propose involves pattern complexity, which is generally conceptual and some- times measured with the help of a simple auxiliary model as in Mangalam & Prabhu (2019); Nakki- ran et al. (2019). Here we try to formalize the idea of pattern complexity: the complexity of a dataset defined the expected class conditional entropy: C(((n;, y)) denotes the entropy functional. The complexity of patterns depends on the complexity of the dataset they belong Higher C' means larger complexity because there averagely more patterns each class to be recognized (consider an animal dataset with 10 subspecies in each species vs. an animal dataset with 100 subspecies in each species).
y. (arl))) where H
Channel
Category 3
50%
Simple
1 data 1
50%
Complex
(a) Simple Patterns
(h) Complex Patterss
(c) Data Composition
Figure 8: The PS10 dataset. (a) Simple patterns: 10 patterns per category, complexity log, 10. (b) Complex patterns: 100 patterns per category, complexity log, 100. (c) Data composition: half of the data only contain simple patterns while another half only contain complex patterns.
Equipped with the formal definition complexity, we construct a Pattern Separation 10 (PS10) dataset with ten categories and explicitly separated simple patterns and complex patterns. We first generate a simple sub-dataset together with a complex sub-dataset IR3 As shown Figure 8(a) and Figure 8(b), patterns visualized colors because they lie in R The category label can be identified by either simple patterns or complex patterns. We then merge the two sub-datasets into dataset. The merging method in Figure 8(c) is specifically designed such that the simple subset and complex subset are fed into different channels of the WideResNet. This mimics the intuition of patterns as the eye pattern and the nose pattern have different locations in an image human face. To be compatible with the sliding window fashion of convolutional computation, we make patterns the same across spatial dimensions of height and weight have the same image size as CIFAR1O.
Stage
Stage
Stage
leaming rate in stage
leaming stage
learning Tile stuge
13
Accurcr
Ucody
Figure 9; Experiments with IrDecay and without IrDecay (constant learning rates) w.r.t accuracies in different patterns. From left to right: Train with IrDecay: Train with a constant learning rate equal to the learning rate in Stage 1, 2, and 3 of IrDecay, respectively. The X axis shows the epoch number.

---

Under review as a conference paper at ICLR 2020
5.2 THE EFFECT OF DECAY: IMPROVE LEARNING OF MORE COMPLEX PATTERNS
To reveal the effect of decaying the learning rate, we compare experiments with and without IrDecay. For those without IrDecay, we set the learning rates equal to the learning rate of each stage in IrDecay. We measure not only the total accuracy but also the accuracies on simple and complex patterns separately. These accuracies are plotted Figure 9.
The first plot in Figure 9 clearly shows the model first learns simple patterns quickly. The boost in total accuracy mainly comes from the accuracy gain on complex patterns when the learning rate is decayed. Plots 2, 3, and 4 show the network learns more complex patterns with smaller learning rate, leading to the conclusion that learning rate decay helps the network learn complex patterns.
5.3 THE EFFECT OF AN INITIALLY LARGE LEARNING RATE: AVOID FITTING NOISY DATA
Figure seems to indicate that an initially large learning rate does nothing more than accelerating training: in Plot 4, a small constant learning rate achieve roughly the same accuracy compared with IrDecay. However, by adding 10% noisy data to mimic real-world datasets, we observe some- thing interesting. Figure 10 shows the accuracies data when we add noise into the dataset. Plot 2 in Figure 10 shows an initially large learning rate helps the accuracy on complex patterns. Plot in Figure 10 further shows the accuracy gain on complex patterns comes from the suppression of fitting noisy data. (Note that a larger accuracy on noisy data implies overfitting the noisy data, which undesirable.) In short, the memorizing noisy data hurts the learning of complex patterns but can be suppressed by an initially large learning rate.
simple patterns, complex patterns, and noise
Simple Accuracy
Complex Accuracy
Nose Accuracy
110-
100
100
60
20-
15
20
50
20
60
with decay keep le small
large with dleay keep small
keep Ir small
Figure 10: Comparison between IrDecay and a constant small learning rate on the PS10 dataset with 10% noise. Accuracies on simple patterns, complex patterns, and noise data are plotted respectively.
Empirically, Li et al. (2019) report that an initially large learning rate with decay outperforms a small and constant learning rate. They suspect that the network starting with an initially small learning rate will be stuck at some spurious local minima. Our experiments provide an alternative view that spurious local minima may stem from noisy data, And the regularization effect of an initially large learning rate is to suppress the memorization of noisy data.
THE IMPLICATION OF LRDECAY ON MODEL TRANSFERABILITY
Section 5 examines the proposed explanation on the PS10 dataset. Now we further validate the explanation on real-world datasets. Because there are no clearly separated simple and complex patterns in real-world datasets, it is difficult to directly validate the explanation. The proposed explanation suggests that SGD with IrDecay learns patterns of increasing complexity. Intuitively, more complex patterns are less transferable, harder to generalize across datasets. Thus an immediate implication is that SGD with IrDecay learns patterns of decreasing transferability. We validate by transfer- learning experiments on real-world datasets, to implicitly support the proposed explanation.
The transferability is measured by transferring model from ImageNet to different target datasets. To get models in different training stages, we train a ResNet-50 on ImageNet from scratch and save checkpoints of models in different stages. The learning rate is decayed twice, leading to three
8

---

Under review as a conference paper at ICLR 2020
stages. Target datasets for transferring are: (1) Caltech256 (Griffin et al., 2007) with 256 general object classes; (2) CUB-200 (Wah et al., 2011) with 200 bird classes; (3) MITIndoors (Quattoni & Torralba, 2009) with 67 indoor scenes; (4) Sketch250 (Eitz et al., 2012) with sketch painting in 250 general classes. Sketch250 is the most dissimilar to ImageNet because it contains sketch paintings.
We study two widely-used strategies of transfer learning: "fix" (ImageNet snapshot models are only used as fixed feature extractors) and "finetune" (feature extractors are jointly trained together with task-specific layers). Let acc; denotes the accuracy of stage snapshot model on ImageNet and tace; denotes the accuracy of transferring the snapshot to the target dataset, then the transferability of additional patterns learned in stage i is defined as face;-taer:
acc;-ace;-1 , 2, 3. By definition, the transferability of patterns from ImageNet to ImageNet is 1.0, complying with common sense. The transferability plotted in Figure 11. Table 2 contains the accuracies used to compute
Fintume
Stage
Stage
Stage
Stage
0.8
0.8
Trandeenaity
Transfrnaatty
0.0
0.0
0.44
0.2
0.2
Imagehitt
Calteck556
CUB-200
MiTinboors
Skeech750
Imageheet
Caltec1556
CUR-200
MiTindoors Skwtsk050
Dataset
Dataset
Figure 11: Transfcrability of additional patterns learned in each stage w.r.t different target datasets.
In all experiments, we find that the transferability of additional patterns learned in stage 3 is less than that stage Besides, in Sketch250 dataset, the transferability of additional patterns learned in stage is negative. These findings support our claim that additional patterns learned in later stages of IrDecay are more complex and thus less transferable. They also suggest deep model-zoo developer provide pre-trained model snapshots in different stages that downstream users select the most transferable snapshot model according to their tasks.
7 CONCLUSION
In this paper, we dive into how learning rate decay (IrDecay) helps modern neural networks. We uncover the insufficiency of common belief's and propose novel explanation: the effect of decaying learning rate is to improve the learning of complex patterns, and the effect of an initially large learning rate is avoid memorization of noisy data. It is supported by experiments with tractable pattern complexity as well as on rerl-world datasets. It would be interesting to further bridge the proposed explanation and the formal analysis of optimization procedure.
dataset
REFERENCES
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):5549-15588 2019.
Yoshua Bengio. Practical Recommendations for Gradient- Basen Training of Deep Architectures. In Neural Networks: Tricks of the Trade: Second Edition, Lecture Notes in Computer Science, pp- 437-478. Berlin, Heidelberg, 2012. ISBN 978-3-642-35289-88
Nils Bjorck, Carla Gomes, Bart Selman, and Kilian Q Weinberger. Understanding Batch Normal- ization. In NIPS, pp. 7694-7705, 2018.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 120112222 2011.
Mathias Eitz, James Hays, and Mare Alexa. How do humans sketch objects? ACM ToG, 31(4): 44 2012.

---

Under review as a conference paper at ICLR 2020
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. Snapshot ensembles: Train get for free. In ICLR, 2017a.
Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolutional networks, In CVPR, volume pp. 3, 2017b.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ICML, pp. 448-456, 2015.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015.
Bobby Kleinberg, Yuanzhi Li, and Yang Yuan. An Alternative View: When Does SGD Escape Local Minima? In ICML. pp. 2698-2707, 2018.
Jonas Kohler, Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann, Ming Zhou, and Klaus Neymeyr. Exponential convergence rates for Batch Normalization: The power of length- length-direction decoupling in non-convex optimization. AISTATS, pp. 806 815, 2019.
Simon Kornblith, Jonathon Shlens, and Quoc V. Le. Do better imagenet models transfer better? In CVPR, pp. 2661-2671, 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Yann LeCun, Ido Kanter, and Sara A. Solla. Second Order Properties of Error Surfaces: Learning Time and Generalization. In NIPS, pp. 918-924, 1991.
Yann A. LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. Efficient BackProp. In Neural Networks: Tricks of the Trade: Second Edition, Lecture Notes in Computer Science, pp. 9-48. Berlin, Heidelberg, 1998. ISBN 778-3-642-35288-33
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks. In NeurlPS, 2019.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In ICLR, 2019.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging Common Assumptions in the Unsupervised Learn- ing of Disentangled Representations. In ICMI., pp. 41 4424 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive Gradient Methods with Dynamic Bound of Learning Rate. In /CLR, 2019.
Karttikeya Mangalam and Vinay Uday Prabhu. Do deep neural networks learn shallow learnable examples first? ICML workshop. 2019.
Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L. Edelman, Fred Zhang, and Boaz Barak. SGD on Neural Networks Learns Functions of Increasing Complex- ity. In NeurlPS, 2019.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and Transferring Mid-Level Image Representations using Convolutional Neural Networks. In CVPR, pp. 1717-1724, 2014.
10

---

Under review as a conference paper at ICLR 2020
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In ICML, pp. 1310-1318, 2013.
A. Quattoni and A. Torralba. Recognizing indoor scenes. In CVPR, pp. 413-420, June 2009.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding Transfer Learning for Medical Imaging. NeurIPS, 2019.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the Convergence of Adam and Beyond. In ICLR, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, and Michael Bernstein. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252 2015.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How Does Batch Nor- malization Help Optimization? In NIPS, pp.
2018.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen monyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419): 140-111444 2018.
Leslie N. Smith. Cyclical learning rates for training neural networks. In WACV, pp. 464-472, 2017.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to Sequence Learning with Neural Net- works. In NIPS, pp- 3104-31 12, 2014.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The Marginal Value of Adaptive Gradient Methods in Machine Learning. In NIPS, pp. 4148-4158, 2017.
Peng Xu, Bryan He, Christopher De Sa, Ioannis Mitliagkas, and Chris Re. Accelerated Stochastic Power Iteration. In AISTATS, pp. 58 67, 2018.
Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. Hessian-based Anal- ysis of Large Batch Training and Robustness to Adversaries. In NIPS, pp. 4949-4959, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NIPS, pp. 3320-3328, 2014.
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In BMVC, 2016.
Matthew D. Zeiler. ADADELTA: An Adaptive Learning Rate Method. arXiv:1212.5701 [cs),
December 2012.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 AUTODECAY
Experiments in Section 5.2 implies that not all complex patterns are learnable under constant learning rate. The training under a certain learning rate has no effect when the loss plateaus. This indicates we expedite the training process by killing the over- over-training of each stage (decay the learning rate when the loss plateaus) with little influence on the performance. To validate the implication, we propose AutoDecay to shorten the useless training and check if the performance of the model can be untouched. In Figure 7, it appears obvious to decide the optimal moment to decay when we have big picture of the training process. The problem is, however, how can we make decision to decay depending on the current and past observations. It is non-trivial problem given that the statisties exhibit noticeable noise.
A.1.1 PROBLEM FORMULATION
We formalize the observed training loss into two parts: ε(t) {(t) + c(t), with E(t) the ground truth loss (unobservable) and e(t) the noise introduced by SGD. Here t indicates the training process (typically the epoch number) and takes value in A = 21, 1,,,3,...., ...). To simplify the problem, we assume e(t) is independent with t and e(t) is independent of e(t') + t) in SGD, The nature of noise gives rise to the zero-expectation property E e(t) = O. Denote o² Var <(t) the variance of the noise. Due to the noise of SGD, the observed training loss usually vibrates in short time window but decreases in long time window. Our task is to find out whether the loss value is stable in the presence of noise.
A.1.2 PROBLEM SOLUTION
Exponential Decay Moving Average (EDMA) with Bias Correction. Observations with lower variance are more trustworthy. However, there is nothing we can do about the variance of É(t). We consider computing low-variance statistic about f(t). We adopt moving average with bias correction(Knggna & Ba, 2015). Let g(t) be the moving average of E(t) and g(t) be the moving average of ((t). The explicit form is in Equation 1, where 3 E
(0, 1) is the decay factor in EDMA.
E1
t-1
fil(t- i)
1-3 6-1
g(t)
===
Bil(t (((t-)),t2)
E1=0 8
1- 3
(1)
g(t)
E1-1
84 1)
1-
Σ
t-1
3tl(t - 1),
Σ1- B
1- 3
1=0
The recursive (and thus implicit) form is in Equation 2. It enables us to compute the statistic 9 online (without storing all the previous ( < ()) at the cost of maintaining f()).
f(0) = 0, (() = Bf(( 1) + (1- 3)((1) = g(() = f(t)
(1
(221 1)
1- Br
(2)
i(1)
f(0) o. f0) Bf( - +1 3)((1) 9(1)
1- Br 1)
As g() is a linear combination of (é(i)) < t). it is easy to show g(t) is unbiased:
Eg(t) = Eiz
56-1 g' 1)
Σ 1-1
B' 1)
= gil(t (t)
1=0
Ei z
Σ ビ 8i
Σ gi
The variance of g() is
12

---

Under review as a conference paper at ICLR 2020
Varğ(t) (1- Var (1-
gli(t 1) (1 g) 32i
1- 1-11+ 8r
1+ 1+ 11-11- B² o²
(3)
(1 31)2
The fact that 3 E (0, 1) indicates Varğ(t) is monotonically decreasing. Typically / 0.9 (Fig- ure 13), and the variance can rapidly converge to 0.05a², much smaller than the variance of the noise. ((t) well represents the unobservable g(t). If {(t) gets stable, we shall observe that g(t) is stable, too.
Criterion of Being Stable. only want decay the learning rate when the loss plateaus, i... when the loss is stable. For observed values of = (ū(i))i - W + 1 < i S t) within the window size of W. we call them stable max G G-min < 17. where e a small constant that prevents zero-division error, and 17 indicates the tolerance of variation.
min G+t
Criterion of Significant Drop. When we keep decaying the learning rate, there comes time when the learning rate is too small and the network cannot make any progress. When it happens, we should terminate the training. Termination is adopted when their is no significant drop between the stable
value and the original value (0)) To be specific, the criterion of significant drop is g(t)+ < S. where is a small constant that prevents zero-division error, and indicates the degree of drop.
g(0)+c
The entire procedure of AutoDecay is described in Figure 12.
e(t)
Keep
X
Terminate
x
g(t) W - G stable? 2
(t) drop? V Decay
Figure 12: Decision Procedure of AutoDecay. The counter is reset to at the action of "Decay".
A.1.3 EXPERIMENTS
We try AutoDecay on ImageNet (Russakovsky et al., 2015) to test whether it can expedite the train- ing without hurting the performance. We are not trying to set up a new state-of-the-ar record. We train a ResNet-50 model on ImageNet following the official code of PyTorch. The only change is we replace the StepDecay strategy with the proposed Auca strategy. Each experiment costs roughly two days with 8 TITAN X GPUs. The results in Table 14 show that AutoDecay can shorten the training time by 10% without hurting the performance (even bringing slight improvement), successfully vaidates the proposed explanation in this paper.
ImageNet
85
1-8
Method
65.
83
1-
epochs
topl
top5
StepDecay AutoDecay
90
75,80
92.76
75.91
92.81
62
61
03
0
20
10
t
80
100
Figure 14: Results of AutoDecay.
Figure 13: Variance reduction when
8 = 0.9
A.2 LARGER LR LEADS TO DIVERGENCE IN GD FOR MODERN NEURAL NETWORKS
When increase the learning rate mildly for Gradient Descent, we immediately observe diverging learning curves (Figure 15), which echos with the reason mentioned in Section 4.1 why the Gradient Descent explanation fails to work in modern neural networks: modern neural networks have very
13

---

Under review as a conference paper at ICLR 2020
large spectrum local minimum, and small growth of learning rate lead to divergence. other words, training modern neural networks with GD must use a small enough learning rate, dismissing the value of learning rate decay.
70-
70
70-
70-
70-
(0)
60
00
50
40
30
30
30
30
30
20
20
1000
1000
2.5
2.5
2.5
2.5
2.5
2.0
200
1.5
1.5
1.5
1.0
1.0
1.0
1.0
10
0.0
1.0
0.01
1930
0.0
0.0
Figure 15: Training of WideResNet CIFARIO by Gradient Descent with a mildly larger learning
rate. X-axis indicates number of epochs. Arrows and texts show the moment of learning rate decay.
A.3 ACCURACIES TO COMPUTE THE TRANSFERABILITY IN SECTION
Finetune
Fix
Dataset
stagel
stage2
stage2
stage3
stage3
stage1
stage2
stage2
stage3
stage3
ImageNet
54.01
69.33
1.00
75.91
1.00
54.01
69.33
1.00
75.91
1.00
Caltecr256
77.87
82.73
0.32
84.20
0.22
74.10
79.79
0.37
81.95
0.33
CUB
75.89
79.74
0.25
81.34
0.24
58.11
63.95
0.38
66.09
0.33
MITIndoors
72.91
77.09
0.27
78.73
0.25
61.34
67.54
0.40
69.78
0.34
Sketch250
77.80
79.92
0.14
80.64
0.11
64.34
64.58
0.02
64.18
-0.06
Table 2: Accuracy and transferability of ImageNet models in different stages. Normal values indi- cate accuracy and bold values indicate transfcrability.
14


Reviews:
[{"review_id": "rJxKrLM3KS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates the role of learning rate decay in neural network training. While there are prevalent ideas of how/why learning rate decay help both optimization and generalization of neural networks, this work proposes interpretation based on pattern complexity. The mechanism the paper proposes is that initial learning rate helps ignore noise in the beginning and decayed learning rate help to learn complex patterns. \n\nThe question the paper tackles is a very important question in understanding deep learning that requires careful study. As the learning rate schedule benefits neural network models beyond specific domain, this question has high significance and potential impact.\n\nThe authors propose a view from a pattern complexity. I think this view point is well motivated in the light of Li et al. (2019). The artificial task constructed by authors reveals that with different phases of learning rates, different types of complexity are being learnt. Also transfer learning tasks also reveals that one might want to use models before dropping the learning rate for purpose of transfer learning.\n\nWhile the question studied is of high importance, I am not confident that the methods presented justifies the claims in the paper. Especially authors analysis on how previous understanding of learning rate decay is flawed doesn’t seem fully supported. With current submission I slightly lean towards rejecting. \n\nOne reason is that overall the details of experiments are not specific enough that I personally wouldn’t feel comfortable reproducing the results. For example, in Section 4, was data augmentation or weight decay used? What was the mini-batch size used for SGD experiments? Which learning rate is used for  large learning rate and to which learning rate was it decayed to? I think even one is using full batch GD, the performance shown in Figure 4 which is just above 60% test accuracy is quite low for models like WideResNet.\n\nAlso I do not agree with the claim that SGD explanation leads to training curves as in Figure 6. The plot indicates large learning rate / lower learning rate have about the same probability of reaching the 'global minima’. However as described in section 3.2, the role of lowering the learning rate should be increasing the probability of global minima. To me experiments shown in Figure 7, supports both explanations from Lecun et al (1991) Kleinberg et al. (2018) described in Table1. \n\nFew extra comments: \n\nOne question I have regarding transferability analysis. Kornblith et al. (2019) showed that better Imagenet models transfers better. However section 6 shows that  transferability is higher for stage 2 which has higher error. How do you reconcile this discrepancy? \n\nI don't quite comprehend the motivation for putting two complexity patterns in disjoint channels. Especially the comment “This mimics the intuition of patterns as the eye pattern and the nose pattern have different locations in an image of human face.” is unclear. Could you elaborate?\n\nFigure 15 seems incomplete. Should there be arrows for decay points? Please also include details of those experiments too. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SygcfD6htS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions:\nThis paper investigates the use of learning rate decay in deep neural networks.  The main contribution is an empirical analysis trying to understand lr decay. Authors claim that the high-learning rate phase provides regularization and prevents the network to fit/memorize noisy data initially to focus on simple patterns. \n\nAuthors design a set of experiments showing that lr decay allows SGD to first fit ‘simple patterns’ instead of more complex/noisy one. They also show that the model transferability decreases through training.\n\nIn addition to those experiments, authors provide experimental results which aims at contradicting common beliefs on lr decay.\n\nNovelty/Significance:\nMy main concern is about the novelty/significance of the paper.  Similar argument has already been made in (Nakiran et al., 2019) or (Li et al. 2019), which are cited in the paper, and other works such as “On the Spectral Bias of Neural Networks”. In addition, prior to this work, the paper “Three factors Influencing Minima in SGD” empirically showed that the noise in SGD (which is controlled by the learning rate) prevents memorization (see Figure 6 in their paper). Although the latter work did not focus the lr decay, it is not clear to me how the main argument of the current paper differs from it. \n\nAdditional comments:\n-\tIt is unclear to me why the model the model before and after decay should have the same training performance in section 4.2. This would assume a specific geometry of the loss function (i.e. that the loss surface is not significantly narrower so that you can reach it with a high-learning rate), but authors do not provide evidence of it.\n-\tIt would be nice to provide more details about the experimental settings. What is the base learning rate? Did you try different learning rate? Do you use with SGD-momentum, weight-decay?\n-\tHow to combine the current explanation of lr decay with the observation that learning rate warm-up has been a successful way to train neural network in the large batch setting?\n-\tThe transferability experiment confounds two factors: the number of iteration steps   and the learning rates. Does training a model with low learning rate, but with a number a step equal to high learning rate lead to lower transferability?\n-\tIt would be nice to validate the hypothesis on more datasets/models.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lvGJSE9H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper investigates the way decaying the learning rate helps the training of neural networks. First the paper discusses about other existing hypothesis such as the “Gradient Descent Hypothesis” by Lecun et al 1991 and SGD explanation by Kleinberg et al 2018. Then the paper tries to find contradicting examples against those two hypothesis with experiments. Then they propose their explanation which suggests that initially fitting noisy data and then decaying it helps it to learn more complex data. Then the paper tries to experimentally explain why the other explanations fail and theirs is better.\n\nThis paper is very badly written. The authors should definitely rethink about the organization of the paper. The first pages is mostly about the background material with figures taken from other papers. The terminology that is being used in this paper is very vague. They used the term complex patterns in the paper, but don’t even explain it until page 6. When the paper explains it, still the notion itself how to compute it in a tractable way is a bit vague for neural networks.\n\nThis paper proposes all those different explanations of how SGD works just as if they are completely orthogonal. However, for example learning both the proposed explanation in this paper and the fact that learning rate decay improves stability can both be true. \n\nThe experimental arguments are quite vague in this paper. The authors should give more details about their experimental setup, for example what is the starting learning rate, ending learning rate, details of scheduling, what type of distributed training method (sync or async SGD?) and etc… is missing. Without those details it is very difficult to interpret the experimental conclusions in for example section 4.1. In Figure 5, what does the converge interval mean? In Figure 7 and 4 all the curves are so similar, why so?\nThe arguments that learning rate decaying helps models to learn complex patterns and without decay the models can not learn those is a very strong claim and not very well supported in this paper.\n\nThe details of how the PS10 dataset is constructed is somewhat vague. In terms of arguments that decaying learning rates help the neural networks to learn simpler examples first and the harder ones have already been done in other papers such as Li et al 2019 which is also cited in this paper. It is not completely clear what this paper contributes over those other existing papers :)\n\n\nQuestions:\nHow does the cyclic learning rates come into the picture. Can one explain why cyclic learning rates works with the hypothesis that this paper processes?\nHow would different types of annealing methods come into the picture: e.g. linearly decaying the learning rate, exponential decay and etc…\nWhy are there two columns of “supported” in Table 1? What is the difference between them.\nHow does the adaptive learning rate algorithms such as Adam come into the picture with the learning rate decay hypothesis?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper seeks to understand the effect of learning rate decay in neural net training. This is an important question in the field and this paper also proposes to show why previous explanations were not correct. However, the reviewers found that the paper did not explain the experimental setup enough to be reproducible. Furthermore, there are significant problems with the novelty of the work due to its overlap with works such as (Nakiran et al., 2019), (Li et al. 2019) or (Jastrzębski et al. 2017).
Decision: Reject