How Does Learning Rate Decay Help Modern Neural Networks?

1 INTRODUCTION . Modern neural networks are deep , wide , and nonconvex . They are powerful tools for representation learning and serve as core components of deep learning systems . They are top-performing models in language translation ( Sutskever et al. , 2014 ) , visual recognition ( He et al. , 2016 ) , and decision making ( Silver et al. , 2018 ) . However , the understanding of modern neural networks is way behind their broad applications . A series of pioneering works ( Zhang et al. , 2017 ; Belkin et al. , 2019 ; Locatello et al. , 2019 ) reveal the difficulty of applying conventional machine learning wisdom to deep learning . A better understanding of deep learning is a major mission in the AI field . One obstacle in the way of understanding deep learning is the existence of magic modules in modern neural networks and magic tricks to train them . Take batch normalization module ( Ioffe & Szegedy , 2015 ) for example , its pervasiveness in both academia and industry is undoubted . The exact reason why it expedites training and helps generalization , however , remains mysterious and is actively studied in recent years ( Bjorck et al. , 2018 ; Santurkar et al. , 2018 ; Kohler et al. , 2019 ) . Only when we clearly understand these magical practices can we promote the theoretical understanding of modern neural networks . Learning rate is “ the single most important hyper-parameter ” ( Bengio , 2012 ) in training neural networks . Learning rate decay ( lrDecay ) is a de facto technique for training modern neural networks , where we adopt an initially large learning rate and then decay it by a certain factor after pre-defined epochs . Popular deep networks such as ResNet ( He et al. , 2016 ) , DenseNet ( Huang et al. , 2017b ) are all trained by Stochastic Gradient Descent ( SGD ) with lrDecay . Figure 1 ( a ) is an example of lrDecay , with the learning rate decayed by 10 every 30 epochs . The training is divided into several stages by the moments of decay . These stages can be easily identified in learning curves ( such as Figure 1 ( b ) ) , where the performance boosts sharply shortly after the learning rate is decayed . The lrDecay enjoys great popularity due to its simplicity and general effectiveness . Common beliefs in how lrDecay works are derived from the optimization analysis in ( Stochastic ) Gradient Descent ( LeCun et al. , 1991 ; Kleinberg et al. , 2018 ) . They attribute the effect of an initially large learning rate to escaping spurious local minima or accelerating training and attribute the effect of decaying the learning rate to avoiding oscillation around local minima . However , these common beliefs are insufficient to explain our empirical observations from a series of carefully-designed experiments in Section 4 . In this paper , we provide an alternative view : the magnitude of the learning rate is closely related to the complexity of learnable patterns . From this perspective , we propose a novel explanation for the efficacy of lrDecay : an initially large learning rate suppresses the memorization of noisy data while decaying the learning rate improves the learning of complex patterns . This is validated on a carefully-constructed dataset with tractable pattern complexity . The pattern complexity in realworld datasets is often intractable . We thus validate the explanation by testing its implication on real-world datasets . The implication that additional patterns learned in later stages of lrDecay are more complex and thus less transferable across different datasets , is also justified empirically . A comparison between the proposed explanation and the common beliefs is summarized in Table 1 . Our explanation is supported by carefully-designed experiments and provides a new perspective on analyzing learning rate decay . The contribution of this paper is two-fold : • We demonstrate by experiments that existing explanations of how lrDecay works are insufficient in explaining the training behaviors in modern neural networks . • We propose a novel explanation based on pattern complexity , which is validated on a dataset with tractable pattern complexity , and its implication is validated on real-world datasets . The explanation also suggests that complex patterns are only learnable after learning rate decay . Thus , when the model learns all simple patterns , but the epoch to decay has not reached , immediately decaying the learning rate will not hurt the performance . This implication is validated in Section A.1 . 2 RELATED WORK . 2.1 UNDERSTANDING THE BEHAVIOR OF SGD . Recently , researchers reveal the behavior of SGD from multiple perspectives ( Li et al. , 2019 ; Mangalam & Prabhu , 2019 ; Nakkiran et al. , 2019 ) . They respect the difference among data items rather than treat them as identical samples from a distribution . They study the behavior of SGD in a given dataset . In Mangalam & Prabhu ( 2019 ) , they show that deep models first learn easy examples classifiable by shallow methods . The mutual information between deep models and linear models is measured in Nakkiran et al . ( 2019 ) , which suggests deep models first learn data explainable by linear models . Note that they are not relevant to learning rates . Li et al . ( 2019 ) analyze a toy problem to uncover the regularization effect of an initially large learning rate . Their theoretical explanation is , however , based on a specific two-layer neural network they design . Different from these works , Section 5 studies the behavior of SGD induced by lrDecay in a modern WideResNet ( Zagoruyko & Komodakis , 2016 ) , finding that learning rate decay improves learning of complex patterns . We formally define pattern complexity by expected class conditional entropy , while the measure of pattern complexity in Mangalam & Prabhu ( 2019 ) ; Nakkiran et al . ( 2019 ) relies on an auxiliary model . 2.2 ADAPTIVE LEARNING RATE METHODS . Adaptive learning rate methods such as AdaGrad ( Duchi et al. , 2011 ) , AdaDelta ( Zeiler , 2012 ) , and ADAM ( Kingma & Ba , 2015 ) are sophisticated optimization algorithms for training modern neural networks . It remains an active research field to study their behaviors and underlying mechanisms ( Reddi et al. , 2018 ; Luo et al. , 2019 ) . However , we focus on learning rate decay in SGD rather than on the adaptive learning rate methods . On the one hand , SGD is the de facto training algorithm for popular models ( He et al. , 2016 ; Huang et al. , 2017b ) while lrDecay is not common in the adaptive methods ; On the other hand , many adaptive methods are not as simple as SGD and even degenerate in convergence in some scenarios ( Wilson et al. , 2017 ; Liu et al. , 2019 ) . We choose to study SGD with lrDecay , without introducing adaptive learning rate to keep away from its confounding factors . 2.3 OTHER LEARNING RATE STRATEGIES . Besides the commonly used lrDecay , there are other learning rate strategies . Smith ( 2017 ) proposes a cyclic strategy , claiming to dismiss the need for tuning learning rates . Warm restart of learning rate is explored in Loshchilov & Hutter ( 2017 ) . They achieve better results when combined with Snapshot Ensemble ( Huang et al. , 2017a ) . These learning rate strategies often yield better results at the cost of additional hyperparameters that are not intuitive . Consequently , it is still the de facto to decay the learning rate after pre-defined epochs as in Figure 1 ( a ) . We stick our analysis to lrDecay rather than to other fancy ones because of its simplicity and general effectiveness . 2.4 TRANSFERABILITY OF DEEP MODELS . Training a model on one dataset that can be transferred to other datasets has long been a goal of AI researches . The exploration of model transferability has attracted extensive attention . In Oquab et al . ( 2014 ) , deep features trained for classification are transferred to improve object detection successfully . Yosinski et al . ( 2014 ) study the transferability of different modules in pre-trained networks , indicating that higher layers are more task-specific and less transferable across datasets . By varying network architectures , Kornblith et al . ( 2019 ) show architectures with a better ImageNet accuracy generally transfer better . Raghu et al . ( 2019 ) explore transfer learning in the field of medical imaging to address domain-specific difficulties . Different from these works that only consider the transferability of models after training , we investigate another dimension of model transferability in Section 6 : the evolution of transferability during training with lrDecay . 3 COMMON BELIEFS IN EXPLAINING LRDECAY . 3.1 GRADIENT DESCENT EXPLANATION . The practice of lrDecay in training neural networks dates back to LeCun et al . ( 1998 ) . The most popular belief in the effect of lrDecay comes from the optimization analysis of Gradient Descent ( GD ) ( LeCun et al. , 1991 ) . Although SGD is more practical in deep learning , researchers are usually satisfied with the analysis of GD considering that SGD is a stochastic variant of GD . Specifically , LeCun et al . ( 1991 ) analyze the property of a quadratic loss surface which can be seen as a second-order approximation around a local minimum in nonconvex optimization . Learning rates are characterized by the relationship with eigenvalues of the Hessian at a local minimum . Denote η the learning rate , H the Hessian , λ an eigenvalue of H , and v an eigenvector of λ . The behavior of the network along the direction v can be characterized as ( 1 − ηλ ) kv , with k the iteration number . Convergence in the direction of v requires 0 < η < 2/λ , while η > 2/λ leads to divergence in the direction of v. If 0 < η < 2/λ holds for every eigenvalue of the Hessian , the network will converge quickly ( Figure 2 left ) . If it holds for some directions but not for all directions , the network will diverge in some directions and thus jump into the neighborhood of another local minimum ( Figure 2 middle ) . If the learning rate is too large , the network will not converge ( Figure 2 right ) . In particular , when oscillation happens , it means the learning rate is too large and should be decayed . The effect of lrDecay hence is to avoid oscillation and to obtain faster convergence . Note LeCun et al . ( 1991 ) only analyze a simple one-layer network . It may not hold for modern neural networks ( see Section 4.1 ) . 3.2 STOCHASTIC GRADIENT DESCENT EXPLANATION . Another common belief is the Stochastic Gradient Descent explanation , arguing that “ with a high learning rate , the system is unable to settle down into deeper , but narrower parts of the loss function. ” 1 Although it is common , this argument has not been formally analyzed until very recently . Under some assumptions , Kleinberg et al . ( 2018 ) prove SGD is equivalent to the convolution of loss surface , with the learning rate serving as the conceptual kernel size of the convolution . With an appropriate learning rate , spurious local minima can be smoothed out , thus helping neural networks escape bad local minima . The decay of learning rate later helps the network converge around the minimum . Figure 3 is an intuitive one-dimensional example . The first plot shows that a large learning rate helps escape bad local minima in both sides . The lrDecay in subsequent plots increases the probability of reaching the global minimum . Although intuitive , the explanation requires some assumptions that may not hold for modern neural networks ( see Section 4.2 ) . 1http : //cs231n.github.io/neural-networks-3/ # anneal

Summary: The paper investigates the role of learning rate decay in neural network training. While there are prevalent ideas of how/why learning rate decay help both optimization and generalization of neural networks, this work proposes interpretation based on pattern complexity. The mechanism the paper proposes is that initial learning rate helps ignore noise in the beginning and decayed learning rate help to learn complex patterns.