A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation

1 INTRODUCTION . The introduction of heuristics such as normalization layers ( Ioffe & Szegedy , 2015 ; Ba et al. , 2016 ) , residual connections ( He et al. , 2016 ) , and learning rate strategies ( Loshchilov & Hutter , 2016 ; Goyal et al. , 2017 ; Smith , 2017 ) have greatly accelerated progress in Deep Learning . Many of these ingredients are now commonplace in modern architectures , and some of them have also been buttressed with theoretical guarantees ( Balduzzi et al. , 2017 ; Poggio & Liao , 2017 ; Hardt & Ma , 2016 ) . However , despite their simplicity and efficacy , why some of these heuristics work is still relatively unknown . Existing attempts at explaining these strategies empirically have been limited to intuitive explanations and the use of tools such as spectrum analysis ( Sagun et al. , 2017 ) , linear interpolation between two models and low-dimensional visualizations ( Li et al. , 2017 ) of the loss surface . In our work , we instead use recent tools built specifically for analyzing deep networks , viz. , mode connectivity ( Garipov et al. , 2018 ) and singular value canonical correlation analysis ( SVCCA ) ( Raghu et al. , 2017 ) . We investigate three strategies in detail : ( a ) cosine learning rate decay , ( b ) learning rate warmup , and ( c ) knowledge distillation , and list the summary of our contributions at the end of this section . Cosine annealing ( Loshchilov & Hutter , 2016 ) , also known as stochastic gradient descent with restarts ( SGDR ) , and more generally cyclical learning rate strategies ( Smith , 2017 ) , have been recently proposed to accelerate training of deep networks ( Coleman et al. , 2018 ) . The strategy involves reductions and restarts of learning rates over the course of training , and was motivated as means to escape spurious local minima . Experimental results have shown that SGDR often improves convergence both from the standpoint of iterations needed for convergence and the final objective . ∗Work performed while interning at Salesforce Research Learning rate warmup ( Goyal et al. , 2017 ) also constitutes an important ingredient in training deep networks , especially in the presence of large or dynamic batch sizes . It involves increasing the learning rate to a large value over a certain number of training iterations followed by decreasing the learning rate , which can be performed using step-decay , exponential decay or other such schemes . The strategy was proposed out of the need to induce stability in the initial phase of training with large learning rates ( due to large batch sizes ) . It has been employed in training of several architectures at scale including ResNets and Transformer networks ( Vaswani et al. , 2017 ) . Further , we investigate knowledge distillation ( KD ) ( Hinton et al. , 2015 ) . This strategy involves first training a ( teacher ) model on a typical loss function on the available data . Next , a different ( student ) model ( typically much smaller than the teacher model ) is trained , but instead of optimizing the loss function defined using hard data labels , this student model is trained to mimic the teacher model . It has been empirically found that a student network trained in this fashion significantly outperforms an identical network trained with the hard data labels . We defer a detailed discussion of the three heuristics , and existing explanations for their efficacy to sections 3 , 4 and 5 respectively . Finally , we briefly describe the tools we employ for analyzing the aforementioned heuristics . Mode connectivity ( MC ) is a recent observation that shows that , under circumstances , it is possible to connect any two local minima of deep networks via a piecewise-linear curve ( Garipov et al. , 2018 ; Draxler et al. , 2018 ) . This shows that local optima obtained through different means , and exhibiting different local and generalization properties , are connected . The authors propose an algorithm that locates such a curve . While not proposed as such , we employ this framework to better understand loss surfaces but begin our analysis in Section 2 by first establishing its robustness as a framework . Deep network analyses focusing on the weights of a network are inherently limited since there are several invariances in this , such as permutation and scaling . Recently , Raghu et al . ( 2017 ) propose using CCA along with some pre-processing steps to analyze the activations of networks , such that the resulting comparison is not dependent on permutations and scaling of neurons . They also prove the computational gains of using CCA over alternatives ( ( Li et al. , 2015 ) ) for representational analysis and employ it to better understand many phenomenon in deep learning . Contributions : • We use mode connectivity and CCA to improve understanding of cosine annealing , learning rate warmup and knowledge distillation . For mode connectivity , we also establish the robustness of the approach across changes in training choices for obtaining the modes . • We demonstrate that the reasons often quoted for the success of cosine annealing are not substantiated by our experiments , and that the iterates move over barriers after restarts but the explanation of escaping local minima might be an oversimplification . • We show that learning rate warmup primarily limits weight changes in the deeper layers and that freezing them achieves similar outcomes as warmup . • We show that the latent knowledge shared by the teacher in knowledge distillation is primarily disbursed in the deeper layers . 2 EMPIRICAL TOOLS . 2.1 MODE CONNECTIVITY . Garipov et al . ( 2018 ) introduce a framework , called mode connectivity , to obtain a low loss ( or high accuracy , in the case of classification ) curve of simple form , such as a piecewise linear curve , that connects optima ( modes of the loss function ) found independently . This observation suggests that points at the same loss function depth are connected , somewhat contrary to several empirical results claiming that minima are isolated or have barriers between them1 . Let wa ∈ RD and wb ∈ RD be two modes in the D-dimensional parameter space obtained by optimizing a given loss function L ( w ) ( like the cross-entropy loss ) . We represent a curve connecting 1Draxler et al . ( 2018 ) independently report the same observation for neural network loss landscapes , and claim that this is suggestive of the resilience of neural networks to perturbations in model parameters . wa and wb by φθ ( t ) : [ 0 , 1 ] → RD , such that φθ ( 0 ) = wa and φθ ( 1 ) = wb . To find a low loss path , we find the set of parameters θ ∈ RD that minimizes the following loss : ` ( θ ) = ∫ 1 0 L ( φθ ( t ) ) dt = Et∼U ( 0,1 ) L ( φθ ( t ) ) where U ( 0 , 1 ) is the uniform distribution in the interval [ 0 , 1 ] . To optimize ` ( θ ) for θ , we first need to chose a parametric form for φθ ( t ) . One of the forms proposed by Garipov et al . ( 2018 ) is a polygonal chain with a single bend at θ as follows φθ ( t ) = { 2 ( tθ + ( 0.5− t ) wa ) , if 0 ≤ t ≤ 0.5 2 ( ( t− 0.5 ) wb + ( 1− t ) θ ) if 0.5 < t ≤ 1 To minimize ` ( θ ) , we sample t ∼ U [ 0 , 1 ] at each iteration and use ∇θL ( φθ ( t ) ) as an unbiased estimate for the true gradient∇θ ` ( θ ) to perform updates on θ , where θ is initialized with 12 ( wa+wb ) . 2.1.1 RESILIENCE OF MODE CONNECTIVITY . To demonstrate that the curve-finding approach works in practice , Garipov et al . ( 2018 ) use two optima found using different initializations but a common training scheme which we detail below . We explore the limits of this procedure by connecting optima obtained from different training strategies . Our goal of this investigation is to first establish the robustness of the framework in order to seamlessly use it as a tool for analysis . In particular , we experiment with different initializations , optimizers , data augmentation choices , and hyperparameter settings including regularization , training batch sizes , and learning rate schemes . We note in passing that while the framework was proposed to connect two points in the parameter space that are at equal depth in the loss landscape , it is well-defined to also connect points at different depths ; in this case , the path corresponds to one that minimizes the average loss along the curve . Conventional wisdom suggests that the different training schemes mentioned above will converge to regions in the parameter space that are vastly different from each other . Examples of this include size of minibatches used during training ( Keskar et al. , 2016 ) , choice of optimizer ( Heusel et al. , 2017 ; Wilson et al. , 2017 ) , initialization ( Goodfellow et al. , 2016 ) and choice of regularizer . Having a high accuracy connection between these pairs would seem counterintuitive . For obtaining the reference model ( named mode G ) , we train the VGG-16 model architecture ( Simonyan & Zisserman , 2014 ) using CIFAR-10 training data ( Krizhevsky et al. , 2014 ) for 200 epochs with SGD . We then build 6 variants of the reference mode G as follows : we obtain mode A using a training batch size of 4000 , mode B by using the Adam optimizer instead of SGD , mode C with a linearly decaying learning rate instead of the step decay used in mode G , mode D using a smaller weight decay of 5 × 10−6 , mode E by increasing the variance of the initialization distribution to 3 × √ 2/n and mode F using no data augmentation . Note that for the set of modes { A , B , C , D , E , F } , all the other hyper-parameters and settings except the ones mentioned above are kept same as that for mode G. We use the mode connectivity algorithm on each of the 6 pairs of modes including G and another mode , resulting in curves GA , GB , GC , GD , GE , and GF . Figure 1 shows the validation accuracy for models on each of the 6 connecting curves during the 20th , 40th , 60th and 80th epochs of the mode connectivity training procedure and also for models on the line segment joining the two endpoints ( corresponding to the initialization for θ at epoch 0 ) . As described in Section 2.1 , for a polychain curve GX ( connecting modes G and X using the curve described by θ ) , model parameters φθ ( t ) on the curve are given by pφθ ( t ) = 2 ( tpθ + ( 0.5 − t ) pG ) if 0 ≤ t ≤ 0.5 and pφθ ( t ) = 2 ( ( t− 0.5 ) pX + ( 1− t ) pθ ) if 0.5 < t ≤ 1 where pG , pθ and pX are parameters of the models G , θ , and X respectively . Thus φθ ( 0 ) = G and φθ ( 1 ) = X . In a few epochs of the curve training , for all 6 pairs , we can find a curve such that each point on it generalizes almost as well as models from the pair that is being connected . Note that by virtue of existence of these 6 curves , there exists a high accuracy connecting curve ( albeit with multiple bends ) for each of the ( 7 2 ) pairs of modes . We refer the reader to Appendix 7 for a t-SNE plot of the modes and their connections , and also for additional plots and details . Having established the high likelihood of the existence of these curves , we use this procedure along with interpolation of the loss surface between parameters at different epochs as tools to analyze the dynamics of SGD and SGDR .

Summary: This paper empirically explores heuristics commonly used in deep learning: learning rate restarts, warmup and distillation. The authors utilize two recently proposed tools for neural network analysis: mode connectivity (MC) finding a low loss pathway between two given points in the space of DNN parameters and CCA measuring the correlation of  DNN layer activations. Conducting a set of experiments and analyzing the results the authors refine the intuition behind the considered heuristics and dynamics of corresponding training procedures.