user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation

Paper content:
The convergence rate and final performance of common deep learning models have significantly benefited from recently proposed heuristics such as learning rate schedules, knowledge distillation, skip connections and normalization layers.
In the absence of theoretical underpinnings, controlled experiments aimed at explaining the efficacy of these strategies can aid our understanding of deep learning landscapes and the training dynamics.
Existing approaches for empirical analysis rely on tools of linear interpolation and visualizations with dimensionality reduction, each with their limitations.
Instead, we revisit the empirical analysis of heuristics through the lens of recently proposed methods for loss surface and representation analysis, viz. mode connectivity and canonical correlation analysis (CCA), and hypothesize reasons why the heuristics succeed.
In particular, we explore knowledge distillation and learning rate heuristics of (cosine) restarts and warmup using mode connectivity and CCA.
Our empirical analysis suggests that
: (a) the reasons often quoted for the success of cosine annealing are not evidenced in practice
; (b) that the effect of learning rate warmup is to prevent the deeper layers from creating training instability; and
(c) that the latent knowledge shared by the teacher is primarily disbursed in the deeper layers.
The introduction of heuristics such as normalization layers BID19 BID0 , residual connections BID11 , and learning rate strategies BID26 BID9 Smith, 2017) have greatly accelerated progress in Deep Learning.
Many of these ingredients are now commonplace in modern architectures, and some of them have also been buttressed with theoretical guarantees BID1 BID28 BID10 .
However, despite their simplicity and efficacy, why some of these heuristics work is still relatively unknown.
Existing attempts at explaining these strategies empirically have been limited to intuitive explanations and the use of tools such as spectrum analysis (Sagun et al., 2017) , linear interpolation between two models and low-dimensional visualizations of the loss surface.
In our work, we instead use recent tools built specifically for analyzing deep networks, viz., mode connectivity and singular value canonical correlation analysis (SVCCA) (Raghu et al., 2017) .
We investigate three strategies in detail:
(a) cosine learning rate decay,
(b) learning rate warmup, and
(c) knowledge distillation, and list the summary of our contributions at the end of this section.Cosine annealing BID26 , also known as stochastic gradient descent with restarts (SGDR), and more generally cyclical learning rate strategies (Smith, 2017) , have been recently proposed to accelerate training of deep networks BID3 .
The strategy involves reductions and restarts of learning rates over the course of training, and was motivated as means to escape spurious local minima.
Experimental results have shown that SGDR often improves convergence both from the standpoint of iterations needed for convergence and the final objective.Learning rate warmup BID9 also constitutes an important ingredient in training deep networks, especially in the presence of large or dynamic batch sizes.
It involves increasing the learning rate to a large value over a certain number of training iterations followed by decreasing the learning rate, which can be performed using step-decay, exponential decay or other such schemes.
The strategy was proposed out of the need to induce stability in the initial phase of training with large learning rates (due to large batch sizes).
It has been employed in training of several architectures at scale including ResNets and Transformer networks (Vaswani et al., 2017) .Further
, we investigate knowledge distillation (KD) BID13 . This strategy
involves first training a (teacher) model on a typical loss function on the available data. Next, a different
(student) model (typically much smaller than the teacher model) is trained, but instead of optimizing the loss function defined using hard data labels, this student model is trained to mimic the teacher model. It has been empirically
found that a student network trained in this fashion significantly outperforms an identical network trained with the hard data labels. We defer a detailed discussion
of the three heuristics, and existing explanations for their efficacy to sections 3, 4 and 5 respectively.Finally, we briefly describe the tools we employ for analyzing the aforementioned heuristics. Mode connectivity (MC) is a recent
observation that shows that, under circumstances, it is possible to connect any two local minima of deep networks via a piecewise-linear curve BID5 . This shows that local optima obtained
through different means, and exhibiting different local and generalization properties, are connected. The authors propose an algorithm that
locates such a curve. While not proposed as such, we employ
this framework to better understand loss surfaces but begin our analysis in Section 2 by first establishing its robustness as a framework.Deep network analyses focusing on the weights of a network are inherently limited since there are several invariances in this, such as permutation and scaling. Recently, Raghu et al. (2017) propose
using CCA along with some pre-processing steps to analyze the activations of networks, such that the resulting comparison is not dependent on permutations and scaling of neurons. They also prove the computational gains
of using CCA over alternatives ( BID25 ) for representational analysis and employ it to better understand many phenomenon in deep learning.
Heuristics have played an important role in accelerating progress of deep learning.
Founded in empirical experience, intuition and observations, many of these strategies are now commonplace in architectures.
In the absence of strong theoretical guarantees, controlled experiments aimed at explaining the the efficacy of these strategies can aid our understanding of deep learning and the training dynamics.
The primary goal of our work was the investigation of three such heuristics using sophisticated tools for landscape analysis.
Specifically, we investigate cosine annealing, learning rate warmup, and knowledge distillation.
For this purpose, we employ recently proposed tools of mode connectivity and CCA.
Our empirical analysis sheds light on these heuristics and suggests that:
(a) the reasons often quoted for the success of cosine annealing are not evidenced in practice;
(b) that the effect of learning rate warmup is to prevent the deeper layers from creating training instability; and
(c) that the latent knowledge shared by the teacher is primarily disbursed in the deeper layers.Inadvertently, our investigation also leads to the design of new heuristics for practically improving the training process.
Through our results on SGDR, we provide additional evidence for the success of averaging schemes in this context.
Given the empirical results suggesting the localization of the knowledge transfer between teacher and student in the process of distillation, a heuristic can be designed that only trains portions of the (pre-trained) student networks instead of the whole network.
For instance, recent results on self-distillation BID6 show improved performance via multiple generations of knowledge distillation for the same model.
Given our results, computational costs of subsequent generations can be reduced if only subsets of the model are trained, instead of training the entire model.
Finally, the freezing of weights instead of employing learning rate warmup allows for comparable training performance but with reduced computation during the warmup phase.
We note in passing that our result also ties in with results of Hoffer et al. FORMULA2 The learning rate is initialized to 0.05 and scaled down by a factor of 5 at epochs {60, 120, 160} (step decay).
We use a training batch size of 100, momentum of 0.9, and a weight decay of 0.0005.
Elements of the weight vector corresponding to a neuron are initialized randomly from the normal distribution N (0, 2/n) where n is the number of inputs to the neuron.
We also use data augmentation by random cropping of input images.
Figures 7, 8 and 9 show the Validation Loss, Training Accuracy and Training Loss respectively for the curves joining the 6 pairs discussed in Section 2.1.1.
These results too, confirm the overfitting or poor generalization tendency of models on the curve.

assistant: text: We use empirical tools of mode connectivity and SVCCA to investigate neural network training heuristics of learning rate restarts, warmup and knowledge distillation.