An experimental study of layer-level training speed and its impact on generalization

How optimization influences the generalization ability of a DNN is still an active area of research . This work aims to unveil and study a factor of influence : the speed at which each layer trains . In our preliminary work , we develop a visualization technique and an optimization algorithm to monitor and control the layer rotation rate , a tentative measure of layer-level training speed , and show that it has a remarkably consistent and substantial impact on generalization . Our experiments further suggest that weight decay ’ s and adaptive gradients methods ’ impact on both generalization performance and speed of convergence are solely due to layer rotation rate changes compared to vanilla SGD , offering a novel interpretation of these widely used techniques , and providing supplementary evidence that layer-level training speed indeed impacts generalization . Besides these fundamental findings , we also expect that on a practical level , the tools we introduce will reduce the meta-parameter tuning required to get the best generalization out of a deep network . 1 INTRODUCTION . Generalization and gradient propagation are two popular themes in the deep learning literature . Concerning generalization , it has been observed that a network ’ s ability to generalize depends on a subtle interaction between the optimization procedure and the training data ( Zhang et al. , 2017a ; Arpit et al. , 2017 ) . Concerning gradient propagation , several works have shown that the norm of gradients can gradually increase or decrease as a function of layer depth ( i.e . vanishing and exploding gradients ( Bengio et al. , 1994 ; Hochreiter , 1998 ; Glorot & Bengio , 2010 ) ) , so that some layers are trained faster than others . This work explores an interaction between generalization and the intricate nature of gradient propagation in deep networks , and focuses on the following research question : how does the speed at which each layer trains influence generalization ? Our endeavour is motivated by the following intuition : if the training data influences a neural network ’ s generalization ability when using gradient-based optimization ( Zhang et al. , 2017a ; Arpit et al. , 2017 ) , the input and feedback signals that a layer receives ( during the network ’ s forward and backward passes ) could also influence the generalization ability induced by the layer ’ s training . These signals result from a transformation involving the other layers of the network such that , for example , the input signals of the last layer could be more conducive to good generalization if the first layers have been significantly updated already , instead of being randomly initialized ( cfr . the works on transfer learning ( Donahue et al. , 2014 ; Oquab et al. , 2014 ) ) . More generally , the speed at which each layer trains during the network ’ s training , since it directly influences how the input and feedback signals of the other layers evolve over training , could have an impact on generalization . Figure 1 supports our intuition with a toy example where training a single layer of an 11 layer MLP network , although always reaching 100 % train accuracy , results in different test accuracies depending on the layer ’ s localisation in the network architecture . Our study starts from an educated guess about how to measure layer-level training speed appropriately : we measure it through layer rotation rates , i.e . the rates at which the weight vectors of layers rotate ( another approach would be to measure the norm of the weight updates at each training step , as is done in Bengio et al . ( 1994 ) ; Hochreiter ( 1998 ) ; Glorot & Bengio ( 2010 ) ; Pascanu et al . ( 2013 ) ; Arjovsky et al . ( 2016 ) ) . The study is then composed of the three following steps : 1 . Developing tools to monitor and control layer rotation rates ; 2 . Using our controlling tool to systematically explore layer rotation rate configurations , varying the layers which are prioritized ( first layers , last layers , or no prioritization ) and the global rotation rate value ( high or low rate , for all layers ) ; 1 3 . Using our monitoring tool to study the layer rotation rates that emerge from standard training settings.1 The outcomes of our study , supported by an extensive amount of experiments , are the following : ( i ) layer rotation rates have a consistent and substantial impact on generalization ; ( ii ) weight decay is a key ingredient for enabling the emergence of beneficial layer rotation rates during SGD training ; ( iii ) adaptive gradient methods ’ impact on generalization and training speed does not result from parameter-level , but rather from layer-level adaptation of the learning rate ; While the influence of layer-level training speed on generalization has remained unstudied , our observations thus suggest that its impact is ubiquitous in current deep learning applications . Our preliminary work offers useful guidance for meta-parameter tuning and novel insights around two widely used techniques : weight decay and adaptive gradient methods . Moreover , while layer rotation rate as a measure of layer-level training speed originated from an educated guess , the impressive consistency of its impact on generalization supports the pertinence of this choice . Our work thus also contributes to the open problem of correctly measuring layer-level training speed . To encourage further validation of our claims , the tools and source code used to create all the figures of this paper are provided at -github link hidden to preserve anonymity- ( code uses the Keras ( Chollet et al. , 2015 ) and TensorFlow ( Agarwal et al. , 2016 ) libraries ) . We also encourage interested readers to browse the supplementary material of this paper , as additional results are presented and discussed . 2 RELATED WORK . Recent works have demonstrated that generalization in deep neural networks was largely due to the optimization procedure and its puzzling interaction with the training data ( Zhang et al. , 2017a ; Arpit et al. , 2017 ) . Our paper discloses an aspect of the optimization procedure that influences generalization in deep learning : the rate at which each layer ’ s weight vector is rotated . This novel 1Our preliminary study focuses on convolutional neural networks used for image classification . factor complements batch size and global learning rate , two parameters that have been extensively studied in the light of generalization ( Keskar et al. , 2017 ; Jastrzebski et al. , 2017 ; Smith & Le , 2017 ; Smith & Topin , 2017 ; Hoffer et al. , 2017 ; Masters & Luschi , 2018 ) . The works studying the vanishing and exploding gradients problems ( Bengio et al. , 1994 ; Hochreiter , 1998 ; Glorot & Bengio , 2010 ) heavily inspired this paper . These works introduce two ideas which are central to our investigation : the notion of layer-level training speed and the fact that SGD does not necessarily train all layers at the same speed during training . Our work explores the same phenomena , but studies them in the light of generalization instead of trainability and speed of convergence . Our paper also proposes Layca , an algorithm to control the rate at which each layer ’ s weight is rotated during training . It is related to the works that sought solutions to the gradient propagation problems at optimization level ( Pascanu et al. , 2013 ; Hazan et al. , 2015 ; Singh et al. , 2015 ; Arjovsky et al. , 2016 ; Pennington et al. , 2017 ) . These works , however , do not use weight rotation as a measure of layer-level training speed , and also focus on speed of convergence instead of generalization . Recently , a series of papers proposed optimization algorithms similar to Layca and observed an impact on generalization ( Yu et al. , 2017 ; Zhang et al. , 2017b ; Ginsburg et al. , 2018 ) . Section A.2 in our Supplementary Material provides evidence that these methods may be equivalent to Layca in practice , despite avoiding some of Layca ’ s operations . Our paper thus supplements these works by providing an extensive study of the phenomena underlying their observations . Several works have recently argued that weight decay ’ s regularization effect emerged from its ability to increase the effective learning rate ( van Laarhoven , 2017 ; Hoffer et al. , 2018 ; Anonymous , 2019 ) . A concise description of when and to what extent weight decay increases the effective learning rate is however lacking , such that using weight decay is still necessary to benefit from its regularization effect in practice.2 Our work also analyses weight decay , but from the perspective of layer rotation rates instead of effective learning rates . We show that this new perspective enables a more succinct description of weight decay ’ s regularizing effect , that we are able to reproduce without any additional meta-parameter tuning when using Layca , our tool for controlling layer rotation rates . 3 TOOLS FOR MONITORING AND CONTROLLING LAYER ROTATION RATES . This paper ’ s goal is to study the relation between layer-level training speed and generalization . However , the notion of layer-level training speed is unclear , and its control through SGD is potentially difficult because of the intricate nature of gradient propagation ( cfr . vanishing and exploding gradients ) . Therefore , our work starts by making an educated guess about how to measure layer-level training speed and by developing tools to monitor and control this metric during training . This section presents the metric and tools , while the impact on generalization is studied in Sections 4 and 5 . 3.1 HOW CAN WE MEASURE LAYER-LEVEL TRAINING SPEED ? . Training speed can be understood as the speed with which a model converges to its optimal solution -not to be confounded with learning rate , which is only one of the parameters that affect training speed in current deep learning applications . The notion of layer-level training speed is ill-posed , since a layer does not have a loss of its own : all layers optimize the same global loss function . Given a training step , how can we know by how much each layer ’ s update contributed to the improvement of the global loss ? Or , in other words , how can we measure at what rate relevant features are learned by each layer individually ? Previous work on vanishing and exploding gradients focused on the norm and variance of gradients as a measure of layer-level training speed ( Bengio et al. , 1994 ; Hochreiter , 1998 ; Glorot & Bengio , 2010 ) . Provided the empirical work on activation and weight binarization during ( Courbariaux & David , 2015 ; Rastegari et al. , 2016 ; Hubara et al. , 2016 ) or after training ( Agrawal et al. , 2014 ; 2van Laarhoven ( 2017 ) proposes to keep the norm of the weights fixed to 1 in order to eliminate the effect of weight decay , but not to reproduce it . Hoffer et al . ( 2018 ) ; Anonymous ( 2019 ) are able to reproduce the regularizing effect of weight decay by tuning the effective learning rate , but their tuning strategy consists in copying the effective learning rate that emerges when training the same network with weight decay . Algorithm 1 Layca , an algorithm that enables control over the amount of weight rotation per step for each layer through its learning rate parameter ( cfr . Section 3.2 ) . Require : o , an optimizer ( SGD is the default choice ) Require : T , the number of training steps L is the number of layers in the network for l=0 to L-1 do Require : ρl ( t ) , a layer ’ s learning rate schedule Require : wl0 , the initial multiplicative weights of layer l end for t← 0 while t < T do s0t , ... , s L−1 t = getStep ( o , w 0 t , ... , w L−1 t ) ( get the updates of the selected optimizer ) for l=0 to L-1 do slt ← slt − ( slt·w l t ) w l t wlt·wlt ( project step on space orthogonal to wlt ) slt ← slt‖w l t‖2 ‖slt‖2 ( rotation-based normalization ) wlt+1 ← wlt + ρl ( t ) slt ( perform update ) wlt+1 ← wlt+1 ‖wl0‖2 ‖wlt+1‖2 ( project weights back on sphere ) end for t← t+ 1 end while Carbonnelle & De Vleeschouwer , 2018 ) , we argue that the norm of a weight vector does not matter , but only its orientation . Therefore , we suggest to measure training speed through the rotation rate of a layer ’ s weight vector ( also denoted by layer rotation rate in this paper ) . More precisely , let wtl be the flattened weight tensor of the lth layer at optimization step t , then the rotation rate of layer l between steps t1 and t2 is defined as the angle between wt1l and w t2 l divided by the number of performed steps t2 − t1 . 3 In order to visualize how fast layers rotate during training , we propose to inspect how the cosine distance between each layer ’ s current weight vector and its initialization evolves across training steps . We denote this visualization tool by layer-wise angle deviation curves hereafter .

Summary: This paper insists layer-level training speed is crucial for generalization ability. The layer-level training speed is measured by  angle between weights at different time stamps in this paper. To control the amount of weight rotation, which means the degree of angle movement, this paper proposes a new algorithm, Layca. This algorithm projects the gradient vector of SGD (or update vector of other variants) onto the space orthogonal to the current weight vector, and adjust the length of the update vector to achieve the desirable angle movement. This paper conducted several experiments to verify the helpfulness of Layca.