On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay

1 Introduction . Normalization approaches , such as batch or layer normalization , have become vital for the successful training of modern deep neural networks [ 12 , 2 , 24 , 21 , 27 ] . Despite much recent work [ 3 , 22 , 9 , 28 ] , it is still not completely understood how normalization influences the training process . In this work , we investigate the surprising periodic behavior that may occur when a neural network is trained with a commonly used combination of some kind of normalization , in our case batch normalization ( BN ) [ 12 ] , and weight decay regularization ( WD ) . Examples of this behavior are provided in Figure 1 . The dynamics of neural network training with BN and WD have been examined extensively in literature due to the non-trivial competing influence of BN and WD on the norm of neural network ’ s weights . More precisely , using BN makes ( a part of ) neural network ’ s weights scale-invariant , i.e. , multiplying them by a positive constant does not change the network ’ s output . Although scale invariance allows optimizing on a sphere with a fixed weight norm [ 6 ] , classic SGD-based approaches are usually preferred over constraint optimization methods in practice due to more straightforward implementation . Making an SGD step in the direction of the loss gradient always increases the norm of scale-invariant parameters , while WD aims at decreasing the weight norm ( see illustration in Figure 2 ) . In sum , training the neural network with BN and WD results in an interplay between two forces : a “ centripetal force ” of the WD and the “ centrifugal force ” of the loss gradients . Many works notice the positive effect of WD on optimization and generalization caused by the control of the scale-invariant weights norm and the subsequent influence on the effective learning rate [ 25 , 10 , 29 , 18 , 19 , 26 , 20 ] , i.e. , the learning rate on a unit sphere in the scale-invariant weights space . However , the general dynamics of the norm of the scale-invariant weights are viewed in the literature from two contradicting points , and this work is devoted to resolving this contradiction . ∗First two authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) On the one hand , Li et al . [ 19 ] claim that learning with SGD , BN , and WD leads to an equilibrium state , where the “ centripetal force ” is compensated by the “ centrifugal force ” and eventually the norm of scale-invariant weights ( along with other statistics related to the training procedure ) will converge to a constant value . Several other works hold a similar equilibrium view [ 25 , 5 , 26 ] . On the other hand , a number of works [ 17–19 ] underline that using WD may cause approaching the origin ( zero scale-invariant weights ) , which results in training instability due to increasing effective learning rate . Particularly , Li et al . [ 17 ] reveal that approaching the origin in weight-normalized neural networks leads to numerical overflow in gradient updates and subsequent training failure . Li and Arora [ 18 ] also underline that scale-invariant functions are ill-conditioned near the origin and prove in a simplified setting that loss convergence is impossible if both BN and WD are used ( but guaranteed if either of them is disabled ) . Moreover , despite their equilibrium view , Li et al . [ 19 ] empirically observe that the train loss permanently exhibits oscillations between low and high values when full-batch gradient descent is used . In this work , we study the specified contradiction between the equilibrium presumption and the instability presumption and show that both are true only to some extent . Specifically , we show that the training process converges to a consistent periodic behavior , i.e. , it regularly exhibits instabilities which , however , do not lead to a complete training failure but cause a new period of training ( see Figure 1 ) . Thus , our contributions are as follows . • We discover the periodic behavior of neural network training with BN and WD and reveal its reasons by analyzing the underlying mechanism for fully scale-invariant neural networks trained with standard constant learning rate SGD ( Section 4 ) or GD ( Appendix C ) . • We provide a theoretical grounding for our findings by generalizing previous results on the equilibrium condition , analyzing the necessary conditions for destabilization of training , and relating the frequency of destabilization to the choice of hyperparameters ( Section 5 ) . • We conduct a rigorous empirical study of this periodic behavior ( Section 6 ) and show its presence in more practical scenarios with momentum , augmentation , and neural networks incorporating trainable non-scale-invariant weights ( Section 7 ) , and also with Adam optimizer and other normalization techniques ( Appendix I ) . Our source code is available at https : //github.com/tipt0p/periodic_behavior_bn_wd . 2 Background . As discussed in the introduction , batch normalization makes ( a part of ) neural network ’ s weights scaleinvariant . In this section , we describe the properties of scale-invariant functions , upon which we build our further reasoning . Consider an arbitrary scale-invariant function f ( x ) , i.e. , f ( αx ) = f ( x ) , ∀x and ∀α > 0 . Then two fundamental properties may be inferred , see Lemma 1.3 in Li and Arora [ 18 ] : ⟨∇f ( x ) , x⟩ = 0 , ∀x ( 1a ) ∇f ( αx ) = 1 α ∇f ( x ) , ∀x , α > 0 . ( 1b ) Consider optimizing f ( x ) w.r.t . x using ( S ) GD2 with learning rate η and weight decay λ : xt+1 = ( 1− ηλ ) xt − η∇f ( xt ) . ( 2 ) The properties above lead to two important corollaries about the dynamics of the optimization process . First , according to property ( 1a ) , shifting x in the direction of −∇f ( x ) , i.e. , making a gradient descent step , always increases ∥x∥ , while weight decay , on the other hand , decreases ∥x∥ . See Figure 2 for the illustration . The interaction of these “ centripetal ” and “ centrifugal ” forces may cause ∥x∥ to change nontrivially during optimization . Second , according to property ( 1b ) , even though function value f ( x ) is invariant to multiplying x by α , the optimization dynamics changes substantially when optimization is performed at different scales of ∥x∥ . For smaller norms , optimization makes larger steps , which may result in instabilities , while for larger norms , steps are smaller , and optimization process may converge slowly . Since scale-invariant f ( x ) may be seen as a function on a sphere , its optimization dynamics are often analysed on a unit sphere ∥x∥ = 1 . One can obtain equivalent optimization dynamics on the unit sphere as in the initial space by using the notion of effective gradient and effective learning rate . The effective gradient is defined as a gradient for a point on a unit sphere and may be obtained by substituting α = ∥x∥−1 in ( 1b ) : ∇f ( x/ ∥x∥ ) = ∇f ( x ) ∥x∥ . The effective learning rate can be defined as η̃ = η/ ∥x∥2 [ 10 , 20 ] . Change in ∥x∥ does not affect the effective gradient by definition and is reflected only in the effective learning rate : the lower the norm , the higher the effective learning rate , and the larger the optimization steps . 3 Methodology and experimental setup . In order to isolate the effect of the joint use of batch normalization and weight decay and avoid the influence of other factors , we conduct a series of experiments in a simplified setting , when all learnable weights of a neural network are scale-invariant and optimization is performed using SGD with constant learning rate , without momentum or data augmentation . This allows us to better understand the nature of the periodic behavior ( Section 4 ) and analyse its empirical properties ( Section 6 ) . After that , we return to the setting with a more conventional training of standard neural networks and show that the periodic behavior occurs in this scenario as well ( Section 7 ) . We conduct experiments with ResNet-18 and a simple 3-layer batch-normalized convolutional neural network ( ConvNet ) 3 on CIFAR-10 [ 15 ] and CIFAR-100 [ 16 ] datasets . To make standard networks fully scale-invariant , we rely on the approach of Li and Arora [ 18 ] , i.e. , we insert additional BN layers and fix the non-scale-invariant weights to be constant . Specifically , we use zero mean and unit variance in batch normalization layers instead of learnable location and scale parameters and freeze the weights of the last layer at random initialization . The latter action does not hurt the performance in practice [ 11 ] . However , we find that even with low train error , the training dynamics with the fixed last layer may still substantially differ from conventional training , as the neural network exhibits low confidence in predictions . To achieve high confidence for all objects and , consequently , low train loss , we increase the norm of the last layer ’ s weights to 10 . The influence of this rescaling is shown in Appendix G. We optimize cross-entropy loss , use the batch size of 128 and train neural networks for 1000 epochs to show the consistency of the discovered periodic behavior . We consider a range of learning rates , { 10−k , 3 · 10−k } k=0,1,2,3 and choose the most representative ones for each visualization , since it is difficult to distinguish many periodic functions on one plot . For fully scale-invariant neural networks , training with a fixed weight decay – learning rate product converges to similar behavior , regardless of their ratio : we show it empirically and discuss it from the theoretical point of view in Appendix F.1 ; the same was noticed in [ 18 , 19 ] . Thus , in the main text , we provide only the results for the varied 2Since both stochastic and full-batch gradients of a scale-invariant objective possess properties ( 1a ) and ( 1b ) , we do not distinguish between them in our reasoning . 3Both architectures in the implementation of https : //github.com/g-benton/hessian-eff-dim . learning rate and the fixed weight decay of 0.001 . Results for the varied weight decay are presented in Appendix F.3 . At each training epoch , we log standard train / test metrics , the norm of scale-invariant weights ( SI weight norm ) , which is in the focus of this research , and metrics characterizing training dynamics on a unit sphere : effective learning rate and the norm of effective gradients ( mean over mini-batches ) . We plot the two latter metrics over two axes of the phase diagram to visualize their simultaneous dynamics that will help us to understand the mechanism underlying the periodic behavior . 4 Periodic behavior and its underlying mechanism . As discussed in the previous section , we begin our study by considering a simplified setting with a fully scale-invariant neural network trained with standard SGD . Figure 3 shows the presence of the periodic behavior for a scale-invariant ConvNet on the CIFAR-10 dataset for a range of learning rates . In Appendix F.2 , we show the presence of the periodic behavior for other dataset-architecture pairs . The same periodic behavior is also present for neural network training with full-batch gradient descent , see Appendix C. Moreover , this behavior can be observed even when optimizing common scale-invariant functions using the gradient descent method with weight decay ( see Appendix E ) . The observed periodic behavior occurs because of the interaction between batch normalization and weight decay , particularly because of their competing influence on the weight norm . As discussed in Section 2 , weight decay aims at decreasing the weight norm , while loss gradients aim at increasing the weight norm due to scale invariance caused by batch normalization ( see Figure 2 ) . These two forces alternately outweigh each other for quite long periods of training , resulting in periodic behavior . Let us examine a single period in greater detail by analyzing Figure 4 that shows the dynamics of relevant training metrics logged after each SGD step of ConvNet training . At the beginning of the period , the train loss is high , and the large gradients of the loss outweigh weight decay . This results in increasing weight norm and decreasing effective learning rate , i.e. , we move along phase A of the phase diagram . SGD continues optimizing train loss , and at some point , train loss and its gradients become small and outweighed by weight decay . As a result , the weight norm starts decreasing , and the effective learning rate increases , i.e. , we move along phase B of the phase diagram . We note that the transition between phases A and B correlates with achieving near-zero train error . When the weight norm becomes too small , and the effective learning rate becomes too high , SGD makes several large steps and leaves the low loss region . Gradients grow along with train loss and , multiplied by a high effective learning rate , lead to the fast growth of the weight norm , i.e. , we move along phase C of the phase diagram . The detailed plot of phase C in Figure 4 confirms that train loss starts increasing earlier than the weight norm . When the weight norm becomes large , the effective learning rate becomes low and stops the process of divergence . After that , a new period of training begins . We also conducted an ablation experiment to show that the discovered periodic behavior is indeed a result of the competing influence of BN and WD on the weight norm . To do so , we prohibit this influence and train the network on a sphere by fixing the weight norm and rescaling the weights after each SGD step . We firstly fix the weight norm at random initialization , considering different values of the initialization weight norm and hence different ( fixed ) effective learning rates . Figure 5 ( left pair ) shows that in this case , there is no periodic behavior , and the train loss either converges ( for relatively low effective learning rates ) or gets stuck at high values ( for high effective learning rates ) . We repeat this ablation fixing the weight norm at some epoch preceding destabilization in the experiment where we observe the periodic behavior . Specifically , as an initial experiment , we use the one with the learning rate of 0.01 from Figure 3 and fix the weight norm at the 150-th and 180-th epochs , preceding the destabilization at epoch 200 . Figure 5 ( right pair ) shows the absence of the periodic behavior in both cases . When we fix the weight norm closer to the destabilization at the 180-th epoch , we observe a single increase in train loss , as the training process has already become unstable . However , after converging from this increase , train loss never destabilizes again .

Summary: In this paper, the authors look at scale-invariant networks trained with SGD using both batch normalization and weight decay. They study the contradiction between the equilibrium presumption and the instability presumption and claim that the training process converges to a consistent periodic behavior which includes some instabilities during training but does not lead to failure. They define the notion of delta-jumps for the dynamics of training and conjecture that the delta jumps are necessary for destabilization of the dynamics. Extensive empirical experiments are provided for supporting the claims.