On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
Authors: Ekaterina Lobacheva
Maxim Kodryan
Nadezhda Chirkova
Andrey Malinin
Dmitry P. Vetrov
Venue: NeurIPS 2021 Poster (2021)
Keywords: neural network training
batch normalization
weight decay

Abstract: Training neural networks with batch normalization and weight decay has become a common practice in recent years. In this work, we show that their combined use may result in a surprising periodic behavior of optimization dynamics: the training process regularly exhibits destabilizations that, however, do not lead to complete divergence but cause a new period of training. We rigorously investigate the mechanism underlying the discovered periodic behavior from both empirical and theoretical points of view and analyze the conditions in which it occurs in practice. We also demonstrate that periodic behavior can be regarded as a generalization of two previously opposing perspectives on training with batch normalization and weight decay, namely the equilibrium presumption and the instability presumption.

On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
Ekaterina Lobacheva'; Maxim Kodryan' Nadezhda Chirkova"
Andrey Malinin Dmitry Vetroy
HSE University Yandex 9AIRI
Moscow, Russia
elobacheva@hse. ru, mkodryanChse. ru, nchirkovaChse, am969@yandex- team..r dvetrovehse.ru
Abstract
Training neural networks with batch normalization and weight decay has become a common practice in recent years. In this work, we show that their combined use may result a surprising periodic behavior of optimization dynamics: the training process regularly exhibits destabilizations that, however, do not lead to complete divergence but cause a new period training. We rigorously investigate the mechanism underlying the discovered periodic behavior from both empirical and theoretical points of view and analyze the conditions in which occurs practice. We also demonstrate that periodic behavior be regarded two previously opposing perspectives on training with batch normalization and weight decay, namely the equilibrium presumption and the instability presumption.
generalization
Introduction
Normalization approaches, such batch layer normalization, have become vital for the successful
training of modern deep neural networks [2 2 24 21 27). Despite much recent work [ 22 9 28).
is still not completely understood how normalization influences the training process. In this work. we investigate the surprising periodic behavior that may occur when a neural network is trained with commonly used combination of some kind of normalization, in our case batch normalization (BN) 12]. and weight decay regularization (WD). Examples of this behavior are provided in Figure
The dynamics of neural network training with BN and WD have been examined extensively in literature due to the non-trivial competing influence BN and WD on the norm of neural network's weights. More precisely, using BN makes (a part of) neural network's 's weights scale-invariant, i.e., multiplying them by a positive constant does not change the network's output. Although scale invariance allows optimizing on a sphere with fixed weight norm (6. classic SGD--based approaches are usually preferred over constraint optimization methods in practice due to more straightforward implementation. Making an SGD step in the direction of the loss gradient always increases the norm of scale-invariant parameters, while WD aims at decreasing the weight norm (see illustration in Figure 2 In sum, training the neural network with BN and WD results in an interplay between two forces: "centripetal force" of the WD and the "centrifugal force" of the loss gradients. Many works notice the positive effect of WD on optimization and generalization caused by the control of the caale-invariant weights norm and the subsequent influence on the effective learning rate [25] [0 29 18 [9 26, 20), ie.. the learning rate on a unit sphere in the scale-invariant weights space. However, the general dynamics of the norm of the scale-invariane weights are viewed in the literature from two contradicting points, and this work is devoted to resolving this contradiction.
First two authors contributed equally.
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

ConvNet on CIFAR-I0
ResNet-18 on CIFAR-100
Train loss
Si weight norm Pr
Train loss
weight norm
10"
3x
10
10-1
2x
10a4
100
10a4
10"
100
3.0
250
100
200
300
200
spoch
Learning rate n1:
0.1
0.03
0.01
Learning rate nt:
0.1
0.03
0.01
Periodic behavior of ConvNet on CIFAR-10 10 and ResNet-18 on CIFAR- 100 trained using SGD with weight decay of 0.001 and different learning rates. All weights are trainable, including non-scale-invarian ones.
Figure I:
On the one hand, Li et al. [9 claim that learning with SGD, BN, and WD leads to an equilibrium state, where the "centripetal force" is compensated by the "centrifugal force" and eventually the norm of scale-invariant weights (along with other statistics related to the training procedure) will
converge to constant value. Several other works hold a similar equilibrium view [25
S 26 On
the other hand, a number of works [17]19 underline that using WD may cause approaching the
origin (zero scale-invariant weights), which results in training instability due to increasing effective learning rate. Particularly, Li al. [17 reveal that approaching the origin in weight-normalized neural networks leads to numerical overflow in gradient updates and subsequent training failure. Li and Arora [8] also underline that scale-invariant functions are ill-conditioned near the origin and prove in simplified setting that loss convergence is impossible if both BN and WD guaranteed if either of them is disabled). Morcover, despite their equilibrium view, Li et al. [19] empirically observe that the train loss permanently exhibits oscillations between low and high values when full- batch gradient descent is used.
used (but
In this work, we study the specified contradiction between the equilibrium presumption and the instability presumption and show that both are true only to some extent. Specifically, we show that the training process converges to consistent periodic behavior, i.e., it regularly exhibits instabiliites which, however, do not lead to complete training failure but cause new period of training (see Figure T. Thus, our contributions are as follows.
We discover the periodic behavior of neural network training with BN and WD and reveal its reasons by analyzing the underlying mechanism for fully scale-invariant neural networks trained with standard constant learning rate SGD (Section 4 or GD (Appendix C
We provide theoretical grounding for our findings by generalizing previous results on the equilibrium condition, analyzing the necessary conditions for destabilization of training, and relating the frequency of destabilization to the choice of hyperparameters (Section 5
We conduct a rigorous empirical study of this periodic behavior (Section 6 and show its presence in more practical scenarios with momentum, augmentation, and neural networks in- corporating trainable tar--salleivviiinl weights (Section) 7 and also with Adam optimizer and other normalization techniques (Appendix[].
Our source code available at http://hhw...com//p/ppp//raaaaaa wd
Background
As discussed in the introduction, batch normalization makes part of) neural network's weights scale- invariant. In this section, we describe the properties of scale-invariant functions, upon which we build
our further reasoning. Consider arbitrary scale-invariant function (x)) i.e., [(xx = f(x), Vr
and Va > O. Then two fundamental properties may be inferred, see Lemma 1.3 in Li and Arora [18]
(vf(z), = 0, Vr
(1a)
1 V((ax) 1 Vf(x), Vr, 0.
(1b)

---

Consider optimizing fx) w.r.t x using (S)GD with learning rate 7y and weight decay A:
V
Le+1 = -1--))aa=- fxx))
(2)
The properties above lead two important corollaries about the dynamics of the optimization process, First, according to property (Ta), shifting æ in the direction of fa)), i.e., making a gradient descent step, always increases || while weight decay, on the other hand, decreases || See Figure 2 for the illustration. The interaction of these "centripetal" and "'cntrifuuall" forces may cause ||>||] to change nontrivially during optimization. Second, according to property (Ib). even though function value f(x) is invariant to multiplying x by a, the optimization dynamics changes substantially when optimization performed at different scales of ||=]| For smaller norms, optimization makes larger steps, which may result in instabilitics, while for larger norms, steps are smaller, and optimization process may converge slowly.
4 ||r]
1 [|
Figure 2: An illustration of the "centripetal force" of the weight decay and the "centrifugal force" of the function gradient in the optimization of scale- invariant functions.
Since scale-invariant f (1) may be seen as a function on a sphere, its optimization dynamics are often analysed on a unit sphere ||>||| = 1. One can obtain equivalent optimization dynamics on the unit sphere as in the initial space by using the notion of effective grudient and effective learning rate. The effective gradient is defined as a gradient for a point on unit sphere and may be obtained by substituting
||x| in
lb V fz// ||=]]] fxx) ||a]]-
The effective learning rate be defined effective gradient by definition and is reflected only in the effective learning rate: the lower the norm, the higher the effective learning rate, and the larger the optimization steps.
n/ || 10
20 Change in ||>e|| does not affect the
3 Methodology and experimental setup
In order to isolate the effect of the joint use of batch normalization and weight decay and avoid the influence of other factors, we conduct series of experiments in simplified setting, when all learnable weights of a neural network are scale-invariant and optimization is performed using SGD with constant learning rate, without momentum or data augmentation. This allows us to better understand the nature of the periodic behavior (Section 4 and analyse its empirical properties (Section pp. After that, we return to the setting with more conventional training of standard neural networks and show that the periodic behavior occurs in this scenario as well (Section 7
We conduct experiments with ResNet- 18 and simple 3-layer batch- normalized convolutional neural network (ConvNet on CIFAR-10 15] and CIFAR- [16] datasets. To make standard networks fully ocale-invariant we rely on the approach of Li and Arora [18) i.e., we insert additional BN layers and fix the scale-invariant weights to be constant. Specifically, we use zero mean and unit variance in batch normalization layers instead of learnable location and scale parameters and freeze the weights of the last layer at random initialization. The latter action does not hurt the performance in practice [T]. However, we find that even with low train error, the training dynamics with the fixed last layer may still substantially differ from conventional training, as the neural network exhibits low confidence in predictions. To achieve high confidence for all objects and, consequently, low train loss, we increase the norm of the last layer's 's weights to 10. The influence of this rescaling is shown in Appendix G
We optimize cross-entropy loss, use the batch size of 128 and train neural networks for 1000 epochs to show the consistency of the discovered periodic behavior. We consider range of learning rates, {10=*,3. 3. 10 00 and choose the most representative ones for each visualization, since is difficult to distinguish many periodic functions on one plot. For fully scale-invariant neural networks, training with a fixed weight decay- - learning rate product converges to similar behavior, regardless of their ratio: we show it empirically and discuss it from the theoretical point of view in Appendix F.I the same was noticed in [18 [9] Thus, in the main text, we provide only the results for the varied
Since both stochastic and full-batch gradients a scale- inverinnv objective possess properties La and 1b we do not distinguish between them in our reasoning. Both architectures in the implementation of https:s:: / http//gttthhuu..ee poo/-- bentoon/hessiaan-fff--dt
3

---

loes
Text
Weight morth
Plase
Weight dccy 0.00
00-
30"
10"4
10"
10
rale 1
1.0
10"0
20
0.03
0.001
learning
Figure 3: Periodic behavior of scale-invariant ConvNet CIFAR-10.
Phase diagram
One training peried
Phose (destabiization)
6-uums phase B
10
-on betureen afteeed treetee
30
80
elt grastere nom
10.
20
B
10a
weight rere
20
1''
10"1
10
201
428
6.k
50k
1001
125k
715
725
735
145
526
100k
325k
effectiv leaming rate R
5t055
50D stops
128c)
500
sta00
Three phases one training perfoc:
loss starts
Bun possible 6-jumps
Phase
Phose
Prose
incraasing from
Day for guaranteed C-jumps
Figure 4: A closer look at one training period for SGD with weight decay of 0.001 and the learning rate of 0.01. Three phases of the training period are highlighted. The train loss and the effective gradient norm computed over a mini-batch are logged after each SGD step (one epoch consists of 391 SGD steps). The rightmost plot compares empirically observed cosine distance between weights at adjacent SGD steps with theoretically derived bounds in Section 5.1 Cosine distance is presented along with the smoothed trend.
ConvNet on CIFAR- 10 trained using
learning rate and the fixed weight decay of 0.001. Results for the varied weight decay are presented in Appendix F3
At each training epoch, we log standard train / test metrics, the norm of scale-invariant weights (SI weight norm), which is in the focus of this research, and metrics characterizing training dynamics on unit sphere: effective learning rate and the norm of effective gradients (mean over mini-batches We plot the two latter metries over two axes of the phase diagram to visualize their simultancous dynamics that will help us to understand the mechanism underlying the periodic behavior.
Periodic behavior and its underlying mechanism
As discussed in the previous section, we begin study by considering simplified setting with a fully scale-invariant neural network trained with standard SGD. Figure shows the presence of the periodic behavior for scale-invariant ConvNet the CIFAR-10 dataset for a range of learning rates. In Appendix F2 we show the presence of the periodic behavior for other aachset-architecture pairs. The same periodic behavior is also present for neural network training with full-batch gradient descent, see Appendix C Moreover, this behavior can be observed even when optimizing common scale-invariant functions using the gradient descent method with weight decay (see Appendix (E).
The observed periodic behavior occurs because of the interaction between batch normalization and weight decay. particularly because of their competing influence the weight norm, As discussed in Section 2 weight decay aims at decreasing the weight norm, while loss gradients aim at increasing the weight norm due to scale invariance caused by batch normalization (see Figure 2). These two forces alternately outweigh each other for quite long periods of training, resulting in periodic behavior.
Let us examine a single period in greater detail by analyzing Figure that shows the dynamics of relevant training metrics logged after each SGD step of ConvNet training. At the beginning of the period, the train loss is high, and the large gradients of the loss outweigh weight decay. This results increasing weight norm and decreasing effective learning rate, i.e., we move along phase A of the phase diagram. SGD continues optimizing train loss, and at some point, train loss and its gradients become small and outweighed by weight decay. As result, the weight norm starts decreasing, and the effective learning rate increases, i.c., move along phase B of the phase diagram. We note

---

Fix weight initialization
Fix weight before destabilization
Train loss
Test ertor,
Train loss
Test error, %
10
40
30
10-3
20
200
400
600
800
1000
200
400
600
500
1000
700
epoch
epoch
Fix SI weight norm / effective learning rate at:
inttial experiment destabiiizes al ep-200
Fix S1 we ihgttht norm ep.150
fix SI weight norm 1p:180
3.5.
7.8/
4.1/
Je-4
The absence of the periodic behavior for training with the fixed weight norm. Scale- invariant ConvNet on CIFAR-10 10 trained using SGD with weight decay of 0.001 and learning rate of 0.01. Left pair: the weight norm is fixed at random initialization of different scales, Right pair: the weight norm fixed at some epoch of regular training before destabilization.
Figure 5:
that the transition between phases A and B correlates with achieving near-zero train error. When the weight norm becomes too small, and the effective learning rate becomes too high, SGD makes several large steps and leaves the low loss region. Gradients grow along with train loss and, multiplied by high effective learning rate, lead to the fast growth of the weight norm, i.c., we move along phase C' of the phase diagram. The detailed plot of phase C in Figure f confirms that train loss starts increasing earlier than the weight norm. When the weight norm becomes large, the effective learning rate becomes low and stops the process of divergence. After that, a new period of training begins.
We also conducted an ablation experiment to show that the discovered periodic behavior is indeed a result of the competing influence of BN and WD on the weight norm. To do so, we prohibit this influence and train the network a sphere by fixing the weight norm and rescaling the weights after each SGD step. We firstly fix the weight norm at random initialization, considering different values of the initialization weight norm and hence different (fixed) effective learning rates. Figure 5 (left pair) shows that in this case, there is no periodic behavior, and the train loss either converges (for relatively low effective learning rates) or gets stuck high values (for high effective learning rates). We repeat this ablation fixing the weight norm at some epoch preceding destabilization in the experiment where observe the periodic behavior. Specifically, as initial experiment, we use the one with the learning rate of 0.01 from Figure 3 and fix the weight norm at the 150-th and 180-th epochs, preceding the destabilization at epoch 200. Figure 5 (right pair) shows the absence of the periodic behavior in both cases. When we fix the weight norm closer to the destabilization at the 180-th epoch, we observe a single increase in train loss, the training process has already become unstable. However, after converging from this increase, train loss never destabilizes again.
Theoretical grounding for periodic behavior
In this section, we theoretically investigate the reasons for the training destabilization between phases B and C, and after that, we generalize the overall training process equilibrium condition of Li et al. [19] taking into account the discovered periodic behavior. To do so, we study the optimization dynamics of an arbitrary scale-invariant function f(x) trained using (S)GD with learning rate n and weight decay of strength X 2 Hereinafter, we will assume that the ιιλ product is small, i.e., we can suppress O ((π))² terms. We also refer to Appendix A for the proofs, derivations, and further discussion on our theoretical results.
We recall that (stochastic) gradients of an arbitrary scale-invariant function f(x) possess two funda- mental properties (la) and (Ib). Based on these properties, we obtain the dynamics of the parameters norm induced by Eq. which we also leverage in our analysis (derivation of this and other equations is deferred to Appendix A.2
24 = (1-)))p))) + 1 72
where Pr ||x,|| denotes the parameters norm, St = ||V - the gradient norm, Øt [ ||x|) = Pt.et - the effective gradient norm. In this work, we also use the notion of effective learning rate which is formally defined 7 n/
5

---

5.1 The notion of 6-jumps
As scale-invariant functions are essentially defined on a sphere, cosine distance is a natural choice for a metric in parameter space. The following notion defines a situation when adjacent iterates become distant from each other, indicating training destabilization.
Definition 1 We say that dynamics performed 6-jump once the cosine distance between adjacent iterates exceeds some value > 0:
- Cos F(41) > 8.
We conjecture that the necessary condition for the training dynamics' destabilization is performing 5- 6-jumps with sensible values of 6. Otherwise, as long as adjacent iterates remain too close, the model (and hence its training dynamics) cannot change significantly. This holds strictly, for instance, the Lipschitz constant of f is bounded (at least locally), which can be relevant for neural networks with BN [22]. But even f has unstable regions on unit sphere with very high or even unbounded Lipschitz constant, our analysis is still relevant since making larger steps in such regions would lead higher chance of divergence. Further, we show that the closer we approach the origin, the larger effective steps (steps unit sphere) we start making, thereby paving the way for destabilization.
Now, our question is, given the value 8, what are the conditions for ô-jump to occur? By assuming that effective gradients are bounded, i.e., we can set two values 0 S É S L < +oo such that Tt E [e, LJ. we answer this question in the following proposition. Proof can be seen in Appendix A.3
Proposition 1 If (x) is scale-invarian function optimized according to dynamics 2 with bounded effective gradients S < 9. S L < +oo, then for sufficiently small and assuming (1 m7) 2 1. the following approximate conditions on 6-jump hold:
n
Pt
8 5-jump is possible,
(4a)
26
ne
Pi
6-jump is guaranteed.
(4b)
26
Remark Our results hold for any values e, L bounding the effective gradient norm, but the tighter these bounds are, the more precisely our theory describes the properties of the actual dynamics, thus we generally assume that È and L are taken as local bounds on 9t valid for several current iterations.
To connect our theoretical results with practice, we examine the behavior of effective steps length of scale-invariant neural network, compare it with theoretical bounds and observe gradually increasing destabilization of training dynamics. The rightmost plot of Figure 4 visualizes the cosine distance 1 cos(1,, #1+1) between neural network's weights 2; and 22+1 adjacent SGD steps for phase B of the training period. The dashed lines denote the theoretical upper and lower bounds on the cosine distance corresponding the maximal and minimal 6-jumps derived from Eq. (4ah and (4b),
2²²²" To obtain those, we calculated the network's 's parameters respectively: 8max = n² L² 8min = 2pp-
2pp
norm P: each iteration and chose E and L as smooth functions locally bounding the effective gradient norm in phase B (see Appendix D for details). We can see that both bounds, along with the measured cosine distance, start growing the second half of the phase. This indicates that the performed 5-jumps are gradually increasing. hence instability accumulates until the training diverges. We note that such long-lasting increase in cosine distance is common but, in general, not obligatory in the case of training with SGD because SGD may exhibit destabilization even with small 8- 6-jumps due to stochasticity. For full-batch GD, this effect is even more prominent, see Appendix C
Next, we formulate a proposition about how the initial parameters norm value Po and hyperparameters 77 and À affect the time of 5-jumps occurrence and hence the frequency of the periods since training dynamics destabilization is closely connected with d-jumps. Proof can be found in Appendix [A.5 Note that Po should be interpreted as the norm some initial moment t 0 of given period, when the conditions of the proposition are met, (typically, at the beginning of phase B) rather than the norm after initialization, i.e., at the very first iteration of training.

---

Proposition 2 Denote n V2x 2X
Under the assumptions of Proposition 1
1. if Pb > kl and nn fr, then the minimal time required for the ò-jump to occur:
log (pă - ke) - -log
Ke)
V2
min max
(5)
-log(1 477)
2. if p > KL and < nn then the maximal time required for the 8-jump to occur:
log (pB kL) - llo
nt
kLL)
mma 0,
(6)
-logll1 21jA)
Corollary Since both mmax and min inversely proportional to πd log(1 - 5) E for small E, 8-jumps (and hence periods) must occur more often for larger values of 12>
5.2
Generalization of the equilibrium condition
We now generalize the equilibrium condition of Li et al. [19 and characterize the behavior of the parameters norm globally the following proposition. The proof is provided in Appendix AA.
Proposition 3 Denote & v
I. 2X
Under the assumptions of Proposition 1 if 2f S l, then
ml S pi S KL, t>>> 1.
(7)
Furthermore, P > sI, then pt converges linearly to [Kl, x.I.] interval O (1/n)) time.
Note that Li et al. [19] and Wan et al. (66 similarly predict that the equilibrium state be reached
in linear rate regime. The condition 2n I & is generally fulfilled practice for small F product even for globally chosen bounds L. similar assumption made, e.g., in Theorem in Wan et al. 126]. We discuss it in more detail (including the non-fulfillment case) in Appendix A.7
Proposition 3 generalizes the results of Li et al. [19] who claimed that the effective learning rate i = =//p? converges to a constant. Their derivation relies on the assumption of stabilization of the effective gradient variance, which contradicts the observed periodic behavior. We relax this assumption by putting bounds on the effective gradient norm, thus bounding the parameters norm limits. These bounds can be either local, which defines the local trend of parameters or global, which describes its general behavior. Also, note that we, in some sense, extend the results provide the exact limiting interval for p not just bound its variance.
dynamics,
of Wan et al. 26
6 Empirical study of the periodic behavior
After discussing the reasons for the occurrence of the periodic behavior, we now further analyze its properties. particular, investigate: how hyperparameters affect the periodic behavior, how the periodic behavior evolves over epochs, and how minima achieved in different training periods differ both in parameter and functional space. In this section, we again consider the simplified setting with neural network trained with standard SGD.
a fully
Influence of perrpaarametess. We investigate the influence of key training hyperparameters: learning rate and weight decay, but since the training dynamics mainly depend on their product (see discussion in Section and Appendix F.T.. we only vary the learning rate. The results for ConvNet on CIFAR-10 10 are given in Figure 3, the results for other mataset-architectte Appendix
pairs are presented in F22] and the results on the variable weight decay are given Appendix F3 Our first observation is that with higher learning rates, consistent periodic behavior occurs at larger weight norms. This because SGD with a high learning rate can only converge with relatively small gradient norms, which are achieved at large weight norms according to Eq. [bb.. This observation also agrees with Proposition]] 3 in Section 5 The second observation is that periodic behavior present for wide range of learning rates, c.g., 0.003 0.3 for ConvNet CIFAR-10, and the higher the learning
7

---

ConvNet on CIFAR-10
ResNet-18 on CIFAR-100
Ensemblo test amrr, %
Cosine sim. between checkpoints
Enserblle tast arror, %
Cosina sim. between checkpoints
1.00
1,00
0.75
0.75
15
14-
0.25
anchor checkpoint
0.00
*
anchor checkpoint
0.00
200
600
500
1000
400
600
10000
400
800
1000
600
1000
edoeh
*poch
Initial
Ensemblatcos with
Ensembie/cos with
Initial
Ensembleiios with enchor checkpoint
Ensemblercos with independent. network
exp.
anchor checkpoint
independent network
exp.
Figure 6: Similarity in the weight space (cosine sim.) and the functional space (ensemble test error) for different checkpoints of training scale-invariant ConvNet on CIFAR-10 (left pair) and ResNet on CIFAR-10M (right pair) using SGD with weight decay of 0.001 and learning rate of 0.03.
When using a learning rate that is too high, e.g. in Figure 3 we expect training to yield very large weight norms, however weight decay prohibits us from reaching them, thus the gradients are not able to shrink sufficiently, and training gets stuck at high train loss. On the other hand, using a learning rate that too low, e.g.. 0.001 in Figure 3 leads to prolonged training which does not reach a small enough weight norm to yield a high effective learning rate, resulting in the absence of the periodic behavior in the given number of epochs. We note that the periodic behavior is present for the learning rates giving the lowest test error. Appendix [33 we show that varying weight decay leads to similar effects: the periodic behavior is present for wide range of reasonable weight decays but is absent for too low or too high weight decays, and the higher the weight decay, the faster the periods.
rate, the shorter the periods, which agrees with Corollary in Section 5
Dynamics of periodic behavior. We now analyze how the discovered periodic behavior evolves over epochs. As discussed in the previous paragraph, consistent periodic behavior occurs at larger weight norms with higher learning rates. However, the initialization may have substantially different norm. Thus we observe a warm-up stage in some plots of Figures and 3 when the beginning of training spent on moving towards the appropriate norm of scale-invariant weights. Expectedly, this warm-up up stage is more prolonged for lower learning rates. Reaching the proper weight norm initiates a consistent periodic behavior. During the warm-up stage, SGD can still exhibit regular destabilization happening at higher weight norms than in the stage of consistent periodic behavior. We hypothesize that at the early stage of training, SGD converges to less stable basins with larger effective gradients, in which destabilization happens at larger norms of the scale-invariant parameters. We notice that test error decreases after each warm-up destabilization episode and reaches lower level than training with a fixed effective learning rate, as shown in Figure 5 (right pair). In other words, the performance may benefit from the repeating destabilization.
Minima achieved at different training periods. Next, we aim understanding whether minima achieved in different training periods are close in weight and functional spaces. We use the cosine similarity function for the weight space and estimate similarity the functional space by comparing with ensembles of independent models, following Fort et al. [8. If training process converges to the same minimum in each period, then cosine similarity between two minima achieved in different periods should be close to one and their ensemble error should be close to the error of single network. On the contrary, if destabilization moves training so far that is equivalent to retraining a model from a new random initialization, then the cosine similarity between the two minima should be close to zero and their ensemble error should be close to the error of an ensemble of two independently trained networks.
The setup of the experiment is as follows. We select some initial experiment and its checkpoint (called anchor checkpoint) corresponding to the minimum achieved when the training process has already converged to the consistent periodic behavior. After that, we measure weight/funntion similarities between the anchor checkpoint and all the subsequent checkpoints of the initial experiment this is our primary measurement. For comparison, we independently train one more neural network with the same hyperparameters as in the initial experiment but from different random initialization, select its checkpoint with the same test error as that of the anchor checkpoint, and measure the similarity between this new checkpoint and the checkpoints of the original experiment.
8

---

Add modifications one time
Add all modifications together
Train loss
Test ortor, 9
Train loss
Test arror,
1.5
50
10"4
10-3
10
1012
200
400
600
800
1000
200
400
600
500
1000
200
400
600
500
1000
200
400
600
800
1000
epoch
epoch
eporh
epoch
Inital experiment
Tull training
augmentation
Weight decay 0.0001, leaming rate:
momentum
0.03
0.003
0.001
Figure 7: Periodic behavior of ConvNet (of increased width) on CIFAR- 10 trained with the more practical modifications. Leftt: weight decay of 0.001, learning rate of 0.01.
The results for ConvNet on CIFAR-10 and ResNet- 18 CIFAR-100 presented in Figure 6 the results for other aataset-architecture pairs are given in Appendix H Inside one training period. checkpoints do not step far from the anchor checkpoint, ie.. the cosine similarity is close one, and the ensemble test error is close to the error of a single network. However, when the next training period begins after destabilization, SGD moves to another region in the weight space, and both similarities start decreasing: the cosine similarity drops, and ensemble test error becomes smaller than that of single network. Each following training period moves SGD farther away from the anchor checkpoint. For networks on CIFAR-100, late training periods continue to be correlated with the anchor checkpoint, i.c., the cosine similarity only reaches 0.5 value, and ensemble test error does not reach the level of the independent networks ensemble. Still, both similarities continue decreasing. For networks on CIFAR-10. 10, the cosine similarity decreases faster, and the ensemble test error quickly reaches the test error of an ensemble of two independently trained networks. To sum up, minima achieved two neighboring training periods are substantially different, but their similarity usually higher than that of two independently trained networks.
Periodic behavior in a practical setting
In the previous sections, we conducted experiments with scale-invariant neural networks trained with the simplest version of SGD. This allowed us to analyze the periodic behavior of train loss in detail. However, in practice, a portion of the weights of neural network are not scale- invariann,, e.g., the weights of the last layer and learnable BN parameters. Furthermore, networks are trained using more advanced procedures, e.g., SGD with momentum, data augmentation, and a learning rate schedule. At the same time, periodic behavior was mainly not noticed in previous works to the best of our knowledge. In this section, we show the presence of the periodic behavior for standard neural networks trained with momentum and data augmentation and discuss why periodic behavior may be not observed in practice. In Appendix ! networks with other normalization approaches or trained with Adam [3].
we also show the presence of the periodic behavior for the
We select one of our initial experiments and add modifications one at a time to see their effects more clearly. We also present the results for training with all modifications turned on together. The plots for ConvNet on CIFAR-10 are given in Figure 7 and for other setups - Appendix[] [ In this section, we use wider ConvNet, as the standard version is too small to learn the augmented dataset.
Training non-scale-invariant weights. To achieve full ecale-invarianee we froze the weights of the last layer and the parameters of BN layers since they all are not scale-invariant. We now consider the conventional procedure that implies training all neural network weights. In addition to the results presented Figure (left), we refer the reader to Figure We observe that training non-scale-invarian weights retains the periodic behavior and affects the frequency of periods. The last-mentioned effect relates to the trainable last layer that automatically adjusts prediction confidence. In Appendix we show that variable prediction confidence results in different periodic behavior.
G
SGD with momentum. Next, we investigate the effect of using a more complex optimization algorithm. We consider SGD with momentum as the algorithm most commonly used for training convolutional neural networks. We observe that using momentum does not break the periodic

---

behavior and increases the frequency of periods. This agrees with the commonly observed effect that momentum speeds up training [23], i.e. momentum accelerates phases A and B. Interestingly, momentum does not prevent destabilization.
Data augmentation. We next consider training on the dataset with standard CIFAR-I0 data augmentations, see details in Appendix B Augmentation prevents over-fitting to the training data, which results in less confident predictions and larger train loss. As a result, train loss gradients outweigh WD more easily. If the number of parameters in the neural network is insufficient to achieve low train loss gradients, phase A never ends (at least in 1000 epochs), resulting in the absence of the periodic behavior. On the other hand, sufficiently large neural network learns the augmented dataset at some epoch and proceeds to phase B, launching the periodic process. Still, we note that the periodic behavior begins much later than for the network trained without augmentation. This is one of the main reasons why the periodic behavior is often not observed in practice: it requires a much larger number of epochs to start than conventionally used for training.
All modifications together. In the two right plots of Figure 7 we visualize training with momentum, data augmentation, and unfrozen non-scale-invarian parameters used simultaneously and observe the presence of the periodic behavior.
So, what factors do interfere with observing the periodic behavior in practice? We underline two main factors. First, the interplay between different modifications narrows the range of hyperparameter values for which periodic behavior is present. When non-scale invariant parameters are trained, the model converges to low test error only with specific values of weight decay. Moreover, with data augmentation, periodic behavior occurs only with relatively high learning rates (with lower learning rates, the training is too slow to reach phase C' in 1000 epochs), while with momentum, using too high learning rates may result in training failure in phase A. In sum, periodic behavior appears only for a limited range of hyperparameters, Despite that, we note that the model generally achieves its best performance exactly in this range. Second, practical settings also imply learning rate schedules and relatively small number of epochs, which do not preserve periodic behavior. We provide further discussion on comparison of our experimental setup with other works in Appendix
non-scale-invariant
Conclusion
In this work, we described the periodic behavior of neural network training with batch normalization and weight decay occuring due to their competing influence on the norm of the scale-invariant weights. The discovered periodic behavior clarifies the contradiction between the equilibrium and instability presumptions regarding training with BN and WD and generalizes both points of view. In our empirical study, we investigated what factors and in what fashion influence the discovered periodic behavior. In our theoretical study, we introduced the notion of ó-jumps to describe training destabilization, the cornerstone of the periodic behavior, and generalized the equilibrium conditions in way that better describes the empirical observations.
Limitations and negative societal impact. We discuss only conventional training of convolutional neural networks for image classification and do not consider other architectures and tasks. However, we believe that our findings extrapolate to training any kind of neural network with some type of normalization and weight decay. We also focus on particular source of instability induced by BN and WD, yet, other factors may make training unstable [7]. This is an exciting direction for future research. To the best of our knowledge, our work does not have any direct negative societal impact. However, while conducting the study, we had to spend many GPU hours, which, unfortunately, could negatively affect the environment.
Acknowledgments, and Disclosure of Funding
We would like to thank Sofia Sibagatova for the help with Appendix The work was supported in part by the Russian Science Foundation grant No N-1977--3000 The empirical results were supported part
through the computational resources of HPC facilities at HSE University [14]. Additional revenues of the authors for the last three years: laboratory sponsorship by Samsung Research, Samsung Electronics; travel support by Google, NTNU, DESY. UCM.
10

---

References
[1] Arora, S., Li, Z.,, and Lyu, K. (2019). Theoretical analysis of auto rate-tuning by batch normaliza- tion. In International Conference Learning Representations.
[2] Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:
[3] Bjorck, N., Gomes, P., Selman, B., and Weinberger, Q. (2018). Understanding batch normalization. Advances in Neural Information Processing Systems, 31.
[4] Carmon, Y., Duchi, J.. Hinder, O., and Sidford, A. (2017). Lower bounds for finding stationary points ii: First-order methods. Mathematical Programming. 185.
[5] Chiley, V., Sharapov, I., Kosson, A., Koster, U., Reece, R., Samaniego de la Fuente, S., Subbiah, V., and James, M. (2019), Online normalization for training neural networks. Advances in Neural Information Processing Systems. 32.
[6] Cho, M. and Lee, J. (2017). Riemannian approach to batch normalization. Advances in Neural Information Processing Systems, 30.
[7] Cohen, J., Kaur, S., Li, Y., Kolter, J. 7.,, and Talwalkar, A. (2021). Gradient descent on neural networks typically occurs at the edge of stability. In International Conference Learning Representations.
[8] Fort, S., Hu, H., and Lakshminarayanan, B. (2019). Deep ensembles: A loss landscape perspec- tive. arXiv preprint arXiv: 1912002777
[9] Ghorbani, B., Krishnan, S., and Xiao, Y. (2019). An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning, pages 2232-2241. PMLR.
[10] Hoffer, E., Banner, R., Golan, I., and Soudry, D. (2018a). Norm matters: efficient and accurate normalization schemes in deep networks. Advances in Neural Information Processing Systems, 31.
[11] Hoffer, E., Hubara, I., and Soudry, D. (2018b). Fix your classifier: the marginal value of training the last weight layer. In International Conference on Learning Representations.
[12] loffe, and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR.
[13] Kingma, D. P. and Ba, (2015). Adam: method for stochastic optimization. International Conference on Learning Representations.
[14] Kostenetskiy, P. S., Chulkevich. R. A., and Kozyrev, I. (2021). HPC resources of the higher school of economies. Journal of Physics: Conference Series, 1740:012050.
[15] Krizhevsky, A., Nair, V., and Hinton, G. CIFAR-I0 (canadian institute for advanced research).
[16] Krizhevsky, A., Nair, V., and Hinton, G. CIFAR-100 (canadian institute for advanced research).
[17] Li,X., X., Chen, S., and Yang, J. (2020a). Understanding the disharmony between weight normal- ization family and weight decay. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 471 4722.
[18] Li, Z. and Arora, S. (2020). An exponential learning rate schedule for deep learning. In International Conference on Learning Representations.
[19] Li, Z., Lyu, K., and Arora, S. (2020b). Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate. Advances in Neural Information Processing Systems, 33.
[20] Roburin, S., de Marin Y., Bursuc, A., Marlet, R., Pérez, P., and Aubry, M. (2020). spherical analysis of adam with batch normalization. arXiv preprint 1333222006.13382.
11

---

[21] Salimans, T. and Kingma, D. P. (2016). Weight normalization: a simple reparameterization to accelerate training of deep neural networks. Advances in Neural Information Processing Systems, 29.
[22] Santurkar, S., Tsipras, D., Ilyas, A., and Madry, A. (2018), How does batch normalization help optimization? Advances in Neural Information Processing Systems, 31.
[23] Sutskever, I., Martens, J., Dahl, G., and Hinton, G. (2013). On the importance of initialization and momentum in deep learning. In International conference machine learning, pages 1139- 1147. PMLR.
[24] Ulyanov, D., Vedaldi, A., and Lempitsky, V. (2016), Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: 1607.08022.
[25] Van Laarhoven, (2017). 1.2 regularization versus batch and weight normalization, arXiv preprint arXiv: 1706.05350.
[26] Wan, R., Zhu, Z.,, Zhang, X., and Sun, J. (2020). Spherical motion dynamies: Learning dynamics neural network with normalization, weight decay, and sgd. arXiv preprint arXiv:2006.084/1
[27] Wu, Y. and He, K. (2018). Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19.
[28] Yang, G., Pennington, J., Rao, V., Sohl-Dickstein, J., and Schoenholz, S. (2019). A mean field theory of batch normalization. In International Conference on Learning Representations.
[29] Zhang, G., Wang, C., Xu, B., and Grosse, R. (2019). Three mechanisms of weight decay regularization. In International Conference on Learning Representations.
12


Reviews:
[{"review_id": "xW3C72Q0pgq", "reviewer": "Reviewer_ERRM", "summary": "The paper shows that commonly used Batch Norm and Weight Decay on networks with frozen last layer exhibit periodic behavior in training. The authors provide theoretical guarantees and also study extensions of their results to more commonly used settings and show where their results break down.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Building upon some previous work in the theory of deep learning field, the authors show both theoretically and experimentally that for a network trained without last layer, Batch Norm and Weight Decay have opposing actions on control of the norm, leading to a periodic behavior in training. \n\nIt is interesting that despite this rather atypical setting, the authors then expand their experiments and relax the assumptions one by one to reach the more typical regime of training. It is interesting to note that the periodicity of the behavior is broken by momentum, and not visible unless one uses a relatively large learning rate, which is something this reviewer has previously also noticed while training NNs with large learning rates on many occasions.\n\nIt would be interesting to see how these results coexist with the setting of all layers being trained, where it is expected that Neural Collapse (see V. Papyan et al. \"Prevalence of neural collapse during the terminal phase of deep learning training\"), and hence convergence, happens when we train deep nets with both Batch Norm and Weight Decay, see for example:\n- A. Rangamani, \"Dynamics and Neural Collapse in Deep Classifiers trained with the Square Loss\" \n- T. Ergen and M. Pilanci,  \"Revealing the structure of deep neural networks via convex duality\"\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "lcuQYMaG426", "reviewer": "Reviewer_6wdK", "summary": "The paper tackles the issue of explaining periodic behavior that can occur when jointly training neural networks using batch normalization and weight decay, and that causes training to alternate between phases of instability and stability.\nThe authors give a theoretically reasonable explanation of why this phenomenon occurs. They derive the notion of delta-jumps in parameter space that are responsible for training instabilities, and derive theoretical bounds based on parameter norm, effective gradient norm, and learning rate.\nThey investigate this mechanism empirically by training scale-free CNNs on the CIFAR-10 and CIFAR-100 dataset classification task, and also extend their analysis to more practical scenarios, showing that the periodic behavior can still be produced in less artificial settings.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper seems to provide a significant contribution in that it resolves the standing contradicting between the “equilibrium point vs. instability” view of training with batch normalization and weight decay. However, while the authors themselves point out that this effect can in principle also be observed in non-scale free networks, they make clear that it is of limited practical relevance due to the narrow parameter range in which it can occur.\nThe paper outlines its general ideas in a comprehensible way. However, some of the connections between theory and experiment could be made clearer, e.g. how the theoretically derived size of delta-jumps, convergence times etc. relates to the outcomes of their experiments. The rightmost plot in figure 3 was also a little confusing. The results are also generally shown without any statistics.\n\nSome questions that could be clarified:\n\n1. What happens for optimizers that use adaptive learning rates, such as Adam?\n2. With what consistency does the loss achieve a lower minima after a phase of instability? Figure 6 makes it seems like this is the case, while in Figure 4 the result looks different.\n3. “Making an SGD step in the direction of the loss gradient always increases the norm of scale-invariant parameters, while WD aims at decreasing the weight norm.” It is not directly apparent what this is based on. However, as this is the theoretical basis for the entire paper, it would be good to make the citation clearer and explain the mathematical reasoning in more detail.\n\nSome minor comments:\n- Using either epoch or SGD step (or making clearer how they relate to each other) consistently in the x-axis of figure 2+3\n- Warm-up phases can't be seen too well from the plots\n- coarse grain choices of learning rate. What happens in the range between 0.01 and 0.001?\n- With which probability do the delta jumps that lead to instabilities actually occur? Do they happen with certainty if curve is in given interval, and does it always happen?\n- The description of the ensemble method in 5) was a little confusing", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12 hours"}, "extra_text": {}}, {"review_id": "PuEnYJT_ESX", "reviewer": "Reviewer_JkC9", "summary": "This paper studies an interesting contradiction about neural networks with Batch Normalization (BN) and weight decay (WD): some works show that BN and WD can cause instabilities in training, while other works show convergence to an equilibrium that stabilizes training metrics. After conducting very extensive experiments, the authors claim that the training process converges to a consistent periodic behavior, which regularly exhibits instabilities and restarts a new period of training.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really appreciate the authors' work for their extensive experiments across different settings. The experiment setups are stated clearly and the figures are also very clear. I learned a lot from those experiments and I'm convinced that the periodic behavior could be more general than the instabilizing view and the equilibrium view from previous works. I would like to vote for acceptance.\n\nFor the theory part, the propositions in the current form are not very satisfactory. Note that the theory is expected to explain the periodic behavior, rather than just the first period. This means $\\rho_0$ in Proposition 2 should be interpreted as the initial norm at the beginning of each period and thus cannot be set arbitrarily except for the first period. Thus, a better theorem should upper and lower bounds $\\rho_0$ by analyzing the dynamics of previous periods and provide bounds for the time period without $\\rho_0$.\n\nRegarding why this work and Li et al. [17] produce different results, I suspect the most important factor could be the noise. When the noise is large and the learning rate is small, the SDE approximation in Li et al. [17] is valid and the convergence to equilibrium should occur. However, this paper turns off the data augmentation in most of the experiments, and thus the noise may not be sufficient.\n\nI believe it could greatly benefit future research if the authors can compare their experiment setups with previous works. Section 6 has already done a very good job, but it could be more helpful if the authors can point out the most important factor for why previous experiments do not show this periodic behavior, specifically for each the works they are directly following (e.g., Li et al. [15], Li et al. [17]).\n\n(The authors sometimes omit many second-order terms like  $O(\\delta^2)$ or $O((\\eta \\lambda)^2)$ in the derivation. I would suggest the authors use $\\approx$ and $\\lesssim$  instead of $=$ and $\\le$ when such approximation is used. E.g., Proposition 1.) \n\n-----------------\n\nThe rebuttal addressed most of my concerns. Although the theory part in the current form is not very strong, the empirical study in this paper is very nice and the period behavior is very interesting.  I would like to keep my score for acceptance and I look forward to seeing the camera-ready version of the theory part.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As stated in Section 6, the main limitation of this work is that periodic behavior mainly occurs on fully scale-invariant neural nets training for a sufficiently long time. Nevertheless, the training dynamics when $t \\to \\infty$ is a fundamental problem to study in the theory of deep learning, and the authors made a good contribution to this problem in the setting they adopted.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "CC3HzuJvcI", "reviewer": "Reviewer_VdoN", "summary": "In this paper, the authors look at scale-invariant networks trained with SGD using both batch normalization and weight decay. They study the contradiction between the equilibrium presumption and the instability presumption and claim that the training process converges to a consistent periodic behavior which includes some instabilities during training but does not lead to failure. They define the notion of delta-jumps for the dynamics of training and conjecture that the delta jumps are necessary for destabilization of the dynamics. Extensive empirical experiments are provided for supporting the claims.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is written clearly in most parts and the periodic observation is interesting. I also appreciate the extensive experiments. My main concern about this paper is that the observation is limited to a very specific setting and this periodic behavior is usually not observed in practice. Therefore, I believe the problem is not well motivated in the first place and even for this special case, the theory to support the claims is not complete. I am not convinced by the authors’ conjecture that delta-jumps are necessary for training dynamics destabilization. I also have the following comments/questions:\n\n1. It is noted in [1] that destabilization can happen due to the increase in sharpness of the network (the maximum eigenvalue of the training loss Hessian). Have you done any observations regarding this? \n\n2.  Lines 108 to 121: the explanation is insightful but very intuitive and without any figures or any exact expressions. It would be great if you could write the exact expressions and relations. For example, what do you exactly mean by effective learning rate? \\frac{\\eta}{\\norm{x_t}^2} \\proportional \\sqrt{eta lambda}? or Maybe you could show the centrifugal and centripetal forces on a plot.\n\n3. It would be great if you could discuss how limiting the assumption of bounded effective gradients is since this is the key element in generalizing proposition 3.\n\n4. There is no Figure 7.  Please fix the number.\n\n\n\n[1] https://arxiv.org/pdf/2103.00065.pdf\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the limitations of the work and the is no potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three knowledgeable reviewers recommend this paper for acceptance and one more knowledgeable reviewer rates it as marginally below the acceptance threshold. After the discussion period, one reviewer concludes that although the theory part in the current form is not very strong, the empirical study in this paper is very nice and the periodic behaviour is very interesting. Another reviewer comments that s/he stands by her/his opinion that the discovery of the periodic behaviour is very interesting and warrants being shared with the NeurIPS community. Another reviewer indicates that although some of her/his main concerns still stand, s/he is convinced the extensive experiments are insightful and could be relevant in more practical cases and increased her/his score. In summary, after the rebuttal and discussion the general consensus among the reviewers is for accepting this paper. Hence I am recommending the paper for acceptance with the request that the authors take the reviewers’ comments and suggestions carefully into account for the preparation of the final manuscript. 
Decision: Accept (Poster)