Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes
Authors: Maxim Kodryan
Ekaterina Lobacheva
Maksim Nakhodnov
Dmitry P. Vetrov
Venue: NeurIPS 2022 Accept (2022)
Keywords: neural network training
scale invariance
batch normalization

Abstract: A fundamental property of deep learning normalization techniques, such as batch normalization, is making the pre-normalization parameters scale invariant. The intrinsic domain of such parameters is the unit sphere, and therefore their gradient optimization dynamics can be represented via spherical optimization with varying effective learning rate (ELR), which was studied previously. However, the varying ELR may obscure certain characteristics of the intrinsic loss landscape structure. In this work, we investigate the properties of training scale-invariant neural networks directly on the sphere using a fixed ELR. We discover three regimes of such training depending on the ELR value: convergence, chaotic equilibrium, and divergence. We study these regimes in detail both on a theoretical examination of a toy example and on a thorough empirical analysis of real scale-invariant deep learning models. Each regime has unique features and reflects specific properties of the intrinsic loss landscape, some of which have strong parallels with previous research on both regular and scale-invariant neural networks training. Finally, we demonstrate how the discovered regimes are reflected in conventional training of normalized networks and how they can be leveraged to achieve better optima.

Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes
Maxim Kodryan', Ekaterina Lobacheva's Maksim Nakhodnov Dmitry Vetroy
HSS University FAIRI
Moscow, Russia
mkodryan@hse.rr ru, elobacheva@hse. ru, nakhodmovv22221.om org, dvetrov@hse
Abstract
A fundamental property of deep learning normalization techniques, such as batch normalization, is making the pre-normalization parameters scale invariant. The intrinsic domain of such parameters is the unit sphere, and therefore their gradient optimization dynamics can be represented via spherical optimization with varying effective learning rate (ELR), which was studied previously. However, the varying ELR may obscure certain characteristics of the intrinsic loss landscape structure. this work, we investigate the properties of training
neural networks directly on the sphere using a fixed ELR. We discover three regimes of such training depending on the ELR value: convergence, chaotic equilibrium, and divergence. We study these regimes detail both on a theoretical examination of a toy example and on thorough empirical analysis of real scale-invariant deep learning models. Each regime has unique features and reflects specific properties of the intrinsic loss landscape, some of which have strong parallels with previous research on both regular and scale-invariant neural networks training. Finally, we demonstrate how the discovered regimes are reflected in conventional training of normalized networks and how they can be leveraged to achieve better optima.
Introduction
Most modern neural network architectures contain some type of normalization layers, such as Batch Normalization (BN) [00 or Layer Normalization [2]. Normalization makes networks partially scale- invariant (SI), i..., multiplication of their parameters preceding the normalization layers by a positive scalar does not change the model's 's output. In general, the training dynamics of model parameters can be viewed from two perspectives: the direction dynamics, i.e., the dynamics of the projection of the parameters onto the unit sphere, and the norm dynamics. For SI neural networks, the former seems be more important since the direction alone determines the output of the scale-invariant function. However, the latter influences the optimization by changing the effective learning rate (ELR), i.c., the learning rate (LR) a scale-invariant model would have if were optimized on the unit sphere.
Many works have studied the effect of scale invariance on training dynamics through the prism of norm or, equivalently, ELR dynamics [3] 38 24
27 They discovered that during standard training with weight decay the ELR can change in non con-trivial way and lead to different
26
29
35
optimization dynamics: periodic behavior 227, destabilization 22 24 or stabilization on some
sphere [26] 35 However, considering only the ELR dynamics is not enough for a comprehensive study the specifics of training normalized neural networks. particular,
impact of the varying ELR on training may obscure certain important properties of SI neural networks' intrinsic domain, ie., the unit sphere, thus narrowing our intuition about the loss landscape structure. Therefore, in this work, we are focused on the direction dynamics. To eliminate the influence of the changing norm First three authors contributed equally.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Mean gradient
10-
1e-05
3
9.0001
10
9.0005
gradhere
20P
9,001
250
epech
503
750
1000
0
250
opoch
530
150
1005
250
spach
500
750
1000
0.002
0.05
0.005
10:
100
error, 16
Final mean grad
0.1
10
30
Starting point
10
10
EUR 10F
10
10t4 10
EUR 107
30
10
15a4
10
107
10"4
10:1
10"
30"
100
30
ELR
train
Figure 1: Three regimes of SI neural network training on the sphere: (1) convergence for the lowest ELRs, (2) chaotic equilibrium for medium ELRs, and (3) divergence for the highest ELRs. ConvNet on CIFAR-10. 10. Dashed lines on scatter plots denote borders between the regimes.
training, we fix the total norm of all scale-invariant parameters of the normalized neural network and train it with a constant effective learning rate the sphere.
We discover three regimes of such an optimization procedure depending on the ELR value (see Figure[] D.. The first regime (small ELRs) can be considered as a typical convergence to minimum with monotonically decaying training loss. The second one (medium EL.ss) demonstrates a sustained oscillating behavior of the loss around certain value separated from both the global minimum and the random guess behavior. We call this regime a chaotic equilibrium, reminiscent of the equilibrium state reported in several previous works [26 335 The last regime (high ELRs) represents destabilized, diverged training associated with an excessively large optimization step size.
We analyze the underlying mechanisms behind each of these regimes on a toy example and conduct an extensive empirical study with neural networks trained on the sphere using projected stochastic gradient descent (SGD). We point out the main features of each regime and study how they relate to each other and what peculiarities of the optimization dynamics and intrinsic structure of the loss landscape they reveal. Our findings both reconfirmed some of the previous beliefs, like that training with larger (E)LR generally ends up better optima, or that high-sharpness zones are present in the optimization trajectory. and introduced novel insights, like that different (E)LRs of the first regime even with the same initialization and batch ordering actually lead to distinguishable minima with different high-sharpness zones are responsible for the boundary between the first and the second regimes and have close ties with the Double Descent phenomenon. Finally, we demonstrate how our results are reflected in conventional neural network training and suggest several practical implications that help achieve optimal solutions in terms of sharpness and generalization.
profiles depending on the specific (E)LR value, that the
Our code is available at haatte///ggiiabb..o/////t//////p
on_ ottpemesss_o_ttteseee_eeeeeee
2 Background and related work
As mentioned in the introduction, for every function F(0) optimized w.r.t. its parameters O we can separate the dynamics of the parameters direction 0/ [[@]] from the dynamics of their norm ||@]]. If the function F(0) is scale-invariant, i.e., F(cØ) = F()), Ve >> 0, the former is the only factor that determines the value of the function, since F(0/ [[0]]] = F(O). However, even though the SI function's 's value does not depend on the parameters norm, the latter greatly influences its gradients: Fc(O = F (00)/c, Ve > 0 and thus the optimization speed. This argument can be equivalently reformulated in terms of how the norm dynamics control the direction dynamics' speed via the notion of effective learning rate. For the standard (S)GD optimization, ELR is defined as the learning rate divided by the squared parameters norm, and it represents how the learning rate should be adapted to mimic the same direction dynamics if the function were optimized directly on the unit sphere (29].
Most of the previous work has dealt with the implications of using normalization in neural networks, i... making them (partially) scale-invariant, from the perspective of the ELR dynamics. For exam- ple, Arora et al. [ study the convergence properties of batch normalized neural networks and explain why they allow much higher learning rates compared to their non-BN counterparts. Many works have
also focused on training scale-invariant neural networks with weight decay [33 38
24
26
35 27

---

They all discovered the non-triviaa dynamics of ELR induced by the interplay between normalization and weight decay but came different conclusions. Some studies have reported training instability caused by this interplay [22 24,, while others have argued that, on the contrary, the equilibrium state is
eventually reached, i.e., ELR becomes stable after some time [3 26
20 35 Recently, Lobacheva
et al. [27] demonstrated that training with weight decay and constant LR may experience regular periodic behavior for a practical range of LRs.
However, while all of these works mainly study the direction dynamics with varying rate, they lack exploring the properties of optimization of SI models with a fixed effective learning rate. Decoupling the direction dynamics from the changing norm effects could reveal particular features of the intrinsic domain of scale-invar invariant neural networks that were potentially obscured by the dynamical ELR. To cope with it, we optimize our models directly on the sphere using projected SGD with a constant learning rate, which is equivalent to fixing the ELR. Our method reveals some interesting results about SI neural networks and allows to draw parallels with previous scattered deep learning work. We note that similar experiments were conducted by, e.g., Arora et al. 1 and Lobacheva et al. 27 to highlight certain features of standard training when ELR is dynamical. Also, Cho and Lee [5] optimized scale-invariant parameters in normalized neural networks directly on their intrinsic domain using Riemannian gradient descent, but they proposed an optimization method rather than investigating the properties of such training.
3 Theoretical analysis
To explain the observed behavior, we proceed with the theoretical analysis of scale-invariant functions optimization with a fixed effective learning rate. At first, we derive several important general properties of such process that will help to clarify some of its empirical aspects. Then we consider concrete example of a function with multiple scale-invariant parameter groups, similar to normalized neural networks, and study its convergence depending on the ELR value to shed more light on the prime causes of the three regimes. We provide all formal derivations and proof' in Appendix A
3.1 General properties
We begin with an analysis of common features of all SI functions optimization with fixed learning rate on the sphere. For that, consider a function F F(0) of a vector parameter O E R " that can be divided into groups: 0 = (0,,.... ,O,). where each 0, RR and Σ P3 P. We say that each of these groups is scale-invariant, i.e., we can multiply it by a positive scalar value, with the others fixed, and the output of the function does not change. This is typical for neural networks with multiple normalized layers: each of these layers (and even each column/filte in a layer) is scale-invariant. Naturally, if a function is scale-invariant w.r.t. several parameter groups, it is also scale-invariant w.r.t. their union, therefore the whole vector O is scale-invariant too.
Since inveriantt functions are effectively defined on the sphere, we try to minimize F(0) on the sphere of radius p:
min F(0). O
(p)
where (p) = 10 R². |10|| = p}. We do that using the projected gradient descent method with fixed learning rate y7:
(0) t r(0) - VVVF((()
o (0)
(2)
P
Of course, since the function is scale-invariant w.r.t. each individual group 0,, we could optimize it on the product of independent spheres S instead of the single mutual sphere, we do in eq. (T). However, that would either require introducing too many hyperparameters in the model (n different learning rates for n spheres) or overconstrained the problem (one learning rate for all spheres). We also argue that constraining the total parameters norm is much more natural and closer to the standard neural networks training with weight decay (see discussion in Appendix(A.]]
We now define the notions of effective gradient and effective learning rate for each SI group O,, Effective gradient is the gradient w.r.t. 0; measured at the same point but with the group 0; projected
on its unit sphere, i.c., point 0 (01.... Oi/Pi..... ,nn) where pa |0:1|. Due to scale
3

---

invariance w.r.t. 0, the function's 's value is equal at both points and & but, as discussed earlier, the norm of the effective gradient, which we denote as 9t, is pt times the norm of the regular gradient w.r.t. 0:: Di DiPr. where Di Vo, F()) Effective learning rate 1j is the learning rate required to make gradient step w.r.t. O, equivalent to the original but started from the point 0. It is known from the previous literature on scale- invariant functions [29 35 27
that ELR for the group 0 equals ?i n/ p. We also define the total effective gradient norm 9 9P, where g Ve F ( [|, and the
total ELR rj = n/p". which is fixed in the considered setup.
From the constraint on the total norm Σi=1 p = p² we obtain the following fundamental equation relating the ELRs of individual SI groups to the total ELR value
Σ
Σ1-1
n
Th
Another important notion is the effective step size (ESS), which is the product of the effective gradient length by the effective learning rate: Íāg;- This value defines how far we effectively move the scale-invariant parameters 0g after one gradient step. It can be shown that the total squared ESS can be expressed as a convex combination of squared ESS values of individual SI groups:
(ng) 2
Σ wi (Trii)
Σ wi 1, wi
f
The process 2 and eq. 3 yield the following update rule for each group's 's individual ELR:
(1) 1 (1)
(1+1) fi
1 + (1) (2)
9; 1/
from which it follows that for a given SI group the higher (lower) the ESS at the current iteration, the lower (higher) the ELR becomes after it. Since, by definition, ESS and ELR values are highly correlated, we conclude that at each iteration the largest ELRs tend to become smaller and vice versa. This "negative feedback" principle becomes very important in distinguishing between the three regimes of optimization.
3.2 Explaining the regimes
To provide clear and easy-to-follow explanation of the differences between the three regimes of training on the sphere, we construct a simple yet illustrative example of function with several groups of SI parameters and analyze its optimization properties depending on the total ELR value.
The simplest working example of a ocale-invariant function is the following one:
x²
f(z, v)
22 + 32 x2 y E R \ {0).
(6)
This function has also been used in previous works to visually demonstrate various properties of general scale-invariant functions (26] 27). Based on it, we develop an elucidative example of function that contains more than one group of scale-invariant parameters - just as real multilayer
normalized neural networks do. For that, we take a conical combination of n > functions each depending on its own pair of variables:
F(x, ( Σ au Caf(Et
"1
ai Σ
11
x
(7)
x?
?
where x (z1...., y = (V1..... Vn), and each 04 > 0 fixed positive coefficient. In this function, there are groups of SI parameters: v.), 1...., n. In accordance with the setup described previously, we intend to optimize this funetion on the unit sphere using the
projected gradient descent method with a fixed (effective) learning rate, i.e., we plug O = (ar, y) ((11, y1), (In, ),. P = 2n, p = into equations and 2
Now we would like to study the behavior of optimization of F(xx, y) on the unit sphere depending on the learning rate value 77. which equals the total ELR 7j due p 1. But first we need dwell on the convergence properties of each of its subfunctions ag (I;, y.). The next proposition, which we rigorously formulate and prove in Appendix A.3 answers this question.
"See the derivation of this and the following equations of this section in Appendix A.2

---

Proposition 1 For the SI function fo (1, v) =
o² 2z minimized with ELR ij:
1. if < 2, it linearly converges to zero, i.e., its global minimum;
2. if 4. it stabilizes af value (a (a-
Knowing that each of the subfunctions of F(z, y) can converge to the minimum only when its ELR stays below 1/a, and considering the fundamental constraint on ELRs (3), we state that the first regime (convergence) is only possible when the total ELR is below a certain threshold:
(8)
1i=1 OG
What if the total ELR exceeds that threshold? The first regime cannot be observed: convergence becomes impossible, since at least one of the individual ELRs exceeds its convergence threshold value, according to eq. (3). By Proposition [ the value of the corresponding subfunction is separated from zero by some quantity depending on its ELR. This means that the total function value, and therefore the total effective gradient, also nonvanishing (we give more formal argument in Appendix A.4). This brings us to the "undamped" dynamics of ELRs (5), where the effective step sizes do not tend to zero. In that case, the dynamics stabilize only when all the ESS values become equal, which is possible if and only if each individual ELR n takes its corresponding equilibrium value:
17 7ー 93
Due to the negative feedback principle of eq. 5 the system will naturally tend towards this state when ESS values are non-negligible. Note that, in contrast, in the first regime, ESS values decay too rapidly due to convergence, and the model may fail to reach equilibrium. In sum, when the total ELR exceeds the convergence threshold and each individual ELR (in average) equals its equilibrium value corre- sponding to the equalized ESS, the optimization process enters the second regime, i.e., the chaotic equi- librium. Finally, if the total ELR is set too high, the chaos will dominate equilibration and the optimiza- tion dynamics will resemble a random walk. In that case, we observe the third regime, or divergence.
In Figure 2 we plot the evolution of the function 7 during opti- mization on the unit sphere with three different effective learning rates. We set n 3, Og 2', = 0, 1, 2, and start from a point randomly selected on the 6-dimensional unit sphere. In this case, the threshold condition (8) becomes 7j < 1/7 ss 0.143, and thus we select three ELRs corresponding to the three optimization regimes, respectively: 0.1, 0.2, and 0.5. The lines on the plot directly demon- strate the anticipated behavior: the smallest ELR leads to a fast decay of the function's ' value, the medium one makes the function stabilize around certain level, and the largest ELR depicts the most chaotic behavior. In Appendix A.5 we provide plots of the individual ELRs dynamics that show that the chaotic equilibrium iterations closely match the predicted values (9) while in the first regime they are not reached due to quick convergence, and in the third the optimization is too unstable to distinguish any equilibration.
Fix!" (
ELR
200
100
200
1000
Figure 2: Function values evo- lution during optimization of the function 7 using three dif- ferent ELRs corresponding to the three optimization regimes, respectively.
in average by
4 Experimental setup
After discussing the theoretical substantiation of the different regimes of learning on the sphere in the case of a simple SI function, let us move on to an empirical analysis of the learning dynamics of neural networks equipped with Batch Normalization. We conduct experiments with a simple 3-layer BN convolutional neural network (ConvNet) and ResNet-18 on CIFAR-10 [18] and CIFAR- 100 199 datasets. Both networks are in the implementation of Lobacheva et al. [27] and we train them by optimizing cross-entropy loss using stochastic gradient descent with batch size 128 for 1000 epochs. In Appendix(0) [O] we also show that three training regimes may be observed for other neural architectures, such multilayer perceptron and Transformer (34]
5

---

In Section []] the sphere with a fixed ELR. To obtain fully scale-invariant versions of the networks we follow the same approach as in Lobacheva et al. (27]. We fix the non-scale-invarian weights, i.e., we use zero shift and unit scale BN layers and freeze the weights of the last layer at random initialization with increased norm equal to 10. To train a network with ELR nj, we fix the norm of the weights at the initial value p and use the projected SGD with a fixed learning rate 7) = npp.. We consider a wide o. which more than covers the values of ELR usually
we analyze the training regimes of fully scale-in invariant neural networks, trained on
range of ELRs: {10-*, 2 10 5 10 10= encountered during training of regular networks. We also used a more fine grained ELR grid closer to the boundaries between the regimes. In Section we investigate how the observed training regimes transfer to more practical scenarios. We train networks in the whole parameter space with a fixed learning rate and weight decay in Lobacheva al. 277, Wan et al. [35]. We use weight decay of le-4/ / 5e-4 for ConvNet / ResNet-1 8 in all the experiments and a range of LRs,, specific for each scenario. In some experiments, also use momentum of 0.9, standard CIFAR data augmentation, and cosine LR schedule for 200 epochs. More details can be found in Appendix C
To make the experiments with different ELRs/LRs fully comparable, we train the networks from the same random initialization and with the same optimizer random seed. However, we show in Appendix E that the results are consistent if we vary both of them. For each visualization we choose the most representative subset of ELRs/LRs due to the difficulties of distinguishing between many noisy lines on the plots. At each training epoch, we log standard train / test statistics and the mean (over mini-batches) norm of the stochastic effective gradients for fully scale-invariant networks or the mean norm of the regular stochastic gradients in the case of standard networks. We choose this metric for the following two reasons. First, it can serve as a measure of sharpness of the minimum to which we converge in the first regime, since the mean stochastic gradient norm is strongly correlated with the trace of the gradient covariance matrix, which in turn is closely tied with the loss Hessian and the Fisher Information Matrix 13 37 32 (see Appendix BB. Second, it represents the average (effective) step size up to multiplication by (E)LR and thus is also related to optimization dynamics.
5 Three regimes of training on the sphere
In this section, we investigate the three training regimes of fully scale-invariant neural networks, trained on the sphere with fixed ELR. We first analyze the learning dynamics in each regime, including the properties of the resulting solutions and the loss landscape around them. After that, we discuss transitions between the regimes in more detail. We show the results for ConvNet on CIFAR-10, the results for other dataset-architecture pairs are consistent and presented in Appendix D
5.1 Regime one: convergence
Training with the smallest ELRs results in the first training regime, which we call the convergence regime. As shown in Figure D optimization in this regime experiences a typical convergence behavior: after number of epochs the model is able to reach regions with very low training loss and continues converging to the minimum. The speed of convergence depends on the ELR value- - the higher the ELR, the faster the optimization, and the lower training loss is achieved after a fixed number of epochs. Also, training with different ELRs results in solutions with different sharpness (mean norm of stochastic effective gradients) and gemeralization (test error): higher ELRs lead to less sharp solution with better generalization. Similar results are well known in the RI 12 R. i.e., training neural networks using larger learning rates typically results in flatter and better generalizing optima. Furthermore, we discovered that the optima achieved after training with different ELRs not only vary in sharpness and generalization but also reside in different basins, i.c., there is significant loss barrier on the linear interpolation between them (see Appendix for more details).
literature [13 36 23 30
15 21
6
3
The value of ELR affects not only the rate of convergence and the properties of the final solution; it may have an effect on the whole training trajectory as well. To analyze the training trajectories independently from the training speed, we look at the evolution of sharpness and generalization vs. training loss during the optimization. We show the corresponding plots for different ELR values of the first regime in Figure 3 left. For the lowest ELRs the trajectories coincide, but the training is too slow to converge to the low-loss region. For the remaining values, as we increase ELR, we obtain a lower trajectory, and in the low-loss region trajectories of different ELRs appear as parallel lines.

---

Training with fixed ELRs
Fine-tuning with decr./iner. ELR (20k/5k epochs)
ot www.gndden........
starting point
EUR
10-8
3--7
102
starting point
101
16.6
ELR 5o.4
206-
14.4
1e-5
162
Recturing
Starting poin.
Starlnng pan
20
10-4
10"h 12"
30a1
101
incr.
Irath
trath loss
tran 100s
Sacd
Figure 3: First training regime, ConvNet on CIFAR-10. Left: training with fixed ELRs converge to the regions with different sharpness and generalization profiles. Right: fine-tuming with decreased ELR stays on the same trajectory, while fine-tuning with increased ELR jumps out and converges to a flatter region. As baselines, we show training with the decreased/increased ELR from the initialization.
Such differences in trajectories show that training with higher ELR not only results in a flatter final solution but also traverses less sharp regions the whole time: for any fixed value of the training loss, the point corresponding to a higher ELR has lower sharpness and better generalization. This is also consistent with recent results on the impact of higher learning rates on better conditioning of the optimization trajectory [5 12 g).
The parallelism of the trajectories in the low-loss region is particularly interesting and leads us to the following question: do the basins to which training with different ELRs converges have different overall sharpness and generalization profiles? That is, can we hypothesize that all solutions reachable with SGD in given basin have similar sharpness and generalization at the same training loss? To answer that, we take a solution achieved with specific ELR and fine-tune it with lower and higher ELR values (see Figure 3 right). Fine-tuning with lower ELR stays in the same basin and continues to move along the same trajectory as the pre-trained model, hence no sharper sub-minima can be found inside the flat basin. Fine-tuning with higher ELR also maintains the same trajectory first but jumps out of the low-loss region after a while and converges to new basin with a profile attributed to that higher ELR. That means that the initial basin is too sharp for the higher ELR to converge. The results are consistent for different ELR values, see Appendix [H; we also provide results on linear connectivity of pre-trained and fine-tuned solutions in Appendix !!
As can be seen from Figure 3 left, the range of sharpness and generalization profiles reachable with the fixed ELR training in the first regime is limited. For small ELRs, the trajectories coincide and optimization converges to the sharpest available basin. Increasing ELR results in flatter optima, but at some value the optimization stops converging low-loss regions and demonstrates the behavior of the second training regime, which we describe next.
5.2 Regimes two and three: chaotic equilibrium and divergence
further increase of ELR in the first regime leads to a sharp switch to another regime, the chaotic equilibrium. Figure I shows that after certain threshold ELR, the optimization is no longer able to reach low-loss regions and gets stuck at some loss level. We observe that the effective gradients norm also stabilizes, and higher ELRs correspond to a higher loss but lower sharpness.
As we determined in Section 3.2 the second regime corresponds to the state when all individual effective step sizes become equal, which also leads to the stabilization of individual ELRs. In Figure 4 we depict the values (averaged over last 200 epochs) of individual ELRs and effective gradient norms w.r.t. each SI parameters group of each layer of the neural network trained with 7 10-3 We see that the effective step sizes (ELR times effective gradient norm) indeed concentrate around a certain equilibrium value denoted as a dashed black line on the plot. In Appendix J we provide results for other ELRs and additional plots demonstrating the stabilization of individual ELRs and effective gradient norms, which corroborates our definition of the second regime the chaotic equilibrium.
Constant ESS 18.5
5
layer 0
layer 1
layer
0.3
layer 3
20 30
40
00
meen eff good hor
Figure 4: Equilibration of in- dividual ESS values in the second regime. ConvNet on CIFAR-10.
In the second regime, the optimization localizes a certain region the loss landscape and "hops along its walls". For moderate ELRs, this region is locally convex in the sense that the loss on the linear path
7

---

Fine-tuning with second regime ELRs
Fine-tuning with first regime ELRs (2k epachs)
less
Base EUn 0.002
all gractent nom
sta ting point
starting po
Base 8
ELR
ED-
Sase sap finetune ELR 7e-4
0.000s
100
10-
0L005
0.001
20-
fineture EUR 1
0.000
Base exp ELR 7e Sase exp ELR 1e
3000
1200
14:0
30"
30nh
101
31'77
12"1
10
opoch
tratn loss
tram loss
Figure 5: Second training regime, ConvNet on C0..R-10. Left: fine-tuning with a different second regime ELR results in the training loss level corresponding to the new ELR value (such levels for all considered ELRs are depicted with dashed lines). Right: fine-tuning with first regime ELR from the low second regime ELR always results in flattest possible solutions, while from higher starting ELRs, solutions of various sharpness can be reached.
between different checkpoints from the same trajectory is convex and generally much lower than at its edges. However, for the highest ELRs, the dynamics become more chaotic and distinguished barriers may appear on the linear path. We provide the corresponding plots in Appendix K Each ELR of the second regime determines its own loss level - the height at which the optimization "hops". Changing the ELR value during training appropriately changes the training dynamics: increasing/decressing the ELR directly leads to the trajectory with a loss level corresponding to the new ELR value (see Figure left). This significantly distinguishes the second regime from the first, where fine-tuning with a different ELR does not affect the learning trajectory, unless the new ELR is too large, in which case the optimization suddenly jumps out of the basin.
Now, what if we fine-tune a model from the second regime with a smaller ELR corresponding to the first regime? As can be seen from Figure right, the resulting optima highly depend on the starting ELR (more results in Appendix D). Fine-tuning the models, pre-trained with low second regime ELRs, using any first regime ELR converges to basins with the the maximum ELR in the first regime. For larger second regime ELRs, fine-tuning results in a variety of trajectories depending second regime ELRs discovers regions containing only flat minima. Meanwhile, regions discovered with high ELRs contain optima of different sharpness. Moreover, fine-tuning from these regions profiles that was observed in the first regime.
profile attributed to
the chosen ELR of the first regime. That means that training with low
results in the same range of sharpness/generalvaatiee
For the highest considered ELR values, we observe the most unstable optimization behavior corre- sponding to the random guess accuracy (see Figure D. We call it the third, divergent training regime. We explicitly compare the third regime with random walk in Appendix M
5.3 Transitions between the regimes
In the end, we would like to discuss the connections between the three regimes and our global view of training on the sphere. In Figure 6] we plot the training loss for several ELR values that cor- respond to the three regimes and transitions between them. When the ELR is too high (purple line), the total ESS becomes so large that the model is unable to even detect any region in the parameters space with non-trivial quality, and we encounter the third regime. With a smaller ELR (light blue line), the model starts occasionally hitting the edge of the region with a (relatively) low loss but then quickly escapes it and returns back to the random guess behavior; this could be attributed the border regime between the second and the third regimes. Setting the ELR to an even smaller value (green line) results in the chaotic equilibrium, when the loss is stabilized near a certain value, which lower the lower the ELR. For ELR values between the highest ELRs of the first regime and the lowest ones of the second regime (yellow line), we can witness sudden change from equilibrium behavior to convergence. We conjecture that the major difference between the first and second optimization regimes and such a sharp transition between them, particular, is due to the presence of zones of increased sharpness in the optimization trajectories of neural networks [4] 15
Irain loss
ELR
7.5e-4
0.002
10
0.014
0.1
200
400
800
2000
epoch
Figure 6: Three regimes and transitions between them, ConvNet on CIFAR-10.
12 g In the
8

---

SI networks in the whole space
Training with fixed LRs
Training with cosine LR schedule
starting porr.
18
50.0
D.2
2.0
30.0
0.001
5000.0
1003-0
starting poRni
LA erro
0,0001 722 2%
0.5 11.9%
0.2/22.00
0.0005 18.5%
1e:05 /50.0%
0.000 17.3%
0.05 73, m7
0.03 13.9%
starting p01e
LR best
0.1. (4.8%
0.2 5.1%
0.002
0.5 8.2%%
3.0
0.02
0.025 6.0%
0.05 0%
Figure 7: Training regimes in standard training. Left: training of SI networks in the whole parameter space with weight decay, ConvNet on CIFAR-10. Middle and right: conventional training with fixed LRs and cosine I.R schedules, ResNet-18 on CIFAR-10. Results for other lataset-architecture pairs are presented in Appendix N
gradient norm vs. training loss diagram in Figure ! we can observe that the sharpness reaches its peak precisely at the transition point between the first two regimes; this leads us to hypothesis that only training with sufficiently small ELR values allows getting through this bottleneck and enter the convergence regime. Appendix G the epoch- double descent of the test error
we discuss the relationship between this transition and
[8]: we discover that the sharpness peak correlater with the test error peak and also becomes much more noticeable the presence of label noise in the training data. Finally, the smallest ELRs (orange line) correspond to typical convergence to the optimum, i.e., the first regime, and the lower the ELR the sharper the minimum we converge to.
Regimes in standard training
In this section, we study how different training regimes of SI neural networks on the sphere are represented in more practical scenarios and what clues they can give to achieve good minimum in terms of sharpness and generalization. First, we experiment with training SI networks in a whole parameter space, where both direction and norm dynamics influence the optimization trajectory. We train SI networks with weight decay and vary LR instead of ELR. All three regimes are also present in such training (see Figure 7 left). We argue that the difference between the first and the second regimes explains why disparate results on training dynamics are reported in the literature. Lobacheva et al. (27] have discovered that SI neural networks experience convergence with periodic jumps during optimization. We observe the same behavior for high LR values in the first regime, where the trajectories periodically jump out of the low-loss region (the yellow trajectory for LR = 2.0). On the other hand, experiments of Li et al. [26] and Wan et al. [35] with fixed LRs demonstrate that the equilibrium state is reached, which we can observe in the second training regime.
Now, let us consider conventional neural networks training, which implies optimization of all net- work's weights (including the
ones) in the entire parameter space with momentum and data augmentation. Here we use ResNet-15 18 network of standard size with weight decay and try both constant and the standard cosine LR schedule (see Figure 7 middle and right). In conventional training, only the first two training regimes are present because LRR also affects the non-scale-invaree parameters, and so training with very high LR values results in NaN values in the loss, since the weights of the last layer can become large enough to lead to numerical overflows. The transition between the first two regimes is not as easy to locate in the previous experiments. All the training trajectories end up in regions with relatively high training loss, mainly because of the data augmenta- tion, which makes the task much harder to learn and requires more time to converge to the low-loss region. Moreover, the task may become too hard for some models to achieve a low training loss at all. We show in Appendix [F] that similar results are observed in case of training SI networks that are too small for the task in hand.
In the experiments with constant LRs, training with lower LRs (< 0.01) passes the region with the highest sharpness along the trajectory and starts converging to a flatter region, while training with higher LRs (2 0.01) gets stuck in the high-sharpness zone. The former can be attributed to the first training regime, and the latter to the second one. Note that, in accordance with Section [ the best test accuracy is achieved with the largest LR of the first regime.
In the experiments with cosine LR schedule, we decrease L.R from some starting LR value to zero during 200 epochs of training, If we start with low LR (5 0.02), optimization is carried out all or

---

almost all the time with LRs of the first regime, and the first regime trajectory is observed. Moreover, for higher starting LRs, the solution with better sharpness profile and generalization is achieved. If we start with a high LR (2 0.5), then most of the training is done with LRs of the second regime, and optimization does not have enough time with low L.R values to converge to the low-loss region. The medium starting LR values are of the most interest here (between 0.02 and 0.5). In this case, training first has substantial time with gradually decreasing LRs the second regime, which helps it to localize the region with the flattest optima, and then has just enough time with the first regime LRs to pass the high-sharpness zone and converge to the minimum. With such training, solutions with better sharpeningggeneraaizztooon /generalization profiles are achieved, which is consistent with the results of Section 5.2 and the existing literature on pre- pre-training with high LR values 23
Conclusion and Discussion
In this work, we investigated the properties of training scale-invariant neural networks on the sphere using projected SGD. This method allowed us to study the intrinsic domain of SI models more accurately. We discovered three regimes of such training depending on the effective learning rate value: convergence, chaotic equilibrium, and divergence. They uncover multiple properties of the intrinsic loss landscape of scale-invariant neural networks: some are related to diverse previous results in deep learning, and some appear as novel. We showed that these regimes are present in conventional neural network training as well and can provide intuition on how to achieve better optima.
Possible future directions Speaking about the potential prospects for further research, three directions can be distinguished. First, it would be interesting to develop more solid and general theoretical groundings for the observed phenomena. For instance, a closer look at the negative feedback principle in eq. (5) could potentially shed more light on the ESS/ELR dynamics in the first and second regimes compared to the current decay vs. alignment dichotomy and maybe better explain occasional transitions from the second to the first regime for certain ELRs in Section [5.3 Second, the analysis could be extended to other tasks, architectures, and especially optimizers like Adam [16), which can be challenging due to their indirect impact on the effective direction and effective learning rate in SI models (29]. Third, it is very tempting to investigate other potential practical implications of the three regimes for conventional training. The analysis of various LR schedules from the perspective the discovered training regimes is intriguing and might help in designing more efficient and more explainable LR schedules. For example, from our experiments we observe that decreasing (E)LR in the first regime during training does not influence the final solution but only slows down the convergence; at the same time, for the optimal final performance, optimization should run through the low second regime (E)LRs, which allows to achieve basins containing solutions with the best sharpnes//enerraiiatton properties.
Limitations and societal impact The main limitations of our work are applying a single optimiza- tion method on the sphere (projected SGD), experimenting only with BN networks SI models, and considering only one of the symmetries that may conceal the actual properties of neural networks intrinsic domain [20). To our knowledge, our work does not have any negative societal impact.
Acknowledgments and Disclosure of Funding
We would like to thank Mikhail Burtsev for the valuable discussions. The publication was sup- ported by the grant for research centers in the field of Al provided by the Analytical Center for the Government of the Russian Federation (ACRF) in accordance with the agreement on the provision of subsidies (identifier of the agreement 000000D73032 (P550002)) and the agreement with HSE University N070-2021-001:
The empirical results were supported part through the computational resources of HPC facilities at HSE University [17. Additional revenues of the authors for the last three years: laboratory sponsorship by Samsung Research, Samsung Electronics and Huawei Technologies; honorarium for teaching by MSU; internship at Yandex.
References
[1] Arora, S., Li, 7., and Lyu, K. (2019). Theoretical analysis of auto rate-tuning by batch normaliza- tion. In International Conference Learning Representations.
10

---

[2] Ba. J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. 17Xiv1607.006400
arXiv preprint
[3] Barrett, D. and Dherin, B. (2021). Implicit gradient regularization. In International Conference on Learning Representations.
[4] Chiley, V., Sharapov. I., Kosson, A., Koster, U., Reece, R., Samaniego de la Fuente, S., Subbiah, V., and James, M. (2019). Online normalization for training neural networks. Advances in Neural Information Processing Systems, 32.
[5] Cho, M. and Lee, J. (2017), Riemannian approach to batch normalization. Advances in Neural Information Processing Systems, 30.
[6] Cohen, J., Kaur, S., Li, Y., Kolter, J. Z., and Talwalkar, A. (2021). Gradient descent on neural networks typically occurs at the edge of stability. International Conference on Learning Representations.
[7] Deng, L. (2012). The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-144
[8] Gilmer, J., Ghorbani, B., Garg, A., Kudugunta, S., Neyshabur, B., Cardoze, D., Dahl, G. E., Nado, Z. and Firat, O. (2022). A loss curvature perspective on training instabilities of deep learning models. In International Conference on Learning Representations.
[9] Hoffer, E., Banner, R., Golan, I., and Soudry, D. (2018). Norm matters: efficient and accurate normalization schemes in deep networks. Advances in Neural Information Processing Systems, 31.
[10] loffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 446. PMLR.
[11] Iyer, N., Thejas, V., Kwatra, N., Ramjee, R., and Sivathanu, M. (2020). Wide-minima density hypothesis and the explore-exploit learning rate schedule. arXiv preprint arXiv:2003.03397
[12] Jastrzebski, S., Arpit, D., Astrand, O., Kerg, G. B., Wang, H., Xiong, C., Socher, R., Cho, K., and Geras, K. J. (2021). Catastrophic fisher explosion: Early phase fisher matrix impacts generalization. In International Conference on Machine Learning, pages 4772-4784 PMLR.
[13] Jastrzebski, S., Kenton, Z., Arpit, D.,, Ballas, N., Fischer, A., Bengio, Y., and Storkey, A. (2017). Three factors influencing minima in sgd, arXiv preprint 1rXiv:1711.04622)
[14] Jastrzebski, S., Kenton, Z., Ballas, N., Fischer, A., Bengio, Y., and Storkey, A. (2019). On the relation between the sharpest directions of DNN loss and the SGD step length. In International Conference on Learning Representations.
[15] Jastrzebski, S., Szymczak, M., Fort, S., Arpit, D., Tabor, J., Cho, K., and Geras, K. (2020). The break-even point on optimization trajectories of deep neural networks. In International Conference on Learning Representations.
[16] Kingma, D. P. and Ba, (2015). Adam: A method for stochastic optimization. International Conference on Learning Representations.
[17] Kostenetskiy, S., Chulkevich, A., and Kozyrev, I. (2021). HPC resources of the higher school of economics. Journal of Physics: Conference Series, 1740:012050.
[18] Krizhevsky, A., Nair, V., and Hinton, G. CIFAR-10 (canadian institute for advanced research).
[19] Krizhevsky, A., Nair, V., and Hinton, G. CIFAR-100 (canadian institute for advanced research).
[20] Kunin, D., Sagastuy- Brena, J., Ganguli, S., Yamins, D. L.,, and Tanaka, H. (2021). Neural mechanics: Symmetry and broken conservation laws deep learning dynamics. International Conference on Learning Representations.

---

[21] Lewkowycz, A., Bahri, Y., Dyer, E., Sohl- Sabl-Dicksttin, J., and Gur-Ari, G. (2020). The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02218.
[22] Li, X., Chen, S., and Yang, J. (2020a). Understanding the disharmony between weight normal- ization family and weight decay. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4725-4722
[23] Li, Y., Wei, C., and Ma, T. (2019). Towards explaining the regularization effect of initial large learning rate in training neural networks. Advances in Neural Information Processing Systems, 32.
[24] Li, Z. and Arora, S. (2020). An exponential learning rate schedule for deep learning. In International Conference on Learning Representations.
[25] Li, Z.., Bhojanapalli, S., Zaheer, M., Reddi, S., and Kumar, S. (2022). Robust training of neural networks using scale invariant architectures. In International Conference on Machine Learning, pages 12656- 12684. PMLR.
[26] Li, Z., Lyu, K., and Arora, S. (2020b). Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate. Advances in Neural Information Processing Systems, 33.
[27] Lobacheva, E., Kodryan, M., Chirkova, N., Malinin, A., and Vetrov, D. P. (2021). On the periodic behavior of neural network training with batch normalization and weight decay. In Ranzato, M., Beygelzimer, A., Dauphin, Y.. Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages 21545-21550 Curran Associates, Inc.
[28] Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. (2020). Deep double descent: Where bigger models and more data hurt. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
[29] Roburin, S., de Mont-Marin, Y., Bursuc, A., Marlet, R., Pérez, P., and Aubry, M. (2020). A spherical analysis of adam with batch normalization. arXiv preprint arXiv: 13382.
[30] Senderovich, A., Bulatova, E., Obukhov, A., and Rakhuba, M. (2022). Towards practical control of singular values of convolutional layers. Advances Neural Information Processing Systems, 35.
[31] Smith, S. L., Dherin, B., Barrett, D., and De, S. (2021). On the origin of implicit regularization in stochastic gradient descent. In International Conference on Learning Representations.
[32] Thomas, V., Pedregosa, F., Merriënboer, B., Manzagol, P.-A., Bengio, Y., and Le Roux, N. (2020). On the interplay between noise and curvature and its effect on optimization and generalization. In International Conference on Artificial Intelligence and Statistics, pages 3503- 3513. PMLR.
[33] Van Laarhoven, T. (2017). L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05355
[34] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, L. u. and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems.
[35] Wan, Zhu, Z., Zhang, X., and Sun, J. (2021). Spherical motion dynamics: Learning dynamics of normalized neural network using sgd and weight decay. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages 6380-6391. Curran Associates, Inc.
[36] Wu, L., Ma, C., et al. (2018). How sgd selects the global minima in over pver-parameterized learning:
A dynamical stability perspective. Advances in Neural Information Processing Systems, 31.
[37] Xing, C., Arpit, D., Tsirigotis, C., and Bengio, Y. (2018). A walk with sgd. arXiv preprint arXiv: Tow:1800 1802.08770.
[38] Zhang, G., Wang, Xu, B., and Grosse, R. (2019). Three mechanisms of weight decay regularization. In International Conference Learning Representations.
12

---

[39] Zhang, X., Zhao, J., and LeCun, Y. (2015), Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the end of Section 7
(c) Did you diseuss any potential negative societal impacts of your work? [Yes] See the end of Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] URL the end of Introduction.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4 and Appendix C
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We show that the described training regimes are consistent for different random seeds in Appendix
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix C
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]] See Section
(b) Did you mention the license of the assets? [Yes]] See Appendix C CIFAR datasets
are distributed under MIT license and code is under Apache-2.0 License license (we provide the link the GitHub in Section
(c) Did you include any new assets either We provide our code, URL at the end of Introduction.
supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [No] We train networks on the CIFAR datasets which are widely used in NeurlPS papers.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] We train networks on the CIFAR datasets which are widely used in NeurlPS papers.
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "sCDIvW4Q_m", "reviewer": "Reviewer_AvYP", "summary": "This paper studies properties of training scale invariant nets using fixed learning rates. In this setting, it further characterized 3 distinct regimes where qualitative behavior NN training and generalization is different. These properties hold both in an illustrative toy problem and practical neural nets.", "questions": "I like the paper overall, I'm curious if the authors can suggest future avenues of direction if and how these insights can be used to improve either training or generalization performance of practical (maybe scale invariant) neural nets.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Solid empirical results and conclusions follow directly from the evidence\n- The paper also does a good job of setting up the background and situating itself with related work.\n- Toy problem is not novel but is indeed illustrative\n\nWeaknesses\n- Figures are too small, I would encourage the authors to make them bigger for improved reading experience ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dtuNdCx-4h", "reviewer": "Reviewer_UmdY", "summary": "This paper studies the training behavior of scaling invariant neural networks using projected SGD with fixed effective learning rate on the sphere. The topic is meaningful because neural network models become (semi) scale invariant when normalization techniques are applied. Theoretically, the authors point out several relations satisfied by the effective learning rates and effective step sizes of individual parameter groups and the whole parameter vector. With the relations, the dynamics of individual parameter groups' effective learning rates are derive, showing a \"negative feedback\" principal, which is later used to explain the \"chaotic equilibrium\" regime of training. Also theoretically, a simple scale invariant function is analyzed to study the different optimization behaviors shown by different effective learning rate. \n\nEmpirically, experiments are conducted on both scale invariant neural networks and regular neural networks. Three regimes are identified for the behavior of the training loss: convergence, chaotic equilibrium, and divergence. The three regimes happen in order when the effective learning rate is increased. Extensive experiments are conducted to observe and compare the test error and sharpness of solutions for the three regimes. Discussions are made trying to explain the three regimes using the theoretical results. ", "questions": "1. It there any difference in the training behavior of scale invariant networks and regular networks? If such differences exist, are they able to be explained by the theoretical insights obtained in this paper?\n\n2. For the toy function (7), it seems there no divergence regime by proposition 1. Theoretically understanding, what happens when the behavior moves from regime 2 to regime 3?\n\n3. The first line of equation (1) is confusing. ", "limitations": "Limitations are discussed in the last section. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The theoretical results obtained in Section 3 is new. Also, the extensive experiments on scaling invariant neural networks with fixed effective learning rate are new. Though, the reviewer does not think the observations dramatically change our understanding on neural networks' training process. It is a common belief that, even for non-scale-invariant networks, the loss experiences smooth descending, oscillation, and divergence when the learning rate is increased. The experiments and resulting observations on the sharpness of solutions, and jump between basins, also exist in previous works. The results for scale invariant networks are not different. Also, the theoretical results are not sufficient and convincing enough to explain the empirical behaviors. \n\nQuality and Clarity: The paper is well written and easy to follow. The descriptions and discussion on numerical experiments are clear.\n\nSignificance: The work has fair significance on our understanding to neural networks with normalizations. It shows some widely known phenomena in training also hold for scale invariant models trained with fixed effective learning rate. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YbW-RqefNb2", "reviewer": "Reviewer_nwDp", "summary": "The paper investigates the directional optimization dynamics of training scale invariant neural networks. By fixing the effective learning rate (ELR), the paper seeks to decouple the influence of norm dynamics on the gradient from directional dynamics. The authors observe that training can behave according to three distinct regimes depending on the chosen ELR value. The various regimes are carefully analyzed both theoretically and empirically through the use of toy examples as well as more realistic settings. The authors utilize their framing to offer explanations to various conflicting observations in previous works, as well as share suggestions on how to use their insights for training more conventional neural networks.", "questions": "**Questions:**\n- Section 3.1, starting line 102: what is the difference of the method described currently versus the statement ‘one learning rate for all spheres is overconstraining’? This difference still appears unclear after reading Appendix B\n- Section 3.2, starting line 152: the analysis in this paragraph at some point switches from features of regime 1 to regime 2, yet it’s not quite clear exactly where this happens?\n- Section 3.2, line 158/159: ‘is also non-vanishing (we give a more formal argument in the appendix)’ → how should one interpret the relevance of ‘non-vanishing’? Perhaps point to specific appendix?\n- Section 3.2, starting line 171: for the toy example, it is clearly demonstrated how constant ELR are chosen to end up in one of the three regimes. Would the authors have a suggestion how this type of reasoning of picking ELR values could be extended to the more realistic settings beyond trial and error?\n- Section 4 (related to the previous question): could the authors elaborate as to how the learning rates are chosen for the conducted experiments?\n- Section 5.1: in the discussion surrounding picking different ELR values in the first regime, it is not entirely clear how the following concepts all move together: speed, loss, sharpness, robustness. Would it perhaps be possible to include a small table with this? E.g. rows: (speed, loss, sharpness, robustness), columns: 1st regime low ELR, 1st regime high ELR\n- Section 6, starting line 336: could the authors elaborate why NaN values would eliminate the third regime in this setting compared to the previous settings?\n\n**Suggestions:**\n- [Throughout paper] sphere → hypersphere or n-sphere\n- Section 3.1, equation 1: perhaps change R^P to S^(P-1) to indicate hyperspherical space?\n- Section 3.2, line 122: ‘it can be shown’ → maybe point to relevant appendix or related work?\n\n**Possible typos**:\n- Section 3, line 87: ‘alike normalized neural networks’\n- Section 3.2, line 125: ‘from what follows’ → ‘from which it follows’\n- Section 3.2, line 156: ‘of individual’ → of the individual\n", "limitations": "The authors adequately addressed limitations of their work from a technical point of view. However, as pointed out in the '*weaknesses section*': (1) more practical advice around picking 'good' ELR values/ranges (or when this is possible/impossible), and (2) clearly stating hopes/expectations as to how/where this work could be improved upon would likely strengthen this work. Negative societal impact does not seem relevant.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- [**quality**] the main statements presented in the paper are generally well supported by clear, theoretical foundations accompanied by well explained proofs in the appendix. \n- [**clarity**] the paper is clearly written and overall easy to follow. It also provides an extensive appendix containing additional discussion as well as further edge case experimental results to aid replication and completeness.\n- [**significance**] the authors carefully analyzed a very specific setting that does not seem to have been explored in isolation before. However, because it touches on quite broad learning settings for neural network training - the (partial) use of scale-invariant modules - it could provide guidance and save time for other researchers meaning to conduct research in this topic, or practitioners analyzing observed learning behavior of their models.\n\n\n**Weaknesses**\n- [**clarity**] the provided plots are not always easy to interpret, as they are quite small with lines strongly overlapping. A possible suggestion to increase the legibility would be to enlarge the most important plots and move others to the appendix.\n- [**significance**] while the paper does a great job at completing an isolated analysis seemingly absent from the relevant literature, it is not entirely clear to this reviewer how the provided results, i.e. the existence of three optimization regimes depending on the effective learning rate for (partially) scale-invariant NN, should be utilized in practice. Generally, the paper could be enhanced with more direct take-aways or suggestions as to how the results presented in this work should be used or build upon.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1gBUD5axyrw", "reviewer": "Reviewer_FkbE", "summary": "The authors perform a rigorous study of training regimes for neural networks with normalized weights, which is a common practice. There is a mathematical analysis, using an example problem, and empirical study on standard image classification benchmarks. The paper culminates with a comparison of training regimes on a Resnet18 on cifar10 trained with and without scale invariance. ", "questions": "# Section 6 questions\n\nThe experiment setup in section 6 needs some clarification. What exactly is teh difference between the three cases in Figure 7?\nWhat exactly does it mean to do scale invariance “in the whole parameter space”? How is this different from the models in the previous sections?\nTo clarify, is spherical projection applied during training for these models? If so, for which?\nWhy did you vary LR instead of ELR?\nHow are batchnorms applied in all three cases? Does the ResNet 18 contain batch normalization layers in the conventional training runs?\nFigure 7: Why is the test error missing on the rightmost SI figure? Did these networks perform worse? \nWhich of the methods in Figure 7 is “the best” in terms of accuracy, training performance, robustness, etc.?\nLine 340: How does data augmentation make the problem “too hard”? Shouldn’t it make the problem easier?\n\n# Clarification points\n\nThe first line of the paper calls out batch normalization and layer normalization but these methods normalize hidden states, not the weights themselves. Spherical projection normalization is different. To which normalization techniques exactly does the analysis apply? The first paragraph would be better to clarify this.\n\nThe text needs better descriptions of architectures. What architecture is “ConvNet” exactly? What are the activations, layer width, etc.\n\n# Discussion questions:\n\nGiven these results, what observation can I make in practice to inform decisions on model training? Do the results suggest a way of determining scale invariant weights will improve my model?\n\nDoes your work show any advantages to using scale invariant weights? Are there any roadblocks that your analysis removes from using scale invariant weights in practice?\n\n# Minor\n\nFigure 7: I first read “Conv. training” and “convolutional” instead of “conventional”; it’s a confusing overload of an abbreviation.\n\nExtra space in line 3 of Figure 7 caption , .\n", "limitations": "The text addresses the limitations of the work. As an empirical understanding paper, there are no major limitations I see, and it is a contribution as-is.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The analysis is thorough, and paying attention to normalization methods makes this paper a solid contribution. Practitioners are more familiar with “normal” training techniques, but in the real world, spherical projections are common, but not as well understood. The authors make a good case that the training behavior changes significantly, and the presentation which will help readers build intuition to interpret these in practice. Section 6 which compare-and-contrasts is exceptional at achieving that end.\n\nOne weakness is that the paper only focused on CIFAR-10/0 with convolutional nets. These problems are over-optimized that batch norm alone works. It would be interesting to see if the analysis applied to other domains and architectures, such as MLPs or transformers, where spherical projection is more common. That would maximize the paper’s impact, but it is substantial as-is.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept