The Break-Even Point on Optimization Trajectories of Deep Neural Networks
Authors: Stanislaw Jastrzebski
Maciej Szymczak
Stanislav Fort
Devansh Arpit
Jacek Tabor
Kyunghyun Cho*
Krzysztof Geras*
Venue:  (2020)
Keywords: generalization
sgd
learning rate
batch size
hessian
curvature
trajectory
optimization

Abstract: The early phase of training of deep neural networks is critical for their final performance. In this work, we study how the hyperparameters of stochastic gradient descent (SGD) used in the early phase of training affect the rest of the optimization trajectory. We argue for the existence of the "``break-even" point on this trajectory, beyond which the curvature of the loss surface and noise in the gradient are implicitly regularized by SGD. In particular, we demonstrate on multiple classification tasks that using a large learning rate in the initial phase of training reduces the variance of the gradient, and improves the conditioning of the covariance of gradients. These effects are beneficial from the optimization perspective and become visible after the break-even point. Complementing prior work, we also show that using a low learning rate results in bad conditioning of the loss surface even for a neural network with batch normalization layers. In short, our work shows that key properties of the loss surface are strongly influenced by SGD in the early phase of training. We argue that studying the impact of the identified effects on generalization is a promising future direction.

Published as a conference paper at ICLR 2020
THE BREAK-EVEN POINT ON OPTIMIZATION TRAJEC- TORIES OF DEEP NEURAL NETWORKS
Stanislaw Jastrzebski' .Maciej Szymezak?, Stanislav Fort², Devansh A Jacek Tabor?, Kyunghyun 1,,6, Krzysztof Geras
1 NNww York University, USA
2 "Jagiellonian University, Poland
3 Staanford University, USA
1 "Salesforce Research, USA
5 Facebook AI Research, USA
6 CCIARR Azrieli Global Scholar
ABSTRACT
The early phase of training of deep neural networks is critical for their final per- formance. this work, we study how the hyperparameters stochastic gradient descent (SGD) used in the early phase of training affect the rest of the optimiza- tion trajectory. We argue for the existence of the "break-even" point on this trajec- tory, beyond which the curvature of the loss surface and noise in the gradient are implicitly regularized by SGD. In particular, we demonstrate on multiple classifi- cation tasks that using a large learning rate in the initial phase of training reduces the variance of the gradient, and improves the conditioning of the covariance of gradients. These effects are beneficial from the optimization perspective and be- come visible after the break-even point. Complementing prior work, we also show that using a low learning rate results in bad conditioning of the loss surface even for neural network with batch normalization layers. In short, our work shows that key properties of the loss surface are strongly influenced by SGD the early phase of training. We argue that studying the impact of the identified effects generalization is a promising future direction.
INTRODUCTION
The connection between optimization and generalization of deep neural networks (DNNs) is not fully understood. For instance, using large initial learning rate often improves generalization. which can come at the expense of the initial training loss reduction (Goodfellow et al. 2016 Li al. 2019 Jiang al. 2020). In contrast, using batch normalization layers typically improves both generalization and convergence speed of deep neural networks (Luo et al. 2019 Bjorck et al., 2018 These simple examples illustrate limitations of our understanding of DNNs.
Understanding the early phase of training has recently emerged as promising avenue for study- ing the link between optimization and generalization of DNNs. It has been observed that applying regularization in the early phase of training is necessary to arrive at well generalizing final solu- 2017|Achillc
tion (Keskar et al. 20177; Sagun et al. 2017 Achille et al.) 2017). Another observed phenomenon is that the local shape of the loss surface changes rapidly in the beginning of training (LeCun al.) 2012 Keskar et al. 2017 Achille et al. 2017] Jastrzebsk et al., 2018; Fort & Ganguli 2219). The- oretical approaches understanding deep networks also increasingly focus the early part of the optimization trajectory (Li et al.) 2019 Arora et al.] (019)).
In this work, study the dependence of the entire optimization trajectory the early phase of training. We investigate noise in the mini-batch gradients using the covariance of gradients [ and the local curvature of the loss surface using the Hessian. These two matrices capture important and
Equual contribution. WW define it as K N 2i- (g. T) g), where gi = 9(x,, Ve: 0) is the gradient of the training loss C with respect 6 on 34, N the number of training examples, and g the full-batch gradient.
DN

---

Published as a conference paper at ICLR 2020
complementary aspects of optimization (Roux al. 2008 Ghorbani et al. 2019) and generalization
performance of DNNs (Jiang et al. 2020;] Keskar et al. 2017 Bjorek et al. 2018 Fort et al. 2019). We include more detailed discussion in Sec. 2
Keskar
6
10 12 14
1.0
20
30
40
50
60
70
A (spectral al of
Traiiinn accuracy 196)
Figure 1: Visualization of the early part of the training trajectories on CIFAR-10 (before reaching 65% training accuracy) of simple CNN model optimized using SGD with learning rates = 0.01 (red) and 1 0.001 (blue). Each model on the training trajectory, shown by its test predictions embedded into two-dimensional space using UMAP. The background color indicates the spectral norm of the covariance of gradients K (A left) and the training accuracy (right). For lower 1), after reaching what we call the break-even point, the trajectory is steered towards region characterized larger X (left) for the same training accuracy (right). See Sec. for details. We also include an analogous figure for other quantities that we study in App. A
point, is represented
Our first contribution is simplified model of the early part of the training trajectory of DNNs. Based on prior empirical work (Sagun et al. 2017). we assume that the local curvature of the loss surface (the spectral norm of the Hessian) increases decreases monotonically along the optimization tra- jectory. Under this model, gradient descent reaches a point in the early phase of training at which oscillates along the most curved direction of the loss surface. We call this point the break-even point and show empirical evidence of its existence in the training of actual DNNs.
Our main contribution dence of the entire optimization trajectory on the early phase of training. Specifically, we conjecture that the hyperparameters of stochastic gradient descent (SGD) used before reaching the break-even point control: (1) the spectral norms of K and H. and (2) the conditioning of K and H. In partic- ular, using larger learning rate prior to reaching the break-eve even point reduces the spectral norm of K along the optimization trajectory (see Fig. [ for an illustration of this phenomenon). Reducing the spectral norm of K decreases the variance of the mini-batch gradient, which has been linked to improved convergence speed (Johnson & Zhang 2013).
state and present empirical evidence for conjectures about the depen-
network with batch normalization (BN) layers and find that our predictions valid in this case well. Delving deeper in this line of investigation, using a large learning rate is necessary to reach better-conditioned (relatively to network without BN layers) regions of the loss surface, which previously attributed to BN alone (Bjorck et al. 2018 Ghorbani et al. 2019
Finally, we apply our analysis
show that
Page 2019).
2 RELATED WORK
Implicit regularization induced by the optimization method. The choice of the optimization method implicitly affects generalization performance of deep neural networks (Neyshabur 2017). In particular, using large initial learning rate is known improve generalization (Goodfellow al.
2

---

Published as a conference paper at ICLR 2020
2016 Li et al., 2019). A classical approach to study these questions is to bound the generalization error using measures such as the norm of the parameters at the final minimum (Bartlett et al. 2017] Jiang et al. 2020).
An emerging approach is to study the properties of the whole optimization trajectory. Arora et al. (2019) suggest it is necessary to study the optimization trajectory to understand optimization and generalization of deep networks. In a related work, Erhan et al.] (2010); Achille et al.] (2017) show the existence of critical period of learning. [Brhan al. (2010) argue that training, unless pre- training is used, is sensitive to shuffling of examples in the first epochs of training. Achille et al. (2017): Golatkar al.) (2019): Sagun et al.] (2017); Keskar
al. (2017) demonstrate that adding regularization in the beginning of training affects the final generalization disproportionatel more compared to doing so later. We continue research in this direction and study how the choice of hyperparameters in SGI in the early phase of training affects the optimization trajectory in terms of the covariance of gradients, and the Hessian.
The covariance of gradients and the Hessian. The Hessian quantifies the local curvature of the loss surface. Recent work has shown that the largest eigenvalues of H can grow quickly in the early phase of training (Keskar et al. 2017, Sagun et al. 2017;; Fort & Scherlis 2019 Jastrzebski cct al., 2018). Keskar et al. (2017); Dastrzebski et (2017) studied the dependence of the Hessian (at the final minimum) on the optimization hyperparameters. The Hessian be decomposed into two terms, where the dominant term (at least at the end of training) is the uncentered covariance of gradients G (Sagun et al. 2017 Papyan 2019).
The covariance of gradients, which we denote by K, encapsulates the geometry and the magnitude of variation in gradients across different samples. The matrix K was related the generalization error in Roux et al. (2008); Jiang et al.) (2020). Closely related quantities, such as the cosine alignment between gradients computed on different examples. were recently shown explain some aspects of deep networks generalization (Fort et al., 2019; Liu et al. [020 He & Su (020). ZZhang et al.) (2019) argues that DNNs the Hessian and the covariance of gradients are close in terms of the largest eigenvalues.
Learning dynamics of deep neural networks. Our theoretical model is motivated by recent work on learning dynamics of neural networks (Goodfellow et al.] 2014; Masters & Luschi 2018; Wu
ict al. 2018; Yao et al. 2018; Xing et al.) 2018, Jastrzebski et al. 2018 Lan et al. 2019). We directly inspired by Xing al. (018) who show that for popular classification benchmarks, the
cosine of the angle between consecutive optimization steps in SGD is negative. Similar observations can be found Lan et al. (2019). Our theoretical analysis is inspired Wu al. (2018) who study
how SGD selects the final minimum from a stability perspective. We apply their methodology to the
early phase of training, and make predictions about the entire training trajectory.
3 THE BREAK-EVEN POINT AND THE TWO CONJECTURES ABOUT SGD TRAJECTORY
Our overall motivation is to better understand the connection between optimization and generaliza- tion of DNNs. In this section we study how the covariance of gradients (K) and the Hessian (H) depend on the early phase of training. We are inspired by recent empirical observations showing their importance for optimization and generalization of DNNs (see Sec. 2 for detailed discussion).
Recent work has shown that in the early phase of training the gradient norm (Goodfellow ct al. 2016 Fort & Ganguli 2019. Liu et al. 2020) and the local curvature of the loss surface (Jastrzebski al. 20r8| & Ganguli 2019) can rapidly increase. Informally speaking, one scenario we study here when this initial growth is rapid enough destabilize training. Inspired by Wu et al. (2018), we formalize this intuition using concepts from dynamical stability. Based on the developed analysis, we state two conjectures about the dependence of K and H on hyperparameters of SGD, which we investigate empirically in Sec.
Definitions. We begin by introducing the notation. Let us denote the loss on an example (x, v) by L(x, y; 8), where 0 is a D-dimensional parameter vector. The two key objects we study are the
Hessian of the training loss (H), and the covariance of gradients K N Eialor - -9) (g: g).
3

---

Published as a conference paper at ICLR 2020
where 9: = g(4,, M/: 0) is the gradient of L with respect to 9 calculated i-th example, N is the number of training examples, and is the full-batch gradient. We denote the i-th normalized eigenvector and eigenvalue of matrix A by e's and XA Both H and K are computed at given 8, but we omit this dependence in the notation. Let index steps of optimization, and let O(t) denote the parameter vector at optimization step I.
Inspired by Wu et al. 2018)) we introduce the following condition to quantify stability at a given O(t). Let us denote the projection of parameters 0 onto ej by sh (0, e With a slight abuse of notation let g(b) = (g(0), e.)). We say SGD is unstable along e at ((t) if the norm of elements of sequence 1h(T + 1) (T)) - (g(() diverges when - oc, where (0)) @(t). The sequence
(T) represents optimization trajectory in which every step t' >> t is projected onto etz-
Assumptions. Based on recent empirical studies, we make the following assumptions.
1. The loss surface projected onto f/ is a quadratic one-dimensional function of the form f(v) E. (v) 3 )H, The same assumption was made in Wu et al. 2018). but for all directions in the weight space. |Alain et al. (2019) show empinically that the loss averaged over all training examples is well approximated by a quadratic function along ely-
2. The cigenvectors e/ and ek are co-linear, i.c. c/ _ and A/2 aA for some a E IR. This is inspired by the fact that the top eigenvalues of H can be well approximated using G (non-centered K) (Papyan 2019 SSgun et al. 2017). Zhang et al. (2019) shows empirical evidence for co-linearity of the largest eigenvalues of K and H.
3. If optimization is not stable along e at a given ((t), A/y decreases the next step, and the distance to the minimum along et increases in the next step. This is inspired by recent work showing training can escape a region with too large curvature compared to the learning rate (Zhu et al. 2018; Wu et al.] Dastrzebbk et al. 2018).
4. The spectral norm of H, App increases during training and the distance to the minimum along e/y decreases, unless increasing X/1 would lead to entering region where training is not stable along e/z- This is inspired by (Keskar et al. 2017 Goodfellow et al. 2016
Sagun et al.. 2017: Jasttzebski al. 22018: Fort & Scherlis] 2019 Fort & Ganguli 2019)
who show that in many settings Xir or gradient norm increases in the beginning of training. while at the same time the overall training loss decreases.
Finally, we also assume that S >> N, i.e. that the batch size is small compared to the number of training examples. These assumptions are only used to build theoretical model for the early phase of training. Its main purpose is to make predictions about the training procedure that we test empirically in Sec. 4
Reaching the break-even point earlier for larger learning rate smaller batch size. Let us restrict ourselves to the case when training is initialized at 8()) at which SGD is stable along e (0) We aim to show that the learning rate (n) and the batch size (S) determine H and K in our model, and conjecture that the same holds empirically for realistic neural networks.
Consider two optimization trajectories for 7h and 772, where Th >> 72, that are initialized at the same 8. where optimization is stable along ej (t) and >1x (t) > O. Under Assumption the loss surface along et (4) can be expressed as /()) = E = (v w/*) ) where H,(() E TR. It can be shown that at any iteration t the necessary and sufficient condition for SGD to be stable along c/ (t) is:
1N S) (1 (1)) 5(1) 5(N-1) - 1
where N is the training set size and s(t)² Var (Hr(H,(t) over the training examples. A proof can be found in (Wu et al. 2018). We call this point on the trajectory on which the LHS of Eq, becomes equal to for the first time the break-even point. By definition, there exists only a single break-even point the training trajectory.
Under Assumption 3, X/ (t) and (t) increase over time. If S = N, the break-even point is reached at {1x()) (t) = 2, More generally, it can be shown that for 7/. the break-even point is reached
We include similar argument for the opposite case in App. B
4

---

Published as a conference paper at ICLR 2020
for lower magnitude of X1 (t) than for 122- The same reasoning can be repeated for S (in which case we assume N >> S). We state this formally and prove in App. B
Under Assumption 4, after passing the break-even point on the training trajectory, SGD does not enter regions where either X or X is larger than at the break-even point, as otherwise would lead to increasing one of the terms in LHS of Eq. and hence losing stability along e/.-
Two conjectures about real DNNs. Assuming that real DNNs reach the break-even point, we make the following two conjectures about their optimization trajectory.
The most direct implication of reaching the break-even point is that d and X71 at the break-even point depend 1) and S, which we formalize as:
Conjecture (Variance reduction effect of SGD). Along the SGD trajectory, the maximum altained values of X and Nj are smaller for larger learning rate or a smaller batch size.
We refer to Conjecture as variance reduction effect of SGD because reducing A can be shown to reduce the Le distance between the full-batch gradient, and the mini-batch gradient. We expect that similar effects exist for other optimization or regularization methods. We leave investigating them for future work.
Next, we make another, stronger. conjecture. It is plausible to assume that reaching the break- point affects to a lesser degree X/₃ and A/x for of 1 because increasing their values does not impact stability along eh- Based on this we conjecture that:
Conjecture (Pre-conditioning effect of SGD). Along the SGD trajectory, the maximum attained values of Nig and Nit are larger for larger learning rute or a smaller batch size, where AK" and Aн" are the smallest non-zero eigenvalues of K and H, respectively. Furthermore, the maximum attained values of (r() and Tr(H) are smaller for larger learning rate or a smaller batch size.
We consider non-zero eigenvalues in the conjecture, because K has at most N non-zero eigen- values, where N is the number of training points, which can be much smaller than in over- parametrized DNNs. Both conjectures are valid only for learning rates and batch sizes that guarantee that training converges.
From the optimization perspective, the effects discussed above are desirable. Many papers in the optimization literature underline the importance reducing the variance of the mini-batch gradi- ent (Johnson& & Zhang] 2013) and the conditioning of the covariance of gradients (Roux et al. 2008). There also exists a connection between these effects and generalization (Jiang et al. 2020 which discuss towards the end of the paper.
4 EXPERIMENTS
In this section we first analyse learning dynamics in the early phase of training. Next, we empirically investigate the two conjectures. In the final part we extend our analysis to neural network with batch normalization layers.
We run experiments on the following datasets: CIFAR-10 (Krizhevsky 2009). IMDB dataset (Maas) al., 2011). ImageNet (Deng et al. 2009), and MNLI (Williams al. 2018). We apply to these datasets the following architectures: a vanilla CNN (SimpleCNN) following Keras exam- ple (Chollet et al.] 2015), ResNet-32 (He et al., 2013), LSTM (Hochreiter & Schmidhuber 1997), DenseNet (Huang et al. 2016), and BERT (Devlin et al. 2018). We also include experiments using a multi-layer perceptron trained on the FashionMNIST dataset (Xiao al. 2017) in the Appendix.
All experimental details are described in App. D
Following Dauphin et al. (0014); Alain et al. 2201)), we estimate the top eigenvalues and eigenvec- tors of H on a small subset of the training set (e.g. 5% in the case of CIFAR-10) using the Lanczos algorithm (Lanczos 1950). As computing the full eigenspace of K is infeasible for real DNNs, we compute the covariance using mini-batch gradients. In App. C we show empirically that (after malization) this approximates well the largest eigenvalue, and we include other details on computing the eigenspaces.
5

---

Published as a conference paper at ICLR 2020
10a
n= 0.001
7=0.010
0.01
n
~I: 102
0.00
10
10
10 A
10
10
10
10
The spectral norm of H (A left) and AL (difference in the training loss computed be- tween two consecutive steps, right) versus A different training iterations. Experiment was per- formed with SimpleCNN on the CIFAR-10 dataset with two different learning rates (color). Consis- tently with our theoretical model, A correlated initially with Ajz, and training generally stable (AC 0) prior achieving the maximum value of X
Figure 2:
0.02
n= 0.001
0.001
n=0.010
0=000
0.01
0.00
5
10
15
250
500
750
1000
1250
2
A
Figure 3: The spectrum of K (left) and H (right) at the training iteration corresponding to the largest value of A and } respectively. Experiment was performed with SimpleCNN on the CIFAR-10 dataset with two different learning rates (color). Consistently with Conjecture 2 training with lower learning rate results in finding a region of the loss surface characterized by worse conditioning of K and H (visible in terms of the large number of "spikes" in the spectrum, see also Fig. 4
4.1 CLOSER LOOK AT THE EARLY PHASE OF TRAINING
First, examine the learning dynamics in the early phase of training. Our goal is to verify some of the assumptions made in Sec. 3 We analyse the evolution X and A when using 1j = 0.01 and 1) 0.001 to train SimpleCNN different random initializations of network parameters.
the CIFAR-10 dataset. We repeat this experiment 5 times using
Visualizing the break-even point. We visualize the early part of the optimization trajectory in Fig. M Following Erhan et al. (2010), we embed the test set predictions at each step of training of SimpleCNN using UMAP (Melnnes al. 2018). The background color indicates X (left) and the training accuracy (right) at the iteration with the closest embedding in Euclidean distance.
We observe that the trajectory corresponding to the lower learning rate reaches regions of the loss surface characterized by larger Al. compared to regions reached at the same training accuracy in second trajectory. Additionally, Fig. we plot the spectrum of K (left) and (right) at iterations when Ax and λy respectively reach the highest values. We observe more outliers for the lower learning rate in the distributions of both Åς and λи-
Are X and X correlated in the beginning of training? The key assumption behind our theo- retical model is that X and A3 are correlated, at least prior to reaching the break-even point. We confirm this in Fig. 2 The highest achieved A and X are larger for the smaller 7). Additionally, we observe that after achieving the highest value of H. further growth of A does not translate to an increase of A This expected as X decays to 0 when the mean loss decays to 0 for cross entropy loss (Martens) 2016).
Does training become increasingly unstable in the early phase of training? According to As- sumption 3, an increase of X/ and X translates into decrease in stability, which formalized as stability along e/y- Computing stability along ej directly is computationally expensive. Instead, we measure more tractable proxy. At each iteration measure the loss the training set before and

---

Published as a conference paper at ICLR 2020
10
0.3
0.3
109
0.1
50
100
350
50
300
150
50
300
150
50
100
150
Fpoch
Epoch
Epoch
Epoch
0.001
-
n= 0.010
0.200
=0
-X-
5=000
(a) SimpleCNN trained on the CIFAR-10 10 dataset.
10
10-4
0.1
0.1
50
100
150
50
100
150
50
100
150
100 Epoch
150
Epoch
Epoch
Epoch
1 0.001
+ n 0.010
2-
0.100
100
(b) ResNet-32 trained on CIFAR- 10 dataset.
0.15
10
0.30
10
10-4
0.00
15
10
20
30
Epoch
Epoch
n=
n
100
(c) LSTM trained the IMDB dataset.
Figure 4: The variance reduction and the pre-conditioning effect of SGD in various settings. The optimization trajectories corresponding to higher learning rates (1)) or lower batch sizes (S) are characterized by lower maximum X (the spectral norm of the covariance of gradients) and larger maximum Xi/ /Ak (the condition number of the covariance of gradients). Vertical lines mark epochs at which the training accuracy is larger (for the first time) than a manually picked threshold, which illustrates that the effects not explained by differences in training speeds.
after taking the step, which we denote as AL (a positive value indicates reduction of the training loss). In Fig. 2 we observe that training becomes increasingly unstable (AC starts take negative values) as d5 reaches the maximum value.
Summary. We have shown that the early phase of training is consistent with the assumptions made in our theoretical model. That is, X and A11 increase approximately proportionally to each other, which is also generally correlated with a decrease of a proxy of stability. Finally, we have shown qualitatively reaching the break-even point.
4.2 THE VARIANCE REDUCTION AND THE PRE- CONDIIINIGGGNING EFFECT OF SGD
In this section we test empirically Conjecture I and Conjecture 2 For each model we manually pick suitable range of learning rates and batch sizes to ensure that the properties of K and that we study have converged under reasonable computational budget. We mainly focus on studying the covariance of gradients (K), and leave closer investigation of the Hessian for future work. We use the batch size of 128 to compute K when vary the batch size for training. When we vary the learning rate instead, we use the same batch size as the one used to train the model. App. Cedecribess the remaining details on how we approximate the eigenspaces of K and H.
We summarize the results for SimpleCNN, ResNet-32, LSTM, BERT, and DenseNet in Fig. 4 Fig. and Fig. 6 Curves are smoothed using moving average for clarity. Training curves and additional experiments are reported in App. E
Testing Conjecture 1. To test Conjecture 1, we examine the highest value of A observed along the optimization trajectory. As visible in Fig. 4 using higher y) results in X achieving lower
7

---

Published as a conference paper at ICLR 2020
10"
10
4:27
1x1F
2x39
10
10:
10
100
150
100
150
50
100
Epoch
Epoch
Epoch
Epoch
100000
n-
9- 0.100
100
(a) Varying the learning rate.
(b) Varying the hatch size.
Figure 5: The variance reduction effect of SGD, for ResNet-32 and SimpleCNN. Trajectories cor- responding to higher learning rates (7, left) or smaller batch sizes (S, right) are characterized by lower maximum X/1 (the spectral norm of the Hessian) along the trajectory. Vertical lines mark epochs at which the training accuracy is larger (for the first time) than a manually picked threshold.
maximum during training. Similarly, we observe that using a higher S in SGD leads to reaching higher maximum value of Ak. For instance, for SimpleCNN (top row of Fig. we observe max (Ax) = 0.68 and max(A 3.30 for = 0.1 and r7 = 0.01, respectively.
Testing Conjecture 2. To test Conjecture 2, we compute the maximum value of / along the optimization trajectory. It is visible in Fig. 4 that using higher 7) results in reaching larger maximum value of X / along the trajectory. For instance, the case of SimpleCNN max(Xx/X) 0.37 and (aaxx>//X)
0.24 for 0.1 and r = 0.01, respectively.
counter-intuitive effect of decreasing the batch size. Consistently with Conjecture 2 we ob- serve that the maximum value of Tr(K) is smaller for the smaller batch size. In the case of Sim- pleCNN max(Tr (K)) = 5.56 and max (rr(K) = 10.86 for S = 10 and S = 100, respectively. Due to space constraints report the effect of r) and S Tr(K) in other settings in App.
This effect is counter-intuitive because Tr(K) is proportional to the variance of the mini- gra- dient (see also App-CC C Naturally, using lower batch size generally increases the variance of the mini-batch gradient, and Tr(K). This apparent contradiction is explained by the fact that we mea- sure Tr(K) using different batch size (128) than the one used to train the model. Hence, decreasing the batch size both increases (due to approximating the gradient using fewer samples) and decreases (as predicted in Conjecture 2) the variance of the mini-batch gradient along the optimization trajec- tory.
How early in training is the break-even point reached? We find that A and > reach their highest values early in training, close to reaching 60% training accuracy on CIFAR-10, and 75% training accuracy on IMDB. The training and validation accuracies are reported for all the experi- ments in App. This suggests that the break-even point is reached early in training.
The Hessian. In the above, we have focused on the covariance of gradients. In Fig. 5 we report how X yy depends on 17 and S for ResNet-32 and SimpleCNN. Consistently with prior work (Keskar eet al. [017]] Jastrzebski et al. 2018), we observe that using smaller y) or using a larger S coincides with larger maximum value of X For instance, for SimpleCNN we observe max(> 26.27 26.27 and max(A}) ( = 211.17 for 0.1 and = 0.01, respectively. We leave testing predictions made in Conjecture about the Hessian for future work.
Larger scale studies. Finally, we test the two conjectures in two larger scale settings: BERT fine-tuned on the MNLI dataset, and DenseNet trained on the ImageNet dataset. Due memory constraints, we only vary the learning rate. We report results in Fig. 6 We observe that both conjectures hold in these two settings. worth noting that DenseNet uses batch normalization layers. In the next section we investigate closer batch-normalized networks.
Summary. In this section we have shown evidence supporting the variance reduction (Conjec- ture [I) and the pre-conditioning effect (Conjecture 22 of SGD in range of classification tasks. We also found that the above conclusions hold for MLP trained on the Fashion MNIST dataset, SGD with momentum, and SGD with learning rate decay. We include these results in App. E G
8

---

Published as a conference paper at ICLR 2020
104
10
Agel
04
2x28
Epoch
Epoch
Epoch
Epoch
0-0.001
0.010
1200.001
11
BERT trained the MNLI dataset.
DenseNet trained the ImageNet dataset.
Figure 6: The variance reduction and the pre-conditioning effect of SGD, demonstrated two larger scale settings: BERT on the MNLJ dataset (left), and DenseNet on the ImageNet dataset (right). For each setting report A (left) and Xxx /> (right). Vertical lines mark epochs which the training accuracy is larger (for the first tim) than a manually picked threshold.
4.3 IMPORTANCE OF LEARNING RATE FOR CONDITIONING IN NETWORKS WITH BATCH NORMALIZATION LAYERS
102
107
2x32
10
Lixie
10-1
100 Epoch
100
150
100
150
Epoch
Epoch
n-
0.100
Left right: SimpleCNN-BN.
Jell for SimpleCNN-BN,
lal 96
for SimpleCNN, A for SimpleCNN-BN, and A for
M9s
044
Iivil
0.3
0.2
10-1
0.1
Epach
Epoch
Epoch
0.001
-H-
0.010
0.100
1.000
(b) Left to right: || of the last layer. Ak. and Ax %x/2
/1k. All for SimpleCNN-BI
The effect of changing the learning rate various metrics (see text for details) for Sim- pleCNN with and without batch normalization layers (SimpleCNN-BN and SimpleCNN).
Figure 7:
The loss surface of deep neural networks has been widely reported to be ill-conditioned (LeCun cet al., 2012); Martens] 2016). Recently, Ghorbani et al. (2019); Pagel (2019) argued that the key reason behind the effiacyy of batch normalization (loffe & Szegedy 2015) ) improving conditioning of the loss surface. In Conjecture we suggest that using high 7 (or a small S) results in improving the conditioning of and H. natural question that we investigate this section how the two phenomena are related. We study here the effect of learning rate, and report in App. aaan analogous study for batch size.
Are the two conjectures valid in networks with batch normalization layers? First, to inves- tigate whether our conjectures hold in networks with batch normalization layers, we run similar experiments in Sec. 4.2 with SimpleCNN model with batch normalization layers inserted after each layer (SimpleCNN-BNN), on the CIFAR-I0 dataset. We test E (0.001, 0.01, 0.1, 1.0} (using 7 1.0 leads to divergence of SimpleCNN without BN). We summarize the results in Fig. 7 We observe that the evolution Xg// A and A is consistent with both Conjecture and Conjecture
A closer look at the early phase of training. To further corroborate that analysis applies to networks with batch normalization layers, we study the early phase training of SimpleCNN-Bi complementing the results in Sec. 4.1

---

Published as a conference paper at ICLR 2020
We observe in Fig. (bottom) that training of SimpleCNN- starts in region characterized by relatively high Ag. This is consistent with prior work showing that networks with batch normaliza- tion layers can exhibit gradient explosion in the first iteration (Yang et al. 2019). The value of X then decays for all but the lowest 7)- This behavior is consistent with our theoretical model. We also track the norm of the scaling factor in the batch normalization layers, ||> in the last layer of the (bottom). It is visible that 1 1.0 and r) = 0.1 initially decrease the value of 1 which we hypothesize to one of the mechanisms due which high 7) steers optimization towards better conditioned regions of the loss surface in batch-none nothhhoommllzdd networks. Interestingly. this seems consistent with Luo al. (2019) who argue that using mini-batei statistics in batch normalization acts as an implicit regularizer by reducing ||>/|
network in Fig. 7
Using batch normalization requires using high learning rate. As our conjectures hold for SimpleCNN-BAN natural question if the loss surface can be ill-conditioned with low learning rate even when batch normalization is used. Ghhrbann et al. (2019) show that without batch normal- ization, mini-hatch gradients are largely contained the subspace spanned by the top eigenvectors of noncentered K. To answer this question we track || /Ill/llll |s|l, where denotes the mini-batch gradient, and 9s denotes the mini-batch gradient projected onto the top 5 eigenvectors of K. value of |/ close to implies that the mini-batch gradient is mostly contained in the subspace spanned by the top eigenvectors of K.
We compare two settings: SimpleCNN-BN optimized with 7/ 0.001, and SimpleCNN optimized with 77 = 0.01. We make three observations. First, the maximum and minimum values of ||9l|/|l |s 1.90 (1.37) and 2.02 (1.09), respectively. Second, the maximum and minimum values of X are 12.05 and 3.30, respectively. Finally, / reaches 0.343 in the first setting, and 0.24 in the second setting. Comparing these differences differences that induced by using the highest 1.0 in SimpleCNN-BN, we can conclude that using large learning rate is necessary to observe the effect of loss smoothing which was previously attributed to batch normalization alone (Ghorbani cet al. 2019 Page 2019; Bjorck et al. 2018). This might be directly related to the result that using a high learning rate necessary to achieve good generalization when using batch normalization layers Bjorcck et al. 2018).
Summary. We have shown that the effects of the learning rate predicted in Conjecture and Conjecture 2 hold for network with batch normalization layers, and that using high learning rate is necessary in network with batch normalization layers to improve conditioning of the loss surface, compared to conditioning of the loss surface in the same network without batch normalization layers.
CONCLUSION
Based on our theoretical model, we argued for the existence of the break-even point on the opti- mization trajectory induced by SGD, We presented evidence that hyperparameters used in the early phase of training control the spectral norm and the conditioning of K (a matrix describing noise in the mini-batch gradients) and (a matrix describing local curvature of the loss surface) after reach- ing the break-even point. In particular, using large initial learning rate steers training to better conditioned regions of the loss surface, which is beneficial from the optimization point of view.
A natural direction for the future is connecting our observations to recent studies the relation of measures, such as gradient variance, to the generalization of deep networks (Li et al. 2019 Jiang et al., 2020 Fort al. 2019). Our work shows that the hyperparameters of SGD control these measures after the break-even point. Another interesting direction is to understand the connection between the existence of the break-even point and the existence of the critical learning period in training of DNNs (Achille et al. 2017).
ACKNOWLEDGMENTS
KC thanks NVIDIA and eBay for their support. SJ thanks Amos Storkey and Luke Darlow for fruitful discussions.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep neural networks. CoRR, abs/1711.08885 2017.
Guillaume Alain, Nicolas Le Roux, and Pierre-Antoine Manzagol. Negative eigenvalues of the hessian in deep neural networks. CoRR, abs/1902.02366, 2019.
Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. convergence analysis of gradient de- scent for deep linear neural networks. In International Conference on Learning Representations, 2019.
Peter I. Bartlett, Dylan Foster, and Matus Telgarsky. Spectrally-normalizeed margin bounds for neural networks. Advances in Neural Information Processing Systems. 2017.
Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normal- ization. In Advances in Neural Information Processing Systems. 2018.
François Chollet et al. Keras, 2015.
Yann N. Dauphin, Razvan Pascanu, Çaglar Gülçehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex op- timization. CoRR, abs/1406.255 2572, 2014.
J. Deng, W. Dong, R. Socher, L.JJ Li, K. Li, and L.. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/810.0488 2018.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res., 2010.
Stanislav Fort and Surya Ganguli. Emergent properties of the local geometry of neural loss land- scapes. arXiv preprint arXiv:1910.05929, 2019.
Stanislav Fort and Adam Scherlis. The goldilocks zone: Towards better understanding of neural network loss landscapes. In Proceedingy of the AAAI Conference on Artificial Intelligence, 2019.
Stanislav Fort, Pawet Krzysztof Nowak, Stanislaw Jastrzębski, and Srini Narayanan. Stiffness: A new perspective on generalization in neural networks. arXiv preprint arXiv: 109491, 2019.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Time Matters in Regularizing Deep Net- works: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence. arXiv preprint arXiv: 1.XXvv1905.132777 2019.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. 2016.
Ian J. Goodfellow, Oriol Vinyals, and Andrew M. Saxe. Qualitatively characterizing neural network optimization problems, arXiv preprint arXiv:1412.6544 2014.
Hangfeng He and Weijie Su. The local elasticity of neural networks. In International Conference on Learming Representations, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 1997.
Gao Huang. Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. CoRR, abs/1608 06993,,, 2016.
11

---

Published as a conference paper at ICLR 2020
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32Nd International Conference on Inter- national Conference Machine Learning, 2015.
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Ben- gio, and Amos J. Storkey. Three factors influencing minima in SGD. CoRR, 0tos/1711.0400 1.04623, 2017.
Stanislaw Jastrzebski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length. arXiv preprint arXiv: 1807.0531, 2018.
Yiding Jiang, Behnam Neyshabur, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Fantas- tic generalization measures and where find them. In International Conference on Learning Representations, 2020.
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems 26. 2013.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference Learning Representations, ICLR, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Janice Lan, Rosanne Liu, Hattie Zhou, and Jason Yosinski. LCA: Loss Change Allocation for Neural 2019.
Network Training, arXiv preprint
Comelius Lanezos. An iteration method for the solution of the eigenvalue problem of linear differ- ential and integral operators. Res. Natl. Bur. Stand. B, 1950.
Yann LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robed Müller. Efficient backprop. In Neural Networks: Tricks of the Trade (2nd ed.). 2012.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect initial large learning rate in training neural networks. In Advances in Neural Information Processing Systems. 2019.
Jinlong Liu, Yunzhi Bai, Guoqing Jiang, Ting Chen, and Huayan Wang. Understanding why neural networks generalize well through gsnr of parameters. In International Conference on Learning Representations, 2020.
Ping Luo. Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. In International Conference on Learning Representations, 2019.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 2011.
James Martens. Second-order optimization for neural networks. University of Toronto (Canada), 2016.
Dominic Masters and Carlo Luschi. Revisiting small batch training for deep neural networks. CoRR, Abss/804.076122 2018
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 2018.
Behnam Neyshabur. Implicit regularization in deep learning. CoRR, .abp////9..095... 2017.
David Page. How train your resnet Batch norm. 2019.
Vardan Papyan. Measurements three-level hierarchical structure in the outliers in the spectrum of deepnet hessians. CoRR, hhs/1901.08244 2019.
12

---

Published as a conference paper at ICLR 2020
Nicolas L. Roux, Pierre antoine Manzagol, and Yoshua Bengio. Topmoumoute online natural gra- dient algorithm. In Advances in Neural Information Processing Systems. 2008.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeey Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
Levent Sagun, Utku Evei, V. Ugur Güney, Yann N. Dauphin, and Léon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. CoRR, abs/ abs/1706.04444, 2017.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018.
Lei Wu, Chao Ma, and Weinan E. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnise mnist: novel image dataset for benchmark- ing machine learning algorithms. CoRR, hass/1708.07777, 2017.
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A Walk with SGD. arXiv preprint arXiv: (00::/8002.0777 2018.
Greg Yang. Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. A mean field theory of batch normalization. CoRR,
2019.
Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W. Mahoney. Hessian-based analysis of large batch training and robustness to adversaries. CoRR, abs/ 802.08241, 2018
Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B Grosse. Which algorithmic choices matter which batch sizes? insights from a noisy quadratic model. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Bucc E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8196- 8207. Curran Associates, Inc., 2019.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Minima and Regularization Effects. arXiv preprint arXiv: 1eXx::1803300193 2018.
13

---

Published as a conference paper at ICLR 2020
A ADDITIONAL VISUALIZATION OF THE BREAK-EVEN POINT
We include here an analogous Figure to Fig. | but visualizing the conditioning of the covariance of gradients (A //x left), the trace of the covariance of gradients (Ir(K), middle), and the spectral norm the Hessian (A right).
1000 Erice
Figure 8: Analogous Fig. 1 The background color indicates the conditioning of the covariance of gradients K (Ax /xx, left). the trace of the covariance of gradients (TT(KK, middle), and the spectral norm of the Hessian (A₁, right).
B PROOFS
In this section we formally state and prove the theorem used in Sec. duced in Sec. mind, we state the following:
With the definitions intro-
Theorem 1, Assuming that training is stable along e) (1) at O, then N (1) and X1c(1) (1) at which SGD becomes unstable along eig (f) are smaller for larger smaller S.
Proof. This theorem is a straightforward application of Theorem from Wu et al. (2018) to the
early phase of training. First, let consider two optimization trajectories corresponding using two different learning rates !/I and 7/2 (1) >> 712). For both trajectories, Theorem of Wu et al. (2018)
states that SGD is stable an iteration if the following inequality satisfied:
72(n - 5) S(n - 1)
(1 - (t)) + ((t)
1
(2)
where stt) (t)] Using Assumption we get Var[g:(w)] Var[y, = sf2(()) Using
this we can rewrite inequality 2
(t) (n < (1 (1- (t)) + (t)² S(n-1)
(3)
To calculate the spectral norm of the Hessian at iteration 1" at which training becomes unstable we equate the two sides of inequality (3) and solve for \1 (1"), which gives
(n
2
(11)
w(1)) (n
where denotes the proportionality constant from Assumption 2. Note that if S the right hand side degenerates to "/ which completes the proof for n = S.
To prove the general case, let denote by (tf) and A (t5) the value of A at which training becomes unstable along ely for 77, and 7/,, respectively. Similarly, let us denote by (btt) and (t;) the corresponding values of th.
14

---

Published as a conference paper at ICLR 2020
Let us assume by contradiction that (ti) > (t5). A necessary condition for this inequality to hold is that vttt) > ((t5). However, Assumption implies that prior to reaching the break-even point (t) decreases monotonically with increasing X (ti), which corresponds to reducing distance to the minimum along <tz, which contradicts X (fT) X (45). Repeating the same argument for two trajectories optimized using two different batch sizes completes the proof.
It also straightforward to extend the argument to the case when training initialized at an unstable region along e) (0), which formalize as follows.
Theorem 2. If training is unstable along (t) at tt = 0, then (t) and (t(() at which SGD becomes for the first time stable along ej (t) are smaller for larger nj or a smaller
optimization trajectories corresponding to using two different learning rates 7/1 and 7/2 (71 > 1/2).
Proof. We will
similar argument in the proof of Th. [ Let us consider
Let X (ti) and X (t%), denote the spectral norm of H the iteration which training is for the first time stable along e), for and 1/2, respectively. Following the steps in the proof of Th. we get that
2 (t))
(n- S 8((") (n-1) 1J B
(5)
7
Using the same notation as in the proof of Th. let us assume by contradiction that (ti) >
(f2). Assumption 3 implies that (t) increases with decreasing (t). Hence, if A (f2) is
smaller, means (t5) increased to a larger value, i.c. (t(t) ((t)). However, if (n-9)
(n-1) 1,
necessary condition for (11) > (15) inequality to hold is that (b()) > vi(1),, which leads to contradiction. Repeating the same argument for batch size completes the proof.
C APPROXIMATING THE EIGENSPACE OF K AND H
Using a small subset of the training set suffices to approximate well the largest eigenvalues of the Hessian on the CIFAR-10 10 dataset (Alain et al. 2019 5 Following Alain et al. (2019) we use ap- proximately the same 5% fraction of the dataset in CIFAR-10 experiments, and the SCIPY Python package. We use the same setting on the other datasets as well.
Analysing the eigenspace of K is less common in deep learning. For the purposes of this paper, are primarily interested in estimating Ax" * and Ag. We also estimate Tr(.). is worth noting that Tr(K) is related to the variance in gradients as Tr(K) = 1 the number of parameters and N the number of datapoints.
D EN llg gill?, where is the
full-batch gradient,
The naive computation of the eigenspace of K is infeasible for realistically large deep networks due to the quadratic cost in the number of parameters. Instead, we compute K using mini-batches. To avoid storing D x D matrix in memory, we first sample L mini- gradient of size M and compute the corresponding Gram matrix KM that has entries K- = t. (g. g, 99 g). where is the full-batch gradient, which we estimate based on the L mini-batches. To compute the eigenspace of KM M we use SVD routine from the NumPy package. We take the (L- - 1 smallest eigenvalue as the smallest non-zero eigenvalue of K (covariance matrix computed using L observations has by definition L - non-zero eigenvalues).
Papyan 2019); : Fort & Ganguli (2019) show that the top eigenvalues of H emerge due clustering of gradients of the logits. See also Fort & Ganguli (2019). Based on the proximity of the largest eigenvalues of H and K, this observation suggests that the top eigenvalue of K might be similar to that of K M To investigate this, we run the following experiment on the CIFAR-10 10 dataset using
This might he due to the hierarchical structure of the Hessian (Papyan 2019 Fort & Ganguli 2019). In particular, they show that the largest eigenvalues of the Hessian connected the class structure the data.
15

---

Published as a conference paper at ICLR 2020
SimpleCNN. We estimate A using M 1 and M 128, in both using the whole training set. We subsample the dataset to 10% to speed up the computation. Fig. F shows a strong correlation between x computed with M and with M 128, for three different learning rates.
n- 0.01
n- 0.001
50
c- p.00.p-- 1.70-10
p- 0.00, p-
p 0.22, p-5.80
3.0
20
52
0.100
12.00
Figure 9: Pearson correlation between A calculated using either M 128 M for three different values of 0.1, 0.01 and 0.001.
In all experiments gradients to L = 200 to reduce noise (same conclusion hold for L = 25). For instance, on the CIFAR-10 dataset this amounts using approximately 5% of the training set. In experiments that vary the batch size we use M = 128. Otherwise, we use the value M as the batch size used to train the model.
25, and for the IMDB dataset increase the number of mini-batch
D EXPERIMENTAL DETAILS FOR SEC.
In this section we describe all the details for experiments Sec.
ResNet-32 on CIFAR-10. ResNet-32 (He et al. 2015) trained for 200 epochs with a batch size equal to 128 on the CIFAR-10 dataset. Standard data augmentation and preprocessing is applied. Following He et al. 2015), we regularize the model using weight decay 0.0001. We apply weight decay to all convolutional kernels. When varying the batch size, we use learning rate of 0.05. When varying the learning rate, we use batch size of 128.
SimpleCNN on CIFAR-10. SimpleCNN is a simple convolutional network with four convolu- tional layers based on Keras examples repository (Chollet et al., (015). The architecture is fol- lows. The first two convolutional layers have 32 filters, and last two convolutional layers have 64 filters. After each pair of convolutional layers, we include max pooling layer with window size of 2. After each layer we include ReLU nonlinearity. The output of the final convolutional layer processed by densely connected layer with 128 units and ReLU nonlinearity. When varying the batch size, we use a learning rate of 0.05. When varying the learning rate, we use a batch size of 128.
BERT on MNLI. The model used in this experiment is the BERT-base from Devlin et al. (2018) pretrained on multilingual dat:"] The model is trained on the MultiNLI dataset (Williams et al, 2018) with the maximum sentence length equal to 40. The network is trained for 20 epochs using batch size of 32. Experiments are repeated with three different seeds that control initialization and data shuffling.
MLP on FaShionMNIST.. This experiment is using multi-layer perceptron with two hidden lay- ers size 300 and 100, both with ReLU activations. The data is normalized to the [0, 1] range. The network is trained with batch size of 64 for 200 epochs.
LSTM on IMDB. The network used in this experiment consists of an embedding layer followed by an LSTM with 100 hidden units. We use vocabulary size of 20000 words and the maximum
'The model weights used can he found
(ttproooooe///ee_mmmmmmam cased
1-_22-H1 768 12/1
16

---

Published as a conference paper at ICLR 2020
length of the sequence equal to 80. The model is trained for 100 epochs. When varying the learning rate, we use batch size of 128. When varying the batch size, we use learning rate 1.0. Experiments repeated with two different seeds that control initialization and data shuffling.
DenseNet ImageNet. The network used is the DenseNet-121 from Huang al. (016). The dataset used is the ILSVRC 2012 (Russakovsky et al. 2015). The images are centered and normal- ized. No data augmentation is used. Due large computational cost, the network is trained only for 10 epochs using a batch size of 32.
E ADDITIONAL EXPERIMENTS FOR SEC.
In this section we include additional data for experiments in Sec. 4.2 as well as include experiments using MLP trained on the Fashion MNIST dataset.
SN on CIFAR-10. Fig. 12 and Fig. 13 we report accuracy on the training set and the validation set, Alg. and Tr( for all experiments with SimpleCNN model on the CIFAR-10 dataset
Train accuracy
6.8
0.0
Tr(0
300
0.2
50
100 Epoch
150
50
1000
150
50
100
150
Epoch
Epoch
0= 0.0.001
0.001
+ n=
0,010
n= 0.100
Figure 10: Additional metrics for the experiments using SimpleCNN on the CIFAR-10 dataset with different learning rates. From left to right: training accuracy, validation accuracy, Tr(K).
accuracy
accuracy
0.88
Tr(K
0.6
100
0.4
300
150
Epoch
Epoch
Epocc
S 30
-+
S
300
Figure 11: Additional metrics for the experiments using SimpleCNN on the CIFAR-10 dataset with different batch sizes. From left to right: training accuracy, validation accuracy, and Tr(K).
ResNet-32 on CIFAR-10. Fig. 12 and Fig. 13 we report accuracy on the training set and the validation set, Az, and Tr(K) for all experiments with ResNet-32 on the CIFAR-I0 dataset.
Tran accuracy
Vol. sccuracy C
0.75
30.75
0.50
0.50
300
10-1
Epoch
0.001
Figure 12: Additional figures for the experiments using ResNet-32 32 on the CIFAR-10 dataset with different learning rates. From left right: the evolution of accuracy, validation accuracy, Tr(K).
17

---

Published as a conference paper at ICLR 2020
Train accuracy
0.8
0.8
10"
8.6
0.4
10
50
100
150
100 Epoch
150
50
300
150
Epoch
Epoch
10
-+-
5=000
Additional figures for the experiments using ResNet-32 on the CIFAR-10 dataset with different batch sizes. From left to right: training accuracy, validation accuracy, Tr(K).
Figure 13:
LSTM on IMDB. In Fig. and Fig. 15 we report accuracy on the training set and the validation set, Xz., and (r(K) for all experiments with LSTM on IMDB dataset.
Train accurary
accuracy 0.7
10
0.8
Triko
0.6
10-2
Epoch
1.000
3.000
Figure 14: Additional figures for the experiments using LSTM on the IMDB dataset with different learning rates. From left to right: the evolution of accuracy, validation accuracy, and Tr(K).
Train accurscy
MN MP
accuracy
0.8
0.7
10
10
0.6
50
10
Epoch
Epoch
Epoch
5 10
-H-- 5 100
Figure 15: Additional metrics for the experiments using LSTM on the IMDB dataset with different batch sizes. From left to right: training accuracy, validation accuracy, X and Tr(K).
BERT MNLI. In Fig. 16 report accuracy the training set and the validation and Tr(K) for BERT model on the MNLI dataset.
Train Trann accuracy
Val. sccurasy
0.8
10
0.9
Ex2a
10 Epoch
15
10
Epoch
n=
n
Figure 16: Additional metrics for the experiments using BERT the MNLI dataset with different learning rates. From left to right: training accuracy, validation accuracy, and Tr(K).
DenseNet on ImageNet. Fig. 17 we report accuracy on the training set and the validation set and Tr(K) for DenseNet on the ImageNet dataset.
18

---

Published as a conference paper at ICLR 2020
Train accurscy
1.0
M
Val. accuracy
10
0.4
29381
Epoch 10
10
Epoch
Epoch
0.001
0=0000
7= 0.100
Figure 17: Additional metries for the experiments using DenseNet on the ImageNet dataset with different learning rates. From left to right: training accuracy, validation accuracy, and Tr(K).
MLP on MNSS. Fig. 18 and Fig. 19
we report results for MLP model on the Fash- ionMNIST dataset. We observe that all conclusions carry over this setting.
0.075
10
101
0.050
10
10-2
00.3
30
100
150
200
50
100
150
200
50
100
150
50
100
150
200
Epoch
Epoch
71
00001
9-
12-
0.100
1)
7 0.010
0- 0.100
Figure 18: Results of experiment using the MLP model the MaSSI dataset for different learning rates. From left to right: Al. Xx/ A X and Tr(K).
Train Sccursc
Vel. accuracy
0.9
0.8
0.8
50
1000
150
100
150
Epoch
Epoch
n= 0.001
-
n= 0.010
=
0.100
Figure 19: Training accuracy and validation accuracy for experiment in Fig.
F ADDITIONAL EXPERIMENTS FOR SGD WITH MOMENTUM
Here, we run the same experiment as in Sec. using SimpleCNN on the CIFAR-10 dataset. Instead of varying the learning rate, test different values of the momentum 3 parameter in the range of 0.1, 0.5 and 0.9. We can observe that Conjecture I and Conjecture 2 generalize to momentum in the sense that using higher momentum has analogous effect using higher learning rate, using a smaller batch size in SGD. We report the results in Fig. 20 and Fig. 21
10t
107
0.2
0.1
6x38
2x23
Epoch
Epoch
Epoch
B- 0.000
P- 0.500
8- 0.900
Figure 20: The variance reduction and the pre-conditioning effect for SimpleCNN trained using SGD with momentum. From left right: X Xx and Tr(K).
19

---

Published as a conference paper at ICLR 2020
Train accuracy
1.00
Val. accuracyy
0.8
0.75
0.6
0,4
50
100
150
50
100 Epoch
150
Epoch
p 0.100
---
= 0.500
0.900
Figure 21: Training accuracy and validation accuracy for the experiment in Fig. 20
Next, to study whether our Conjectures are also valid for SGD with momentum, but held constant, we run the same experiment as in Sec. [4.2 using SimpleCNN on the CIFAR-I0 dataset. For runs,
we set momentum to 0.9. Learning rate 0.1 diverged training, so we include only 0.01 and 0.001. We can observe that both Conjectures generalize to this setting. We report results Fig. and Fig. 23
164
10
10
Epoch
Epoch
Epoch
0-
Figure 22: The variance reduction and the pre-conditioning effect for SimpleCNN trained using SGD with momentum. From left to right: A / A and Tr(K).
Train accuracy
3.00
0.8
0.75
0.8
Val.
Epech
Figure 23: Training accuracy and validation accuracy for the experiment in Fig. 22
G ADDITIONAL EXPERIMENTS FOR SGD WITH LEARNING RATE DECAY
To understand the effect of learning rate decay, we run the same experiment as Sec. 4.2 using SimpleCNN on the CIFAR-I0 dataset. Additionally, we divide the learning rate by the factor of 10 after 100th epoch. We can observe that Conjecture [ and Conjecture 2 generalie to scenario with learning rate schedule in the sense that changing learning rate doesn't change the relative ordering of the maximum A and A / We report the results in Fig. and Fig. 25
10a
TAKK)
10a
Epoch
Epoch
Epoch
0.030
0.100
Figure 24: The variance reduction and the pre-conditioning effect for SimpleCNN trained using SGD with learning rate schedule. From left right: X Xx / and Tr(K).
20

---

Published as a conference paper at ICLR 2020
Train accuracy
Val. accuracyy
0.8
0.73
0.8
50
100
150
50
100 Epoch
150
Epoch
n= 0.030
-+- / 0.200
Figure 25: Training accuracy and validation accuracy for the experiment Fig. 24
H ADDITIONAL EXPERIMENTS FOR SIMPLECNN-BN
To further explore the connection between conjectures and the effects of batch- natch-normalization layers on the conditioning of the loss surface, we repeat here experiments from Sec. 4.3 but varying the batch size. Fig. 26 summarizes the results.
On the whole, conclusions from Sec. 4.3 carry over to this setting in the sense that decreasing the batch size has similar effect the studied metrics increasing the learning rate. One exception the experiment using the smallest batch size of 10. In this case, the maximum values of lgg X A smaller than in the experiments using larger batch sizes.
and
[ss
20
10-1
2x30
201
50
100 Epoch
150
50
100
350
50
1000
150
Epoch
Epoch
5=10
*
50
250
5=500
(a) Left to right:
X for SimpleCNN BN, and X for SimpleCNN-BN
Tos
0.4
0.3
Epoch
(b) Left to right: |y|l the last BN layer early in training. A/2 early in training, Xx AA for SimpleCNN-BN.
Figure 26: Evolution of various metrics that quantify conditioning of the loss surface for SimpleCNN with with batch normalization layers (SimpleCNN-BN), for different batch sizes.
21


Reviews:
[{"review_id": "Syg9XXmgoB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies two objects that quantify the optimization trajectory: the Hessian of the training loss (H) that describes the curvature of the loss surface, and the covariance of gradients that quantifies noise induced by noisy estimate of the full-batch gradient. \n\nThe authors predict and demonstrate that learning rate and batch size determine H and K and demonstrate that both large learning rate and small batch size results in two effects on K and H along the trajectory: (1) variance reduction and (2) pre-conditioning. These effects are observed after the break-even point. The further verified these predictions on BN networks.\n\nComments:\n\nUnderstanding the optimization trajectory is an important topic and this work is one step towards better understanding. There are many questions I hope the authors could clarify: \n\n- the concept of break even point, which justify whether the minima is stable or not, is also presented in Wu et al, 2018, including the proof. In the introduction of equation 1, the break even point concept seems to be novel in the context of learning trajectory. What is the difference with Wu et al, 2018? Does this break even point appear only once during training?\n\n- What is the practical usage for identifying the break-even point or stable/unstable of the optimizer? Does reaching the unstable phase of SGD earlier by large learning rate/small batch size mean better performance?\n\n- It seems the author does not apply learning rate decay for all experiments. What is the connection with learning rate decay? As we often see the training loss and validation error decreases significantly during training in step based learning rate schedule, what is their corresponding change of \\lambda_H^1 and \\lambda_K^1?\n\n- The authors did not mention whether momentum is used or not during training, which is commonly used in training neural nets. How would momentum affect the conjectures?\n\n- Regarding to conjecture 1 which states that “larger” learning rate yields lower \\lamda_H^1 and \\lamda_K^1, is there a limit for the range of learning rate? If we use learning rate 10, the training may not even converge. The experiments only verifies three learning rate with a maximum value 0.1, which does not cover “large” learning rate.\n\n- Also as small batch size naturally results in more iterations than large batch given the same number of epochs. Comparing small batch and large batch may need to take this into account. By reaching the break-even-point “early”, does it mean less number of epochs? I would like to see the comparison in terms of iterations rather than epochs. \n\n- In section 4.1, the definition of \\alpha* is not clear. It was noted in Figure 2 as the width of the loss surface and is defined to be the minimum step size along the adjacent iterate direction to have 20% loss increase. What is the unit of t here, is it a batch or epoch?\nHow exactly is \\alpha calculated? Due to the scale invariance [1,2], this \\alpha is not necessarily the true `width` of the loss surface as it could be influenced by the weight norm.\nWhy do different learning rates reach a similar \\alpha? It would be better to mark the starting point of the trajectory and the ending point of trajectory.\n\n- The illustration of Figure 1a is very unclearer. What is the x-axis and y-axis? Where is exactly the break-even point? It would be helpful to mark the starting and ending point of the trajectory. Also it would be helpful to have the values marked in the contours or make a separate value bar.\n\n\n- The authors studied whether the conjectures hold for BN networks in section 4.3. They only verified learning rate but not batch size. Does the batch size part still hold? It is known that BN requires larger batch size to work well, which may contradict with the conjecture that smaller batch size works better.\n\n\nMinor:\n- I was confused by the Figure 3 where different hyperparameters has different values at epoch 0. Does epoch 0 means the first epoch? It would be clear to have the same starting point and make epoch 0 as the initialization point.  \n\n- The experiments of DenseNet on ImageNet seems incomplete as it is only trained for 10 epochs. What is the top-1 error on the validation set?\n\n[1] https://arxiv.org/abs/1703.04933 \n[2] https://arxiv.org/abs/1712.09913\n\n\n-----update after rebuttal----------\n\nAfter reading the response, I think most of my questions are properly answered and corresponding changes are made in the revision. I  changed my score to weak accept. I would hope the authors could further improve the paper by well organizing the additional experiments and findings. Too much experiment variation settings make the paper not easy to follow. I hope the authors could make it clear about the practical benefits of identifying the existence of break even point with common experiments settings, e.g., nonzero momentum, weight decay and step based learning rate decay. In the new provided experiments about learning rate decay, I did not see the normal abrupt change of validation accuracy. More analysis about whether learning rate decay helps reaching break-even point would be great. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJekcJOCtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors demonstrate that, during training, there is a point during the early phase of training that leads stochastic gradient descent (SGD) to a point where the covariance of the gradients (K) has a lower spectral norm (smaller first eigenvalue) and improved conditioning in K and the Hessian of the training loss (H).\n\nThe authors experiments seem to verify that learning rate and batch size do play a part in the spectral norm of K and the conditioning of K. My one issue is that, while effects on K produced by higher learning rates are supposed to be \"good\", the authors do not directly relate this back to model performance. From my years of experience training neural networks, I have seen many scenarios in which higher learning rates result in worse performance, even after reducing the learning rate. Can this be related back to the author's claims? Under what conditions does a higher learning rate lead to these effects on K and H and will it always lead to better model performance?\n\n\nOther comments:\nIn definitions, it says that the eigenvalue of matrix A is \\lambda_A^i, however, later in the 4th part of the assumptions, the spectral norm of H is referred to as \\lambda_1^H. Is there a difference here? Typo?\n\nThe last part of the definitions where \\Phi(\\tau) is introduced should have a formal definition for \\Phi(\\tau) as \\Phi is initially does not take any parameter \\tau.\n\nIn section 4.1, \"further growth of \\lambda_K^1 K does not translate into an increase of \\lambda_K^1\" \\lambda_K^1 is repeated. Typo?\n\n** After author response **\nChanging from weak accept to accept.\n\nThe authors have addressed my concerns about the paper.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HkgQJa95FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work analyzes the optimization of deep neural networks from the point of view of the different learning trajectories obtained during different learning settings as brought about by different hyperparameters in optimization. Specifically, the authors consider how the batch size (S) and the step-size (eta) hyper-parameters modify the learning trajectory. The authors conduct their analysis for networks with and without BatchNorm. \n\nTo get a quantitative understanding of the different trajectories of the optimization landscape the authors monitor and analyze the \nA.  Hessian of the network with respect to the parameters \nB. The covariance matrix of the per-sample gradients \nThe majority of the work focuses on monitoring A and B for various hyper-parameter regimes. \nThe authors discover a nice and clear trend, the leading eigenvalues of H (the Hessian) and K (per sample gradient covariance) are \n1. Heavily correlated to each other (Figure.2 left) \n2. Positively correlated with batch size (S) (Figures 3-5)\n3. Negatively correlated with step-size (eta) (Figures 3-5)\n\nThe 3 trends presented empirically all originate and qualitatively agree very well with equation (1) of the paper that describes stability regions in training.  Equation (1) directly prescribes the maximum allowed eigenvalue for the Hessian matrix H for a given a learning rate, and batch-size to ensure stability. For equation (1) to hold, the authors make  the assumption that H and K share a leading eigenvector direction (this assumption is partially corroborated on the CIFAR10 dataset). The empirical analysis presented in the paper analyzes training on the CIFAR10, ImageNet, IMDB, and the MNLI datasets and uses a variety of common network architectures to evaluate the empirical claims. In general, I am quite happy with the breadth and depth of the experimental results presented in the paper! \n\nThe paper makes a few observations in addition to trends 1-3, It argues that BatchNorm networks need a large learning rate to exhibit smaller dependence on the subspace of the leading eigenvectors of K,  and that the networks reach a breaking point early in training where the leading eigenvalues of H and K are chosen by the trajectory.\n\nOverall the paper is well written and the numerous experimental results are quite impressive. Nonetheless I have a few problems that are discouraging me from fully accepting this paper, so I am currently borderline accepting this paper.\n\nThe following problems should be addressed:\n\ni. The theoretical results in the paper are very similar to the results given in Wu et al. (as mentioned in the main body and Appendix A). Moreover, the modified proofs presented in the appendix are not fully explained and are difficult to follow so I do not find them rigorous. It also seems like some of the empirical analysis is also presented in Wu et al.\n\nii. As stated, Conjecture 2 is not empirically supported in the paper. Currently, Conjecture 2 proposes that for  training with smaller batches or larger learning rates, the reciprocal of the ''conditioning number'' ( least POSITIVE eigenvalue / leading eigenvalue)  reaches a larger maximum value during training. Instead the authors \"estimate\" the least positive eigenvalue by the trace of the matrix which is very much a different notion than the least positive eigenvalue and also displays a different behavior.\nIt seems maybe measuring the trace as an un-normalized quantity for the average eigenvalue is more suitable but that is different from the least positive eigenvalue. Perhaps of the same flavor would be a statement on the stable rank of the matrices in question.\n\n\niii. Figure 1 (left) is quite confusing. I hope the authors can help clarify the heatmap shown along with the trajectories. The caption mentions that the colors signify the value of the leading eigenvalue of K.  First, since this is an explicit quantity can the authors provide a value range for the heatmap? Second, and more importantly, how is this quantity, which depends on K, can be computed for all the points in the 2D grid of UMAP to form this heat map? Recall K is a quantity that is computed for a given parameterization value and fixed number of mini-batch examples. \n\niv. Lack of clarity and rigor for the eigenvalue approximation: In appendix B the authors discuss approximating the eigenvalues of K and H, I think it is okay to develop numerically efficient approximations to the quantities in question but the presentation is not very intuitive nor rigorous. In the first paragraph the authors mention that since H is ill conditioned, a small subset of the training examples gives a good approximation to the Hessian, can the authors elaborate on this statement? \n\nOther more minor comments:\n\nEven though the default step size and batches are mentioned in appendix D it could be nice to mention them in the figures that they are presented in.\n\nPage 7 Fig 5b | The labels of the figures should be adjusted so that they do not overlap with the adjacent figures.\n\nMinor typos/grammar:\npage 1 caption of figure 1 | vertical line marks -> the vertical line marks \npage 3 in definitions paragraph | let us denote loss -> let us denote the loss\npage 3 3rd assumption | training trajectory -> the training trajectory\npage 3, 3rd assumption | escape region-> escape a region\npage 7, other experiments paragraph | depend -> depends,  possibly?\npage 7, Figure 5 (b) Middle |  lambda_k to lambda_k^1", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an interesting study analyzing learning trajectories and their dependence on hyperparameters, important for better understanding of learning in deep neural networks.  All reviewers agree that the paper has a useful message to the ICLR community, and appreciate changes made by the authors in response to the initial reviews.
Decision: Accept (Spotlight)