On the Mysterious Optimization Geometry of Deep Neural Networks
Authors: Chedi Morchdi
Yi Zhou
Jie Ding
Bei Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep learning
optimization geometry
nonconvex optimization

Abstract: Understanding why gradient-based algorithms are successful in practical deep learning optimization is a fundamental and long-standing problem. Most existing works promote the explanation that deep neural networks have smooth and amenable nonconvex optimization geometries. In this work, we argue that this may be an oversimplification of practical deep learning optimization by revealing a mysterious and complex optimization geometry of deep networks through extensive experiments. Specifically, we consistently observe two distinct geometric patterns in training various deep networks: a regular smooth geometry and a mysterious zigzag geometry, where gradients computed in adjacent iterations are extremely negatively correlated. Also, such a zigzag geometry exhibits a fractal structure in that it appears over a wide range of geometrical scales, implying that deep networks can be highly non-smooth in certain local parameter regions. Moreover, our results show that a substantial part of the training progress is achieved under such complex geometry. Therefore, the existing smoothness-based explanations do not fully match the practice. 

Under review as a conference paper at ICLR 2023
ON THE MYSTERIOUS OPTIMIZATION GEOMETRY OF DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Understanding why gradient-based algorithms are successful in practical deep learning optimization is a fundamental and long-standing problem. Most exist- ing works promote the explanation that deep neural networks have smooth and amenable nonconvex optimization geometries. In this work, we argue that this may be an oversimplification of practical deep learning optimization by reveal- ing a mysterious and complex optimization geometry of deep networks through extensive experiments. Specifically, we consistently observe two distinct geomet- ric patterns in training various deep networks: regular smooth geometry and a mysterious zigzag geometry, where gradients computed in adjacent iterations are extremely negatively correlated. Also, such a zigzag geometry exhibits a fractal structure in that it appears over wide range of geometrical scales, implying that deep networks can be highly non- non-smooth in certain local parameter regions. More- over, our results show that substantial part of the training progress is achieved under such complex geometry. Therefore, the existing smoothness-bast -baed expla- nations do not fully match the practice.
INTRODUCTION
Training simple neural networks is known be an NP-complete problem (Blum & Rivest, 1988). However, in modern machine learning, training deep neural networks turns out to be incredibly easy in that many simple gradient-based optimization algorithms can consistently achieve low loss (Rob- bins, 2007; Kingma & Ba, 2015; Duchi et al., 2011). Such an observation inspires researchers to think that there might be specific simple structures deep neural networks that make nonconvex op- timization easy and tractable. Many works have been developed in the past decade to seek justifiable explanations, either theoretically or empirically.
Specifically. from theoretical perspective, many nonconvex optimization theories have been devel- oped to explain the success of deep learning optimization. The key idea to prove that deep neural networks have certain nice geometries that guarantee convergence the global minimum in convex optimization. For example, many types of deep neural networks such as werrpparaeterrzzed residual networks (Zhang works (Allen-Zhu et al., 2019a), nonlinear networks (Zou et al., 2020; Zhou et al., 2016), and lin- ear networks (Frei & Gu, 2021: Zhou Liang, 2017) have been shown satisfy the so-called gradient dominant geometry (Karimi et al., 2016). On the other hand, shallow ReLU networks (Soltanolkotabi, 2017; Zhong al., 2017; al., 2019; Du al., 2019b), deep residual networks (Du et al., 2019a), and some nonlinear networks (Mei et al., 201 8; Du & Lee, 201 8) have been shown to satisfy the local strong convexity geometry. Both geometry types guarantee the convergence of gradient-based algorithms to global minimum at linear rate.
al., 2019; Allen-Zhu et al., 2019b; Du et al., 2019a), recurrent net-
From an empirical perspective, researchers have found skip connections and batch normalization of deep networks can substantially improve the smoothness of the optimization geometry (Li et al., 201 Sa; Santurkar et al., 2018; Zhou et al., 2019). Furthermore, some other works found that there is continuous low-loss path between the minima of deep networks (Verpoort al., 2020; Draxler et al., 2018). In particular, it is observed that a simple linear interpolation between the initialization point and global optimum encounters significant barrier for many deep networks (Goodfellow

---

Under review as a conference paper at ICLR 2023
et al., 2015). Moreover, many networks have been shown possess wide and flat minima that tend to generalize well (Hao et al., 2019; Mulayoff & Michaeli, 2020).
Despite the comprchensiveness of these existing works, they all aim to promote nice geometries to demystify the success of deep learning optimization. While this is an important step toward under- standing deep learning, sometimes the results and conclusions empirical evidence may not necessarily reflect the underlying challenge of optimization in practi- cal deep learning. The success of deep learning optimization may be due complicated unknown mechanisms oversimplified by existing works. This constitutes the goal of this paper - to investi- gate the optimization geometry of deep networks and reveal its complex and mysterious geometric patterns that may challenge the existing perception of deep learning optimization.
be illusional: the theories and
1.1 OUR CONTRIBUTIONS
We apply the full batch gradient descent train various popular deep networks on different datasets and study the geometry along the optimization trajectory via gradient correlation- based metrics (de- fined in Section 2). Specifically, we observe the following distinct geometric patterns.
We consistently observe two distinct geometric patterns in all the experiments: (i) smooth geom- etry where gradients computed in adjacent iterations are highly positively correlated (in terms of the cosine similarity defined in eq. (2)) and point toward similar directions, and (ii) mysterious zigzag geometry where gradients computed in adjacent iterations and point toward opposite directions. Interestingly, we find that for convolutional networks, the training starts with the smooth geometry and transfers to the zigzag geometry later on. On the contrary, the training of residual networks starts with the zigzag geometry and transfers to the smooth geometry afterward. Morcover, a substantial part of the training loss decrease is attained under the complex zigzag geometry in all of the experiments.
highly negatively correlated
We further investigate the mysterious zigzag geometry of deep networks and find that has a complex fractal structure. Specifically, when we zoom into the local geometry by training the deep networks with very small learning rates, we still observe the same zigzag geometry. It shows that the local geometry of deep networks can be highly non-smooth in a wide range of geomet- rical scales. Moreover, the zigzag geometric pattern tends to be stronger when we zoom into a smaller geometrical scale. These observations challenge the existing explanations of deep learning optimization based on smooth-type geometries.
Based on the local statistics of mean gradient correlation, we propose a low-cost geomelry-adapted warm-up learning rate scheduling scheme for large-batch training of residual networks. We show that it leads to comparable convergence speed and test performance to those of the original heuris- tic version with parameter fine-tuning.
2 PRELIMINARIES ON GRADIENT CORRELATION
To understand the optimization geometry in deep learning, we propose investigating the gradients along the optimization trajectory generated by full-batch gradient descent. We consider full-batch gradient descent it is noiseless and reflects the exact underlying gradient geometry of the objective function. Specifically, given a set of training samples {I,, y4 ) where 2, denotes the data and y denotes the corresponding label, the training objective function and the full-batch gradient descent (GD) update at each step (k: = 0, 1,...) are written as follows.
1 (Objective function): (0) = 72 Σ ((ho(z)). v1). (GD): 0x+1 = 0y V (I)
where he denotes the neural network model parameterized by 0, V is the gradient operator with respect to the parameter 0, 1j is the learning rate, and l is the loss function. We will con- sider classification tasks with the cross-entropy loss in this paper. In the training, we collect set of gradients generated along the optimization trajectory of full-batch gradient descent, i.e., (fo). (f.),,. LL, dates and help understand the local optimization geometry of the nonconvex objective function. To provide a quantitative understanding, investigate the following pairwise gradient correlation of
These gradients determine the direction of model up-
2

---

Under review as a conference paper at ICLR 2023
any pair of gradients (V,,( (Ok)) generated in the training:
(VLn (0₂), VLn (Oa))
(Pairwise gradient correlation): ((j,h)) (0))l (V
(2) E e--1, Vj.k
where (-) and || denote the inner product and the m2-norm of vectors, respectively. At every epoch k; we fix window size and compute the pairwise gradient correlations for the set gradients (fx)). ........ V,,,, (0k-4-11) 1)) These gradient correlations form an h x h matrix, which illustrates the local geometry. We choose h for visualization purposes only. Moreover, at every epoch k we also track the following mean gradient correlation P& over adjacent epochs within the window.
h-1
(Mean gradient correlation): T : h-
Σ
u(k + m 1, m)), A: = 1,
(3)
m=
We use h = in eq. (3). The choice of h. is not essential as it only affects the smoothness of the mean gradient correlation curves.
MYSTERIOUS OPTIMIZATION GEOMETRY DEEP LEARNING
In this section, we train various modern deep networks and track the gradient correlation statistics introduced Section From these statistics, we consistently observe a mysterious and complex zigzag geometric pattern in practical deep learning optimization.
3.1 TRAINING CONVOLUTIONAL NETWORKS
Simple CNN. We first train a simple feed-forward CNN with three convolution blocks and one fully- connected block the CIFAR- 10 dataset (Krizhevsky, 2009) using full-batch gradient descent with learning rate 7) = 0.1 for 1500 epochs. We track the training loss, mean gradient correlation (defined in eq. (3)) and pairwise gradient correlation matrix throughout the training process. The results shown in Figure 1. It can be seen that the training loss decreases to almost zero after 1000 epochs. On the other hand, the mean gradient correlation is highly positive for the first 10 epochs and drops to highly negative values after that. This shows that the gradients computed in adjacent epochs point toward the same direction only the beginning of training and point toward opposite directions later implying transition from smooth geometry to highly nonconvex geometry.
CNN CIFAR10
1.0
mean correlation
2.0
1.5
0.0
-0.5
0.0
500
1000
1500
500
1000
1500
number epochs
number
epochs
Enoch5
Epoch675
Epoch1490
0.94
0.89
0.62
0.75
0.91
0:85
-0.73
0.58
041
00003
0.086
40008
3.94
0,5
0.55
-0.91
.0.92
0.55
.0.21
0.041
0.011
0.0.9
0.11
0.95
6:91
60992
0.84
.0.023
011
0.052
0.019
Figure 1: CNN training using gradient descent CIFAR-10. Top row: training loss (left) and mean gradient correlation (right). Bottom row: pairwise gradient correlation matrices with window size h computed at epochs 5. 675, and 1490.
Moreover, the pairwise gradient correlation matrices reveal some interesting and mysterious geomet- ric patterns in training. Due to space limitation, present three representative pairwise gradient correlation matrices computed at epochs 5, 675, and 1490 (corresponds to the red dots in the training loss figure). We have the following observations from these matrices.
3

---

Under review as a conference paper at ICLR 2023
Epochs 1-10: smooth geometry. In the first 10 epochs, the observed pairwise gradient correla- tion matrices are uniformly highly positive, as illustrated by the matrix computed at epoch 5 in Figure 1. This shows that in the initial phase of the training. the gradients computed within small window of epochs are pointing in almost the same direction. Therefore, the optimization geometry of CNN is smooth in the initial training phase.
Epuchs 10-700: zigzag geometry. After 10 epochs, the algorithm's dynamic encounters a mys- terious zigzag geometry, as illustrated by the pairwise gradient correlation matrix computed epoch 675 in Figure 1. Specifically, the entries along the first off-diagonal line take negative values {-0.91, 0.92. - 0.92, -0.9} that very close to -1. This implies that the gradients computed in the adjacent epochs (e.g., epochs 675 and 676) within this window are pointing in al- most opposite directions. Also, the entries along the second off-diagonal line take positive values (0.86, 0.85, 0.84] that are close to 1. This implies that the gradients computed in the epochs with a lag order of two (e.g., epochs 675 and 677, epochs 676 and 678) within this window are pointing in similar directions. Furthermore, the entries along the third off-diagonal line become highly negative again, implying that the gradients computed the epochs with a lag order of three (e.g., epochs 675 and 678) within this window pointing toward opposite directions. These observa- tions show that the optimization dynamics encounter a zigzag-type structure, and the optimization geometry of CNN is highly nonconvex and zigzag in the middle of the training.
Epochs 700-1500: random geometry. When the training converges and achieves near zero loss after 1000 epochs, the pairwise gradient correlations consistently close zero, as illustrated by the pairwise gradient correlation matrix computed at epoch 1490 in Figure This implies that the geometry of CNN is, to extent, random around the global minimum.
VGG-16. We also explore the gradient correlation in training a VGG-16 16 network on the CIFAR-10 dataset using full-batch gradient descent with learning rate 1j 0.1 for 1500 epochs. We observe similar optimization geometry to that of CNN. The results are shown in Figure 2. The mean gradient correlation highly positive for most of the first 500 epochs. This is consistent with the pairwise gradient correlation matrix computed at epoch 5 shown in the figure, whose entries uniformly highly positive. This shows that the gradients computed in the initial training phase are pointing in almost the same direction, therefore implying smooth optimization geometry. However, in the later training phase after 500 epochs, the mean gradient correlation drops to highly negative values, indicating an occurrence of highly nonconvex geometry. In particular, we observe the zigzag geometric pattern in the pairwise gradient correlation matrix computed at epoch 915.
VGG16 CIFAR10
1.0
mean correlation μ
0.5
2.00
0.0
1.50
-0.5
500 number of epochs
1000
1500
500
1000
1500
number of epochs
Fnoch5
Fpoch9115
Epach1 480
0.79
-0.04
0:94
-0.9
0.72
.0020
Q.B
0.79
0.70
-0.91
-0
0.26
-0.31
-C.42
40.30
8.5
0.79
0.5
0.79
0.94
0.99
-0.96
0.87
0:14
00.5
-0.59
0.32
0.5
C.79
0.8
0.8
-0.9
0.08
.0.95
-09
-0.029
0.22
.2.99
-0.15
3.79
0.70
7.79
0.8
5.72
0.81
0.87
-0.9
0.12
-0.28
0:32
-0.58
Figure VGG-16 training using gradient descent on CIFAR-
Other results and conclusion. In Appendix A, we include additional results on training the CNN and VGG-16 networks using other datasets such as MNIST (Deng, 2012) and SVHN (Netzer al., 2011). From all these results, we observe a common phenomenon: the optimization geometry of CNN and VGG-16 has sharp transition from smooth geometry to zigzag geometry in the training.
4

---

Under review as a conference paper at ICLR 2023
3.2 TRAINING RESIDUAL NETWORKS
We further train various residual networks and track the gradient correlation along the gradient descent trajectory. Interestingly, also observe the zigzag geometry in training residual networks, but the overall transition of geometry is very different from that of convolutional networks.
ResNet-18. We train standard ResNet-18 on the CIFAR-10 dataset using full-batch gradient de- scent with learning rate 77 0.1 for 500 epochs, and track the training loss and gradient correlation. The results are shown in Figure from which one can see very different transition of optimization geometry compared to that of convolutional networks. Specifically, one can see that the mean gra- dient correlation highly negative the first 120 epochs. This shows that the gradients computed in adjacent epochs are negatively correlated in the initial training phase, implying highly noncon- vex geometry. Moreover, from the pairwise correlation matrices computed at epochs and 60, one can see that the local geometry gradually transfers to the zigzag geometry. After 150 epochs, the mean gradient correlation increases to highly positive values, implying a very smooth optimization geometry when approaching the minimizer. This is also supported by the pairwise correlation ma- trix computed at epoch 490, whose entries are uniformly close to 1. To summarize, the optimization geometry of ResNet-18 18 has sharp transition from nonconvex zigzag geometry smooth geometry- This is very different from the optimization geometry of convolutional networks, which starts with relatively smooth geometry and transfers zigzag geometry in the later training.
RESNET18 CIFAR10
mean correlation μ
2
0.0
-0.5
100
200
300
400
500
100
200
300
400
500
number epochs
number of epochs
Fpochs
Fpoch60
Fooch490
1
30.54
2.54
.0.4
0.073
:2.09
9.77
-0.69
0.6
0.04
0:9g
0.55
0.98
0.54
1
0.71
0.14
0.15
0.89
+0.87
0.77
0.67
3.54
1
0.94
0.59
0.95
0:54
.0.1
-0,003
0.77
-3.07
00.7
D:75
0.44
0.94
0.59
4.34
0.1
-0.41
0.09
+.87
-0.05
9.95
0:99
0:51
0:15
Figure 3: ResNet- 18 training using gradient descent on CIFAR-10
ResNet-34. We further train a standard ResNet-34 on the CIFAR-10 dataset using full-batch gradient descent with learning rate 7 0.1 for 500 epochs. The results are shown in Figure 4, where one can observe a similar transition of geometry to that of ResNet-18.
RESNET34 CIFAR10
mean correlation
0.5
Train loss
-0.5
100
200
300
400
500
100
200
300
400
500
number of epochs
number of epochs
EpachS
Fpoch85
Fpoch49p
3.49
0.11
0.019
-0.36
0:2
0.91
b.
5.91
-0,19
-0.47
055
-2.76
8.6
-0-13
B:1
3:2
0.57
0.11
4025
.0.55
0.35
00.76
0.76
0.59
0.9
0:11
0.9L
0.9
0,019
0,055
-2.25
.3.19
40.36
2.6
-0.75
-0.75
2.9L
0.33
0:91
1
0.31
0.38
0.2
4.35
-0.99
0.31
.0.33
5.59
0.75
0.9
0:92
0.0
0.91
Figure 4: ResNet-34 training using gradient descent on CIFAR-10
5

---

Under review as a conference paper at ICLR 2023
Other results and conclusion. In Appendix B, we include additional results training residual networks using MNIST and SVHN. From all these results, we observe a common phenomenon: the optimization geometry of residual networks has sharp transition from nonconvex zigzag geometry to smooth geometry in training. On the other hand, in the existing literature, it has been visualized that the geometry of residual networks is smooth around the global minimum (Li et al., 201 8b; Garipov et al., 2018). This is consistent with our observation that the mean gradient correlation highly positive in the later training phase of residual networks. Our results further reveal that residual networks have complex zigzag geometry far away from the global minimum.
3.3 FRACTAL STRUCTURE OF ZIGZAG GEOMETRY
In the previous subsections, we observed the zigzag geometry in training various deep networks us- ing a large learning rate. Here, we further zoom into the local geometry by training these networks with much smaller learning rates, We aim to understand if the zigzag geometry has any local struc- ture. Figure 5 shows the training of VGG-16 on CIFAR-10 10 using full-batch gradient descent with a small learning rate 1 0.01. From the mean gradient correlation figure, it can be seen that the optimization geometry smooth in the first 900 epochs, as also confirmed by the pairwise gradient correlation matrix computed at epoch 5. However, after 900 epochs, the mean gradient correlation instantly drops to highly negative values, and we start to observe the zig7ag geometry as shown by the pairwise gradient correlation matrices computed at epochs 925 and 1490.
VGG16 CIFAR10
mean correlation
2.2
0.5
Train loss
2.0
-0.5
1.8
-1.0
500 number of epochs
1000
1500
500
1000
1500
number of epochs
Eparh5
Epach925
Epoch1490
0:70
0.6
0.79
-0.37
0.95
-0.38
0:95
0.36
0.79
5.78
0.79
0:28
0:97
-3
0.97
0.8
0:78
0.79
0:79
0.97
-0.95
0.96
0.95
0.59
0.95
1
-..99
0.97
6.79
0.70
0:70
.0.07
D.90
-0.98
-0.95
-0.99
C8T
.0.99
-0.09
6.79
78
3.79
0.79
6:96
:3.98
5,98
0.98
2:96
0.508
0.57
-2.99
Figure 5: VGG- 16 training using gradient descent with 1) = 0.01 on CIFAR-10
Figure 6 further shows the training of ResNet-18 18 on CIFAR-10 using full-batch gradient descent with a very small learning rate 1 0.001. It can be seen that the optimization geometry is very smooth for the initial 50 epochs. But it starts to transfer to zigzag geometry after 400 epochs, as can be seen from the pairwise gradient correlation matrices computed at epochs 490 and 990.
RESNET18 CIFAR10
2.50
mean correlation
0.5
2.25
2.00
0.0
1.75
-0.5
1.50
1.25
-1.0
200
400
600
800
1000
200
400
600
800
1000
number of epochs
number of epochs
Fparh5
Fpoch490
Epochoop
0.97
5.90
0.56
-0.92
D.94
-0.09
0:90
0.98
0:67
3.57
0.57
-2.95
0.50
-3
-8
0:36
0.9
1
0.98
9.9
0.97
.0.95
-0995
0.95
0.59
40,99
1
4.999
0.99
6.28
0.97
90
9
.0.92
9:95
-.99
-0.95
-2.99
0.09
-0.99
-0.09
6.55
0 90
3.07
0.57
6.54
3.93
5.58
0.95
1
2:94
0.59
0.09
-9.99
Figure 6: ResNet-18 training using gradient descent with 1r=0.001 on CIFAR-10

---

Under review as a conference paper at ICLR 2023
Other results and conclusion. In Appendix C, include additional results training other models on CIFAR-10 10 using very small learning rates, and one can make very similar observations (namely, Zigzag geometry still appears). From all these results, it seems that the zigzag geometry of deep networks has a complex fractal structure in that it appears within a wide range of geomet- rical scales. It indicates that deep networks can be highly non-smooth and non-Lipschitz in local region of the parameter space. More interestingly, note that the variation of the entries of the zigzag pairwise correlation matrices in Figures 5 and are much larger than those of the zigzag pairwise correlation matrices in Figures 2 and 3. This implies that the zigzag geometry tends to be more nonconvex when zoomed into smaller geometrical scale.
DISCUSSIONS
4.1
DEEP LEARNING OPTIMIZATION THEORY
The observations made in Section 3 imply that many deep neural networks belong to a small class of functions with very special and nonconvex geometry, which a mixture of smooth geometry and complex zigzag geometry. In particular, the gradients under the zigzag geometry constantly shift their directions at different geometrical scales. It implies a highly non-smooth local geometry: the local Lipschitz constants of the gradients are ill-conditioned and change rapidly.
On the other hand, this poses challenge to the existing developments of deep learning optimiza- tion theory, which are often based on proving smooth and Lipschitz-type geometries of deep neural networks that guarantee convergence to a global minimum. For example, smoothness and gradient dominant geometry (also known as volyak-Lojasiewicz geometry) constitute classic tools for estab- lishing global convergence of gradient-type algorithmss in nonconvex optimization (Karimi et al., 2016; Polyak, 1963). Specifically, the gradient dominant geometry requires the optimality gap of the objective function to be bounded by the corresponding gradient norm, i.e.
(Gradient dominant): L. (0) - L. (0") S CIlV
4O E
where C' C'>0 0s universal constant, θ is local neighborhood of the minimizer 0-, and ||-I| denotes the f--norm. This nonconvex geometry has a close connection to other (non)convex geometries, including strong convexity (Nesterov, 2014), weak strong convexity (Necoara al., 2019), and error bound (Luo & Tseng, 1993), all of which guarantee linear convergence of many gradient-based algorithms. In particular, many deep networks have been theoretically proved to have smooth and gradient dominant geometry. including over over-paramen parameterized residual networks (Zhang et al., 2019; Allen-Zhu et al., 2019b; Du et al.. 2019a), some nonlinear networks (Zhou et al., 2016; Zou al., 2020) and deep linear networks (Zhou & Liang, 2017; Frei & Gu, 2021). However, this is coarse geometry model in practical deep learning, as it only involves the gradient norm information and does not capture the highly non-smooth zigzag geometry observed in our experiments. Therefore, we think the existing deep learning optimization theory cannot fully explain the success in practice, and advanced theories are needed to explain (i) why neural networks have such complex and non- smooth zigzag geometry and (ii) why gradient descent can overcome such non smo-th geometry and decrease the training loss in practice.
4.2 JUSTIFICATION OF LEARNING RATE SCHEDULING
Our previous experimental observations provide justifications for using learning rate scheduling in large-batch training of residual networks. Specifically, it is known that training resid- ual networks with large batch size can easily diverge when a large learning rate is used in the initial training phase. To sta- bilize training, researchers proposed a warm-up learning rate scheduling scheme for latge-batch training of residual net- works (Goyal et al., 2017). The idea to start with a very small learning rate and gradually increase to large learning rate over several epochs. After that, the learning rate slowly decreases until the training is saturated (see the green curve in Figure 7). Our observation of mean gradient correlation in training residual networks justifies the effectiveness of this scheduling scheme. Specifically, illustrated in Figure 7, the mean gradient
0.015
0.5
0.010
0.005
-0.5
coreeatoraatioon
schedddee..
0.000
100
200
400
500
Figure 7: Red: mean gradient corre- lation of ResNet- 18 training. Green: warm-up learning rate scheduling.
7

---

Under review as a conference paper at ICLR 2023
correlation in ResNet-18 training is highly negative in the initial training phase, which implies highly nonconvex optimization geometry. Therefore, it is desirable to start with a small learning rate. After that, the gradient correlation gradually increases to highly positive values, and the ge- ometry becomes very smooth. Therefore, a warm-up learning rate scheduling is appropriate. In Section 5, propose adapt the warm-up scheduling scheme to the mean gradient correlation to avoid parameter tuning in practice.
4.3 SIGN SHIFT OF GRADIENT ENTRIES
One interesting question is how many gradient entries shift their sign when encountering the zigzag optimization geometry. In Figure 8, we show the pairwise gradient correlation matrix when encoun- tering the zigzag geometry in training convolutional networks. For each entry of pairwise gradient correlation, we calculate the percentage of the number of gradient entries with opposite signs. We can see that when encountering the zigzag geometry in CNN training, the gradient correlations of adjacent iterations stay around -0.92 -0.9, and about 90% of the gradient entries have opposite signs. This shows that such a complex geometry is across the majority of the parameter dimensions of convolutional networks. In Appendix D, include additional results sign shift of gradient entries in training residual networks, where one can make similar observations.
cormatiion
of shit
LL1
C.11
4.73
3:35
0.02
45
3.31
3:2
8.7
29
4.03
3:25
0.90
.303
3.99
8.2
C42
3:1
9:20
3:54
8.99
(a) Epoch 675 of CNN training CIFAR-10
(b) Epoch 705 of VGG-16 training SVHN
Figure 8: Gradient sign shift in training convolutional networks
5 APPLICATION: GEOMETRY- LEARNING RATE SCHEDULING
The statistics of gradient correlation sketch the global optimization geometry of deep networks. In this section, we further leverage this statistics to develop geometry-adapter learning rate schedul- ing scheme for large-batch training of residual networks. We focus on large-batch training for two reasons: (i) it has become increasingly important and popular for large-scale training with parallel computation (Goyal et al., 2017; You et al., 2018; 2019), and (ii) the noisy stochastic gradient cor- relation still reflect the underlying optimization geometry in the large-batch setting, as show in the following subsection.
5.1 IMPACT OF BATCH SIZE ON GRADIENT CORRELATION
Figure 9 shows the mean gradient correlation training a ResNet-18 with different large-batch sizes on CIFAR-10. In all the training, we linearly scale the learning rate based on the batch size. can be seen that over wide range of large-batch sizes, the mean gradient correlation curves do not change substantially. This implies that mean gradient correlation is a statistic that can reliably reflect the underlying optimization geometry in large-batch training. Please refer to Appendix E for more results training other models, where one can make similar observations.
mean correlation
0.0
mean jorrelation
0.0
uorrelation
0.0
-0.2
-0.2
-0.2
-0.4
-0.6
-0.8
-0.8
0
200
300
400
500
100
200
200
300
400
500
number of epochs
number of epochs
number epochs
(a) Batch size 5000
(b) Batch size 10000
(c) Batch size 20000
Figure 9: Gradient correlation in ResNet-18 training on CIFAR- 10 under different batch sizes
8

---

Under review as a conference paper at ICLR 2023
5.2 CHOMETRY-ADAPTER WARM-UP LEARNING RATE SCHEDULING
In training residual networks, the standard warm-up (WU) learning rate scheduling scheme consists of two phases, and the phase change iteration & E N is heuristically chosen. the warm-up phase (before f-th iteration), the scheme starts from a small learning rate To and increases it exponentially as 7i6 = (xp(wk)))-- for some scheme decays the learning rate via certain standard schemes (e.g., cosine annealing) until the end of the training. Here, we propose a Reometry-Adapter Warm-Up (GAWU) scheme that adaptively chooses the phase change point according to the mean gradient correlation statistics. Specifically, we keep tracking the mean gradient correlation in training and set the phase change point as the first time that the mean gradient correlation drops from positive negative values, indicating transition from smooth to non-smooth geometry. Furthermore, computing the mean gradient correlation is cheap in each iteration. It only requires storing an extra gradient calculated in the previous iteration and computing its correlation with the current gradient.
w > O. In the decay phase (after 1-th iteration), the
We compare the proposed GAWU scheme with the heuristic WU scheme in training Resnet- 18 on the CIFAR-10 dataset. We use SGD as the optimizer and set the batch size to be 5000. For the baseline WU scheme, we consider three heuristic change point settings f 5, 10, 12, where 12 is the best change point that leads to the highest test accuracy that we found by performing grid search over t. For both GAWU and WU, their warm-up phases use the same hyper- syper-parameter s 0.1, and their decay phases use the standard cosine annealing scheme with one period. We set the initial learning rate as 7jo 0.001 for all the experiments.
10
GHNU
0.6
LR schedule
Test Acc (Loss)
Train loss
WU (t = 5)
61% 61%((108)
0.4
WU (1 10)
74%(1.63
LR
0.2
WU (L = 12)
(1.9).
GAWU
76%((1.69)
0
200
400
600
800
1000
400
600
800
number of epochs
number apochs
Figure 10: Comparison between GAWU and WU in training ResNet- 18 on CIFAR-10.
Table 1: Test accuracy of the final model trained by GAWU and WU.
Figure 10 shows the training loss and learning rate scheduling produced by GAWU and WU. It can be seen that the training loss obtained under the GAWU scheme has a comparable convergence speed to that of the training loss obtained under the WU scheme with In fact, in our experiment, the change point detected by GAWU is t = 11, which is very close 12 found empirically. This demonstrates the advantage of adapting the
parameter t = 12.
to the best parameter learning rate to the change of the underlying optimization geometry. Moreover, the test accuracy results presented in Table further confirms the effectiveness of the GAWU scheme.
6 CONCLUSION
In this paper, we reveal that many popular deep networks have mysterious zigzag optimization geom- etry, which has complex fractal structure over wide range of geometrical scales. These observa- tions imply that deep learning optimization cannot be fully characterized by the classic optimization theories, which crucially rely on elegant and smooth type geometries. As future direction, we are motivated to develop a mathematical formalization of the observed zigzag geometry and develop advanced optimization theory to understand deep learning optimization in practice.
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song, On the convergence rate of training recurrent neural networks. In Proc. International Conference on Neural Information Processing Systems, 201 9a.
Zeyuan Allen-Zhu Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In Proc. International Conference on Machine Learning, volume 97, pp. 242- 252, 2019b.

---

Under review as a conference paper at ICLR 2023
Avrim Blum and Ronald Rivest. Training 3-node neural network is np-complete. In Proc. Advances in Neural Information Processing Systems, volume 1, 1988.
Li Deng. The mnist database of handwritten digit images for machine learning research. JEEE Signal Processing Magazine, 29(6):141142,, 2012.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In Proc. International Conference on Machine Learning. volume 80, pp. 1309-1318, 10-15 Jul 2018.
Simon Du and Jason Lee. On the power of over-pane parametrization in neural networks with quadratic activation. In Proc. International Conference on Machine Learning, volume 80, pp. 1 Jul 2018.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. Proc. International Conference on Machine Learning. vol- ume 97, pp. 1675-1685, Jun 2019a.
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In Proc. International Conference on Learning Representa- tions, 2019b.
John Duchi, Flad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159, jul 2011.
Spencer Frei and Quanquan Gu. Proxy convexity: A unified framework for the analysis of neural networks trained by gradient descent. In Advances in Neural Information Processing Systems, volume 34, pp. 7937-7949, 2021.
Haoyu Fu, Yuejie Chi, and Yingbin Liang. Local geometry of cross entropy loss in learning hidden-layer neural networks. In Proc. IEEE International Symposium on Information Theory, pp. 1972-1976, 2019.
Timur Garipov, Pavel Izmailov, Dmitri Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Proc. Advances in Neural Informa- fion Processing Systems, volume 31, 2018
Ian Goodfellow, Oriol Vinyals, and Andrew Saxe. Qualitatively characterizing neural network opti- mization problems. In Proc. International Conference on Learning Representations, 2015.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv: 0260770.2677.... 2017.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Visualizing and understanding the effectiveness of BERT. In Proc. Conference on Empirical Methods in Natural Language P Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4143-4152, November 2019.
Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal- gradient methods under the polyak- condition. In Proc. Machine Learning and Knowl- edge Discovery in Databases, pp. 795-811, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Porc. Interna- lional Conference on Learning Representations, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Proc. Advances in Neural Information Processing Systems, volume 31, 2018a.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. In Proc. Advances in Neural Information Processing Systems, volume 31, 2018b.
10

---

Under review as a conference paper at ICLR 2023
Zhi Quan Luo and Paul Tseng. Error bounds and convergence analysis of feasible descent methods: a general approach. Annals of Operations Research, 46 46(1):157-178, 1993.
Song Mei, Yu Bai, and Andrea Montanari. The landscape of empirical risk for nonconvex losses. The Annals of Statistics, 46(6A):2747-2744 2018.
Rotem Mulayoff and Tomer Michaeli. Unique properties of flat minima in deep networks. In Proc. International Conference Machine Learning, volume 1 19, pp. 7108-7118, 13-18 Jul 2020.
I. Necoara, Yu. Nesterov, and F. Glineur. Linear convergence of first order methods for non-strongly convex optimization. Mathematical Programming, 179 2019.
Yurii Nesterov. Introductory Lectures on Convex Optimization: A Basic Course. Springer, 2014.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
B.T. Polyak. Gradient methods for the minimisation of functionals. USSR Computational Mathe- matics and Mathematical Physics, 3(4):864-878, 1963.
Herbert E. Robbins. A stochastic approximation method. Annals of Mathematical Statistics, 22: 400-407, 2007.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch nor- malization help optimization? In Proc. Advances in Neural Information Processing Systems, volume 31, 2018.
Mahdi Soltanolkotabi. Learning relus via gradient descent. Proc. Advances Neural Information Processing Systems, volume 30, 2017.
Philipp C. Verpoort, Alpha A. Lee, and David J. Wales. Archetypal landscapes for deep neural networks. Proc. National Academy of Sciences, 117(36):21857-21866 2020.
Yang You, Zhao Zhang, Cho-Jui Hsieh, James Demmel, and Kurt Keutzer. Imagenet training in minutes. In Proc. International Conference on Parallel Processing. 2018
Yang You, Jonathan Hseu, Chris Ying, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large- batch training for Istm and beyond. In Proc. International Conference for High Performance Computing, Networking, Storage and Analysis, 2019.
Huishuai Zhang, Da Yu, Mingyang Yi, Wei Chen, and Tie- Yan Liu. Convergence theory of learning over-parameterizee resnet: A full characterization. arXiv: (eXxx:9903.07/2 2019.
Kai Zhong, Zhao Song, Prateek Jain, Peter L. Bartlett, and Inderjit S. Dhillon. Recovery guarantees for one-hidden-lan layer neural networks. In Proc. International Conference on Machine Learning, volume 70, pp. 41 4140-4149, Aug 2017.
Y. Zhou and Liang. Characterization of gradient dominance and regularity conditions for neural networks. ArXiv:1110.0691002 Oct 2017.
Y. Zhou, H. Zhang, and Y. Liang. Geometrical properties and accelerated gradient solvers of non- convex phase retrieval. Annual Allerion Conference on Communication, Control, and Computing, pp. 331-335, 2016.
Yi Zhou, Junjie Yang, Huishuai Zhang, Yingbin Liang, and Vahid Tarokh. SGD converges to global minimum in deep learning via star-convex path. In Proc. International Conference on Learning Representations, 2019.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over- parameterized deep relu networks. Machine Learning, 109(3):467-492. 2020.
11


Reviews:
[{"review_id": "aiL5J5z5E97", "reviewer": "Reviewer_ynbp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the optimization landscape of deep neural networks through tracking the  correlation of adjacent gradient steps of full batch gradient descent. Surprisingly, the authors find a strong zigzag behaviour, where adjacent gradients consistently have strong negative correlation values for a large period of training. Where this long stretch of zigzag behaviour occurs seems model dependent, the authors find that small CNNs and ResNets exhibit it close to the start of training, while VGGs show it towards the end of training. The zigzag behaviour indicates a difficult loss landscape (e.g. highly non-convex), which is in contrast to what prior work on gradient dominance and local strong convexity have identified. Using those insights, the authors develop a learning rate schedule for large batch training that takes the identified geometry into account. By tracking the mean correlation of gradients, the phase change point of the schedule is chosen to coincide with the time that the gradient correlation switches signs for the first time.", "summary_of_the_review": "The observed zigzag phenomenon is interesting but as outlined above I am currently not convinced of the empirical evidence presented in this work. Multiple runs (at least for a key-setting, e.g. ResNet18 + CIFAR10) would help convince me of the robustness of this phenomenon. The authors should also motivate the metric better, showcase how in convex settings, no negative correlation emerges and discuss why different learning rates can completely flip the correlation trajectory. In general, more interpretation of the results is needed, why does a VGG have a different correlation trajectory than ResNets, why does the optimizer make similar progress in the same amount of time in terms of loss both in the difficult part of the landscape (negative correlation) and the easier part. In its current form I recommend rejection of this work. If the authors can convince me of the validity of their metric and provide better interpretation of their results, I am happy to change my score.", "strength_and_weaknesses": "**Strengths**\n\n1. Understanding the loss landscapes of deep networks remains as an open and important problem. While theoretical works have made progress, it is often unclear how the underlying assumptions really reflect practical scenarios. Challenging these results through empirical experiments is very interesting and helpful for further understanding.\n\n2. The setup is very simple and easy to understand. The authors study a very simple measure to assess the geometry of the landscape in the form of gradient correlations. The result of strong negative correlation over long periods of training is very surprising.\n\n**Weaknesses**\n\n1. While the gradient correlation measure is simple, it is not clear to me how well it reflects the simplicity of a loss landscape. The authors should present more evidence (empirical and theoretical) as to why negative correlation reflects difficulty in optimization, especially since training does make progress in terms of loss reduction comparable to the positively correlated stage. How do the gradients correlate if a convex problem is considered, i.e. linear regression? Are there theoretical results for this? How does negative correlation contradict gradient dominance and local strong convexity?\n\n2. The shown figures for the correlation plots are extremely noisy (e.g. Fig. 1, Fig. 2, Fig 3.) which makes me assume that they are all based on a single run (I couldn’t find a statement indicating otherwise in the main text). More empirical evidence indicating that this is a robust phenomenon (i.e. averaging over multiple runs) would convince me more. While the authors argue that this geometry is robust w.r.t. learning rate, Fig. 3 and Fig. 6 seems to indicate differently. Here, a ResNet18 is trained with different learning rates, the large learning rate leading to an initial negative correlation while the smaller one to initial positive correlation. The large learning rate then transitions to positive correlation while the smaller one becomes negatively correlated. This seems to show that the metric is somewhat brittle and re-iterates my point for the need of multiple runs.\n\n3. The previous point again makes me unsure regarding how well gradient correlation serves as a measure of roughness of a landscape. How can the nature of the landscape change for the same dataset and same model? For large learning rates, the landscape is highly non-convex first and then becomes easier (according to correlation) while for small learning rates, the landscape is very easy first and then becomes more difficult. The small learning rate experiments are also not very representative of the full landscape since we are very far from convergence even after 1000 epochs of training. Would the landscape remain easy until convergence or again transition after more training?\n\n4. This work largely focuses on full batch gradient descent but how does this choice affect the obtained results, i.e. do we see any (negative) correlation if we were to use SGD instead and one would for instance consider the averaged gradient over one epoch? Fig. 9 shows some results for large batch sizes (smallest is 5000) but not for smaller, more practical sizes. Are you directly comparing gradients from mini-batches or averaging gradients first? The large number of epochs also seems untypical for the considered settings, is this due to full batch GD?\n\n5. Finally I could also not find any attempt for an explanation of the zigzag phenomenon by the authors. Why would different learning rates affect the correlation in such a strong way? Why does a VGG exhibit such a different correlation trajectory, compared to a ResNet? How is it possible that the optimizer manages to reduce the training loss so well, even though effectively, gradients are simply just changing signs? Especially since the loss does not seem to reduce faster in the “well-behaved” possitively correlated setting over the same amount of time (see e.g. Fig. 2, Fig. 3, Fig. 5), some explanation would be very helpful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U3rfJFJNxp", "reviewer": "Reviewer_mFYm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The goal of this paper is \"investigating the optimization geometry of deep networks\".\nIn particular, correlations between gradients in adjacent time steps is measured in computer vision tasks (mostly convnets on CIFAR).\nIt is observed that correlations during training have periods of positive values, implying that the gradient points at a similar direction in consecutive time steps, and periods of negative values, implying that the gradient points at a nearly opposite direction in consecutive time steps.\nA learning rate scheduling that depends on gradient correlation is proposed and preliminary results are shown.\n\n", "summary_of_the_review": "Most of the results of this paper are not surprising and can be interpreted in the light of simple models (e.g. quadratic loss) or more recent findings (e.g. https://arxiv.org/abs/2103.00065).\nThe section on correlation-dependent learning rate scheduling is interesting and novel, but shown results are very limited and unconvincing.\n ", "strength_and_weaknesses": "Strengths\n\n- The optimization geometry of deep learning is not well understood, and this paper provides a different perspective.\n\n- Correlation-dependent learning rate scheduling seems novel.\n\n\nWeaknesses\n\n- This work does not have a clear hypothesis to test. \nIt seems that the authors implicitly hypothesized that gradient directions should be consistent in subsequent time steps.\nInstead, they found that gradients may be anti-correlated and thought that this is surprising.\nHowever, there does not seem to be any fundamental reason to believe that gradients should be consistent.\n\n- In fact, the observation of anti-correlated gradients can be reproduced even in simple toy models.\nTake, for example, a quadratic loss with Hessian h, anti-correlated gradients would be observed when the learning rate l is 1/h < l < 2/h.\nIn high-dimensional scenarios, different directions in parameter space have different (eigen-)values of the Hessian, and for some (perhaps a large fraction) of them anti-correlation should be expected.\nMost importantly the Hessian changes during training, and in some circumstances it keeps increasing, so much that anti-correlations should be expected even if the learning rate is small.\nThis phenomenon is studied in detail in this paper: https://arxiv.org/abs/2103.00065, which may explain several of the author's observations. \n\n- The authors contrast anti-correlated gradients with \"smooth\" behavior.\nIn fact, they even state that \"the mean gradient correlation drops from positive to negative values, indicating a transition\nfrom smooth to non-smooth geometry.\"\nHowever, gradient anti-correlation does not imply non-smoothness of the loss function, as the simple quadratic example suggests. \n(I assume the usual definition of smoothness, for twice differentiable functions, that the Hessian is bounded.)\n\n- It is true that, if the anti-correlation of gradients would persist in the limit of zero learning rate, then it would imply non-smoothness of the loss.\nHowever, this limit is not shown in the paper, only a couple of values of the learning rate are considered, which is very far from proving any \"fractal structure\".\nEven if the Hessian increases during training, it is expected to remain bounded (see e.g. http://arxiv.org/abs/2204.11326).\n\n- I found that the most interesting part of the paper is about the correlation-dependent learning rate scheduling, section 5.\nHowever, very limited results are shown and it is very hard to draw any conclusions.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NQHwb5opuW-", "reviewer": "Reviewer_MfSA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how geometry of the loss landscape changes throughout training by studying the how gradient vectors of consecutive training steps correlate with each other: high correlations across timestep windows imply a smooth geometry while alternating gradient directions imply a non-convex, “zigzag” geometry. \n\nThe authors study how these correlations evolve by training different architectures (CNNs and ResNets) on the CIFAR-10 image classification dataset. Under the full-batch setting, they find that the geometry of the loss landscape has distinct phases throughout training, and that depend on the architecture used: \n- For CNN architectures it has (1) an initial smooth phase, where consecutive correlate with each other, followed by a (2) zig-zag phase where gradients alternate directions and, when saturating the training loss, ending in a (3) random geometry phase, where gradients don’t correlate with each other\n- For ResNet architectures, the loss landscape instead starts in (1) zig-zag geometry and evolving throughout training to a (2) smooth geometry, even after the learning plateaus.\n\nThe authors also claim to observe an interesting phenomena of **fractal** geometry, where the same evolution of geometry are observed a smaller geometric scale when training with smaller learning rates.\n\nThey also discuss how their observations might justify the success of learning rate schedulers, such as warmup, in networks with residual connections: The lower learning rates are help-up in mitigate the non-convex, zig-zag geometry these networks have early in their training. The authors then propose a learning rate schedules that explicitly leverages their notion of geometry to automatically determine how many steps to do warm-up with, based on the phase changes in geometry.", "summary_of_the_review": "This paper provides a novel analysis of the optimization geometry of neural networks using gradient correlation across consecutive steps. While the analysis is throughout and novel, and brings new insights about optimization landscape of neural networks, they only study optimization on the CIFAR-10 dataset, which makes me worry about the generality of their results.", "strength_and_weaknesses": "In terms of strenghts:\n- In CIFAR-10, their analysis is quite throughout, exploring variations in architecture and optimization regime.\n- Their proposed geometry-aware learning rate scheduler shows that their methodology and findings have potential to make optimization of neural networks more ammenable\n\nHowever my main concern is if these findings regarding the geometry of the loss landscape are “universal” or are instead a peculiarity of the setting used, and if they apply to real use cases of the models.\n    - My main worry is that the authors only train model CIFAR-10. I understand that CIFAR-10 makes it easier to study the full-batch case, but, given their findings (5.1), I think studying how the gradient correlations evolve in other, more realistic, image dataset (say ImageNet) under the stochastic gradient descent case would be important to understand if this happens in realistic cases. I think this make the paper much more impactful.\n    - The authors only study CNNs and CNNs with ResNet connections. In contrast, most of the current advances (to my knowledge) are done with transformer-like models. Given the stark differences in architectures, I am really curious to know if their methodology based on gradient correlation can be applied to transformer-like architectures and if they loss geometry evolves in similar manner.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject