Which Layer is Learning Faster? A Systematic Exploration of Layer-wise Convergence Rate for Deep Neural Networks
Authors: Yixiong Chen
Alan Yuille
Zongwei Zhou
Venue: ICLR 2023 poster (2023)
Keywords: Deep neural networks
Convergence rate

Abstract: The deeply hierarchical structures enable deep neural networks (DNNs) to fit extremely complex target functions. However, the complex interaction between layers also makes the learning process of a particular layer poorly understood. This work demonstrates that the shallower layers of DNNs tend to converge faster than the deeper layers. We call this phenomenon Layer Convergence Bias. We also uncover the fundamental reason behind this phenomenon: Flatter local minima of shallower layers make their gradients more stable and predictive, allowing for faster training. Another surprising result is that the shallower layers tend to learn the low-frequency components of the target function, while the deeper layers usually learn the high-frequency components. It is consistent with the recent discovery that DNNs learn lower frequency objects faster.

Published as a conference paper at ICLR 2023
WHICH LAYER IS LEARNING FASTER? A SYSTEMATIC EXPLORATION OF LAYER-WISE CONVERGENCE RATE FOR DEEP NEURAL NETWORKS
Yixiong Chen Alan Yuille*
Zongwei Zhou²
The Chinese University of Hong Kong- Shenzhen Johns Hopkins University yixiongchen seannnggheennnnkee link. www..edu.uc.......... on. on
ayuillele jhu. edu zzhou820 jh edu
ABSTRACT
The deeply hierarchical structures enable deep neural networks (DNNs) to fit ex- tremely complex target functions. However, the complex interaction between lay- ers also makes the learning process of a particular layer poorly understood. This work demonstrates that the shallower layers DNNs tend to converge faster than the deeper layers. We call this phenomenon Layer Convergence Bias. We also uncover the fundamental reason behind this phenomenon: Flatter local minima of shallower layers make their gradients more stable and predictive, allowing for faster training. Another surprising result is that the shallower layers tend to learn the low-frequency components of the target function, while the deeper layers usu- ally learn the high-frequency components. It is consistent with the recent discov- ery that DNNs learn lower frequency objects faster.
INTRODUCTION
Over the last decade, breakthrough progress has been made by deep neural networks (DNNs) on a wide range of complicated tasks computer vision (Krizhevsky et al. (017), natural language processing (Sutskever et al., 2014), speech recognition (Graves et al.] 2013), game playing (Silver ct al. 2016). and biomedical prediction (Jumper et al., 2021). Such progress hinged on a number of ad advances in hardware technology, dataset construction, and model architectural designs. Among them, the invention and application of very-deep network architectures play a decisive role.
Deepening the network is an effective way to empower its fitting ability. Extensive studies (Delalleau & Bengio 2011 Eldan & Shamir 2016 Lu et al., (017) compared the power between deeper and wider neural networks and showed that the polynomial growth of depth has similar effect the exponential growth of width. Therefore, modern DNNs (Simonyan & Zisserman, 2014 He et al., 2016) usually contain tens of layers ensure their modeling abilities for real-world applications.
Although the practical success of deep architectures is indisputable, they make the learning hardly predictable since complex interaction happens between layers when co-adapting the target (Yosin- ski et al. 2014). By now, we still have a poor understanding of how different layers learn differ- ently. Currently, a widely accepted view relates the vanishing gradient Hochreiter et al.] (2001). The gradients are getting weaker and weaker they move back through the hidden layers, making the shallower layers converge more slowly (Nielsen 2015). Informally, reasonable that larger gradient values bring higher learning speed.
1991);
Even though this view somewhat makes sense, we seem to have little concrete evidence supporting it. In particular, it is dubious how higher-level features can be built based on the unstable features extracted by the unconverged shallower layers (Raghu et al. (017)). This paper aims to find a credible answer for the parameters of which layer are learning faster towards the convergence point (defined as the convergence rate in this work) with a systematic exploration. Our results lead to somewhat startling discoveries.
Our Contributions. Our point start is illustrating that there does not seem to be a reliable positive correlation between the gradient magnitude and the convergence rate of particular layer.

---

Published as a conference paper at ICLR 2023
Instead, we find that shallower layers tend to converge faster than the deeper ones, with smaller gradients. The phenomenon is called layer convergence bias in this paper.
We then turn our attention to excavating the underlying mechanism for the faster convergence of shallower layers. Specifically, we find out that the depth of a layer has a fundamental effect on its training: the parameters of shallower layers layers. This finding reveals that the gradients of shallower layers may be more predictive and thus have the potential allow the larger learning rates (LRs) to be performed, making the convergence faster.
usually optimized on flatter landscapes than deeper
Finally, we find that the layer convergence bias is also tied to the frequency of the function they modeling. When fitting a complex target function, the shallower layers tend to fit the cow-frequency (usually simpler) components. On the contrary, the deeper layers struggle fit the remaining high- frequency components. It is a consistent result of the recent discovery that DNNs prioritize learning low-frequency components of the modeling function, while having very low learning speed on high- frequency components that tend to be more complex (Rahaman al.) 2019). This finding provides us with another perspective to understand why deeper layers learn more slowly.
We believe that understanding the roots of such fundamental convergence bias can give better grasp of the complicated learning process of DNNs. turn, can motivate more in-depth algorith- mic progress for the deep learning community.
This paper is organized as follows. In Section 2. we introduce our method for measuring conver- gence speed for different layers, and formally define the layer convergence bias. In Section 3 we examine the relationship between gradient magnitude and convergence rate, and show that the shal- lower layers tend converge faster even with smaller gradients. Then in Section 4 analyze the mechanism behind the layer convergence bias in DNN training. The layer-frequency correspon- dence is demonstrated in Section [ The practical significance of layer convergence bias is presented in Section 6 We further discuss the related work in Section [7 and conclude in Section g
2 LAYER CONVERGENCE BIAS
The deep architecture of DNNs arguably one of the most important factors for their powerful fitting abilities. With the benefit brought by the deep structures, there are also extra complexities in the training process coming into being. So far, we do not have a firm conclusion about whether some layers are learning faster than others.
For examining the convergence progress for a DNN, a common practice is checking its loss curve. However, this is not applicable for comparing the convergence between different layers. In this work, we define a measurement for layer-wise convergence in the following.
Definition 2.1 (Layer-wise convergence rate) At the training time let the deep neural network with layers (T((0 ) be f((x)) = (T(.) oT o.. (ax) R² + IRo, where i, o are the dimension of its imputs and outputs. We use f/t) to denote the parameters of the I-th layer T((). Assuming that 0lt) can finally converge to its optimal point 0f when + cc, we define the convergence rate of O during the time interval [11, 12] to be
(1((())
1|2 |10((a) - 07 ll2
1
(t2 t1)
[lof(u) 6,112
where to denotes the time point when the training starts.
In this definition, the numerator [10(t - -|102=110'*> 07 2 denotes how much the distance of the parameter 6 to the optimal point is shortened in the period it. ty). The denominator [[o((t) 07 || represents the distance between the initial point to the convergence point, whose primary function is to normalize the speed, allowing the convergence of different layers to compare with each other. Thus, the convergence rate of 0g can be understood as the ratio of normalized distance to time. Common optimization works Yi et al. 1999 Nesterov] 2003) defined the rate of convergence for ||[|++1] (0(+1)>0011
0 as limp-pa
1800-0011 -0'lla It focuses on measuring an exponential level convergence when the
2

---

Published as a conference paper at ICLR 2023
optimization step goes to infinity. Since the difference in convergence rates between layers usually appears at an early stage of training, and it is not large enough to compare at an exponential level, define our convergence metric to present the convergence difference in clearer way.
Observation 2.1 (Layer convergence bias). For 4 l2, 3 O, such that ((tr 1 < </2 <
Ci
C12
(21 when
Layer convergence bias indicates that at an early training phase < E. the parameters Ot, of shallower layer 1 tend to move to f7, faster than deeper layer H12 moving to f/.-- In the following, we use both synthetic and real datasets show that the layer convergence bias appears for both fully-connected neural networks (FCNNs) and convolutional neural networks (CNNs).
3 VERIFICATION OF LAYER CONVERGENCE BIAS
In this section, we try substantiate the central claim of this work. First, we use the FCNNs to show that the shallower layers tend to converge faster than the deeper layers on the regression task, even when the gradient values for shallower layers are smaller. We then use CNNs with modern ar- chitectures to verify that layer convergence bias is a common phenomenon in practical applications. All experimental settings in this work can found Appendix|A. A.1
3.1 LAYER CONVERGENCE BIAS IN FULLY-CONNECHE NETWORKS
For FCNNs, we construct a simple regression task to demonstrate that layers with smaller gradients do not necessarily learn more slowly than layers with larger gradients. The fitting target is f(x) = sin(x) + (3r) + 10 sin (0x) + 30 $30x)) with mean square error loss for training.
First, we use the FCNN [1-32-32-32-1] with the Sigmoid activations as simple example. In the following analysis, the first fully-connected layer (1-32) is named Layer 1, and the subsequent two layers (32-32) called Hidden layer 1, Hidden layer 2 respectively. The gradient values and the convergence processes for these layers are shown in Fig. (a). Two observations can be obtained from the plots: 1) The gradient of Hidden layer / is nearly always smaller than the gradient of Hidden layer 2. 2) Although shallower layers have smaller gradients, they seem to converge faster. For the first 50 epochs, the shallower layers are moving faster to their convergence point (e.g., (ffoo.ssa) "Laer 1 x 0.012, Hidden lager 1 = 0.009, Hidden layer 2 Ro 0.006), which is inconsistent with the previous view that higher gradients lead to faster learning Nielsen 2015).
(to:
Citootao)
To further validate the above results with deeper network, adopt residual connections (He al. 2016) for the FCNN (deep network fails to be trained in this task without residual connections) and use the ReLU activation function. The FCNN [1-(128-1 (728-128)--0 (128-128) with four residual blocks of width 128 shows similar results to the shallow FCNN without residual connection (see Fig. (b)). In this case, the difference in layer-wise convergence rate be observed even earlier (i.e., Cilfo.ts) Ree- Block Rs 2Cr(to.+s)
(188)(128-18)))
Res Block 1). which shows that the layer convergence bias also happens for deeper FCNNs with residual connections. is noteworthy that convergence
metric crucial to observe the layer convergence bias, which is elaborated in Appendix A22
taper
FCNN without residual connection
(b) FCNN with four residual blocks
Figure 1: Left (a.b): The absolute mean gradient values for different layers for FCNNs w/o residual connections in training. For both networks, deeper layers have larger gradients. Right (a,b): The convergence process of different layers for FCNNs. Shallower layers converge faster.
3

---

Published as a conference paper at ICLR 2023
Ceny
Lage
Lapers
Stapr2
Saged
[
0.4
60
0.2
20
90
80
1000
120
23
60
372
120
Epach
(a) ResNet-50 Aac 73.24%
VGG-19 Val Ace 71.9%
Figure 2: The convergence process of ResNet-50 and VGG-19 on ImageNet. During the first 50 epochs, shallower layers converge much faster than deeper layers. After the learning rate decays at the 50th epoch, parameters of deeper layers accelerate to move their convergence points.
Clearly, these results cannot reconcile with the previous view that larger gradients bring a higher learning speed for deeper layers, least for the DNNs used in this work. Instead, from the opti- mization point of view, the parameters of shallower layers are learning faster to converge.
3.2 LAYER CONVERGENCE BIAS CONVOLUTIONAL NETWORKS
Real-world datasets are very different from the synthetic data used in previous experiments. In order to utilize the layer convergence bias to understand and better improve DNNs in real applica- tions, it is important verify whether the layer convergence bias holds for CNNs images.
In the following experiments, we examine the layer-wise convergence process on ImageNet (Rus- sakovsky al. 2015) dataset with both ResNet-50 (He al.] 2016) and VGG-19 (Simonyan & Zisserman 2014). We train the CNNs for 120 epochs with learning rate decay at the 50th epoch (0.1 0.01) and the 100th epoch (0.01 0.001). The training processes shown in Fig. 2
For ResNet-50, we visualize the learning process of the first convolutional layer and its subsequent four stages. One can easily observe that at the beginning of training, the shallower layers converge much faster than the deeper layers (Cttage) 1 = (to
4) However, after the learning rate decays at the 50th epoch, deeper layers begin learn effectively and achieve higher convergence rate than the shallower layers (Ci(tao-tao)
1 R 0.50 Stage 4 large for the deeper layers to learn.
We conjecture that the initial learning rate is too
For VGG-19, we visualize its 1st, 5th, 9th, 13th, and 17th layers. This network show more sig- nificant convergence difference between layers than ResNet-50. At the first training stage with the initial learning rate, []0(*)) 07 |[0(*0) 0 for l E {5, 9, 13, 17), which means that all layers but the first one even slightly diverge. Usually, the divergence appears when the learning rate is large. This phenomenon confirms that the deeper layers cannot effectively learn with the large learning rate at the beginning.
The experiments of FCNNs and CNNs verify that layer convergence bias is common phenomenon for DNNs. In Section 5 and Appendix AA.3 A.4 we discuss the factors that would affect the phe- nomenon, and some in-depth findings they reveal.
4 MECHANISM BEHIND LAYER CONVERGENCE BIAS
So far, our investigation shows that the seemingly-right perspective for linking the layer-wise gradi- ent and convergence rate is tenuous, at best. Both FCNNs and CNNs demonstrate an evident bias that shallower layers learn faster. Can explain why this is the case?
Gradient Predictiveness. Since gradient values cannot determine the convergence rate, we wonder if the directions of the gradients play a more critical role. More chaotic update directions make con- vergence slower. Here we examine the gradient predictiveness (Santurkar et al. 2018) of different layers. If the gradient behavior is "predictive". less change in the gradient directions would appear when 1) the gradients are calculated with different batches of data; 2) the parameters of other layers update. Predictiveness also be simply understood the stability of gradient direction.
4

---

Published as a conference paper at ICLR 2023
Res-B ock
Rec-B
papreip Wendeny
sim(G G.
Ros B ock
sim(G (2)
0.8
Res Brock
0.4
0.6
0.4
022
0.0
50
100
150
200
250
300
0
50
100
100
200
250
300
Epoch
Epoch
(a) Gradient predictiveness worr, data
(b) Gradient predictiveness worf. parameters
Figure 3: The gradient predictiveness of shallower and deeper layers of FCNN. The learning rate decreases from 0.1 to 0.01 at Epoch 150.
Definition 4.1 Let y0))) be a batch of input-label pairs for the DNN to train time i, and (x/(0) (,(0),))))) ((0 be another batch of data. We define the gradient predictiveness of the lth layer at time t w.r.t data be the cosine similarity sim((Gr, Gi,,) ||G E [-111]] Likewise, the gradient
G
predictiveness w.r.f. parameters is defined sim (G, Gr,,,, where
Gt V 60 1(00)
1 1x0 )0)
G V 0:0 L(0
1 ((1) (()))
G V 050 L(o (++1) (00) 111 .... g(tt1; 10 (00),(0)))
((+1)
Here, Gt,, corresponds to the gradient of 8/0 Gi is the gradient of this layer with another batch of data, while G²,, means the gradient after all the other layers have updated to new values. Therefore, sim,Gra, G²,,) indicates the stability of gradients with different data batches. sim(Gr, GY,,) re- flects whether the currently estimated gradient in a consistent decreasing direction when the loss landscape is affected by the updating other layers' parameters. The gradient predictiveness during training is shown in Fig. where Res-Block has more predictive gradients than Res-Block 4.
Visualizing the Loss Landscapes. We curious about why gradients for deeper layers have poorer predictiveness. A hypothesis is that the loss landscapes for deeper layers are more rugged, making the parameters fluctuate more. A straightforward method validate this hypothesis plot- ting the loss landscapes for the parameters. To do this for particular layer 1, one can choose central point or and two direction vectors dr,1, d1,2- Then the loss landscape can be drawn with
[(3,, (2) L(0 + 8idr, + B2dr,2)
in the 3D space with 31, 32 forming a simplified parameter space. In this work, we generate random Gaussian directions for different layers, and normalize them to obtain the same norm of the corre- sponding layer. Specifically, we make the replacement di di ||0| for fully connected layer.
For convolutional layer, we use filter-wise normalization dx < ाলक
di
| as in (Li et al. 2018
where df represents the kth filter of the (th layer. We set both 3n and Bg in the domain of |1,1
Landscapes for FCNN. The loss landscapes for four residual blocks the FCNN are shown in Fig. For the shallower blocks, the surfaces are flatter near the minimizer, meaning that the gradient magnitudes may be small. However, small gradients do not necessarily lead slow learning speed in this case. Combined with the gradient predictiveness discussed above, a flatter loss landscape may lead to more consistent gradient directions, making the learning more smooth.
Landscapes for CNNs. The loss landscapes for ResNet-50 and VGG-19 19 on ImageNet are shown in Fig. 55) is interesting that deep convolutional networks with/without residual connections present totally different loss landscapes. For ResNet-50, its landscapes near the convergence point 0; are smooth and nearly convex, making the neural network easier to train. On the contrary. VGG-19 has much more shattered landscapes, the initial iterations probably lie in the chaotic regions, prohibiting its training (Balduzzi et al.] 2017). This may explain the much less efficient convergence towards the optimal point for VGG than ResNet at the initial phase (Fig.
5

---

Published as a conference paper at ICLR 2023
Res-Block
(b) Res-Block
(c) Res-Block
(i) Res-Block
Figure 4: The loss landscapes of different layers of FCNN. Deeper layers are optimized on more rugged landscapes, slowing down the learning process.
(1) R0 Res-Block
(b) ResNet-50 Res-Block
VGG-19 Layer
VGG-19 Layer
Figure 5: The loss landscapes of different layers of ResNet-50 (a,b) and VGG-19 (c.d) on ImageNet. The shallower layers for both networks have flatter minima, making them converge faster than the deeper layers. The plots for all layers can be found in Appendix A.5
Comparing different layers in the CNNs, the answer for layer convergence bias becomes clearer. The key difference between different layers' loss landscapes of ResNet-50 is the sharpness of the local minima (Fig. (a.))). We conjecture it is because of a well-known fact that the shallower layers of CNNs tend to learn general features which are applicable to various datasets and tasks, while deeper layers usually learn task-specific features (Yosinski et 2014). Before our work, ZZelerr & Fergus 2014) also revealed that the general features in five-layer CNN stabilized faster than the specific features. Since the general features are more evenly distributed, they usually cause less fluctuation for training. leading to flatter optima. Theoretically, flatter minimizers casier to be found by SGD optimizers (Pan et al. 2020). For VGG-19, its shallower and deeper layers also have flatter and sharper minima (Fig. (c.d)), respectively. The shattered loss landscape for its deeper layers may also explain its inefficient learning process with large learning rate (Fig. (b)).
Here we summarize the mechanism behind layer convergence bias: the parameters of shallower layers are easier to optimize due to their flatter loss landscapes. At a higher level, shallower layers learn general features, which usually easier.
DEEPER LAYERS FIT THE HIGH-PREQUENCY COMPONENTS
Recent advances in the learning process of DNNs (Rahaman al., 2019 Ronen et al. 2019 Xu & 'Zhou 2021) revealed that the low-frequency components of the target function are fitted much faster than the high-frequency components. There is natural question about whether there is some inherent link between layer convergence bias and this result. In this section, we investigate the answer, and surprisingly find that: the low-frequency parts are usually fitted by the shallower layers, while the remaining higher frequencies are mainly learned by the deeper layers. It provides us with an alternative perspective to understand the layer convergence bias.
The Correspondence for FCNN. With the residual structures, we can straightforwardly visualize what each block of a FCNN learns. Considering the FCNN with one input layer 2g TT(x) R² R R128 four residual blocks 21
(2 = (= + 2-11 R128 - R128 E 21, 4), and
an output layer
Ts (24) R 128 R The whole network can be expressed as
Tz(Z1 + (2 + (2(22) + (=(=)) = T5(21) + To(T(=z)) + TT(T((>)) + (T((()))))))))))))))
if the output layer T; is linear transformation. The fitting results for each layer shown in Fig, 6 can be seen that the deeper layers tend to the more complex components the target function y sin(x) + (in()3)) sin00x)) s300r).. Besides the curvature, the fitted functions

---

Published as a conference paper at ICLR 2023
0.00
(2(2))
(g(T(
Ts (Tx)))
(r(T((zs))
Figure 6: The visualization of what each residual block of the FCNN learns. From the first to the fourth block, the fitted function becomes more complex with smaller amplitude.
Laye
[
3.00
Layer
4.20
Stige
1:00
101
100
123
20
100
120
(a) ResNet-50
(b) VGG-19
Figure 7: The visualization of response frequencies for CNNs. As the training goes on, deeper layers become more sensitive to perturbations, indicating that they have higher response frequencies.
also consistent with the amplitudes of the components. Specifically, the ranges of the four fitted functions are 2.3, 0.7, 0.5, and 0.06, which are similar to the four components. This result further confirms the relationship between layers and frequencies.
The Correspondence for CNNs. For CNNs, we verify their layer-frequency correspondence through the response frequency (Xu et al. 2019). In nutshell, if an input-output mapping f pos- sesses significant high frequencies, then a small change in its input induces large change in the output. We generate standard Gaussian-distributee 50 and different layers of VGG-19. At the same time, small Gaussian perturbation Ar is added to the input. A larger change Ay of the layer output means the layer handles higher frequencies. The response frequencies are shown in Fig. 7 At the first 5 epochs of training on ImageNet, different layers for both ResNet-50 and VGG-19 But after about ten epochs, the response frequencies for deeper layers (e.g., stage 4 for ResNet-50, layer 13 for VGG-19) increase while the shallower layers show lower response frequencies. There- fore, conclude that the layer-frequency correspondence also holds for CNNs. In addition, it is not an innate nature of the layers, but a result of the training process.
input for different residual blocks of ResNet-
not show significantly different response frequencies.
How the target frequency affects layer convergence bias? To demonstrate the effect of layer- frequency correspondence on the layer convergence bias, we try fitting simpler targets with less high- frequency components, and what would happen the layer-wise convergence rate of FCNN. In Fig. (a-d), we only keep several lowest frequencies of the target, e.g., the target function y sin r is named "Complexity=1", and y = sin(x) sin(33) named "Complexity=2", etc. After discarding more and more high-frequency components, the deeper layers converge faster and faster. In this case, the layer convergence bias does not strictly hold anymore. In Fig. 8 (b), the
Res-Block 4 converges faster than Res-Block 3 after the 5th epoch. In Fig. 8 (c), the Res-Block 4 converges with similar speed Res-Block while the Res-Block 3 even learns faster than Res- Block 2. It seems that removing the higu-freyuency component that corresponds to deep layer can effectively accelerate its training. For CNNs, we also observe similar phenomena (Fig. X (e-h)). On simpler targets (e.g., CIFAR 10), the deeper layers converge faster than more complex targets (e.g., CIFARIO0) 100) An implication of this result is that the data complexity may be too low for the model. In practice, CIFAR datasets only need ResNet-18 to fit well (Wu al. 2020).
In fact, (Rahaman et al. 2019) had shown that different layers have some links to different frequen- cies. but the authors did not provide further insight for this phenomenon. This work verifies the underlying relationship between layers and fitting frequencies, and establishes a connection for this relationship the layer convergence bias.
7

---

Published as a conference paper at ICLR 2023
10 -1
Canoh
(a) FCNN Complexity
(b) FCNN Complexity=3
(c) FCNN Complexity=2
(d) FCNN Complexity=1
Eoge
Opech
C201
(c) ResNet-50 CIFAR100
(1) ResNett-0 CIFARIO
(g) VGG-19 CIFARIO0
VGG-19 CIFARIO
Figure 8: The convergence curves with different learning target complexities. (a-d): Decreasing target complexities for FCNNs. The deeper layers accelerate more than the shallower ones when high-frequency components are removed. (e-h): For CNNs, the deepest layers (i.e., Stage 4 Layer 17) learn faster on CIFAR10 than on CIFARTO T00 while the other layers not change much.
PRACTICAL SIGNIFICANCE
Up now, have been analyzing the layer convergence bias from theoretical perspective. This section discusses its practical use to drive the development of DNN architecture design, and new explanation for the acceleration effect of transfer learning with the help of layer convergence bias.
6.1 DNN ARCHITECTURE DESIGN
Modern CNN architectures (He et al. 2016) usually contain layers from narrow to wide (e.g., 64 channels of the first layer to 2048 chan- nels of the last layer). From the perspective of computational com- plexity, the narrower shallower layers make the corresponding large feature maps less computation-consumiii convergence bias, deeper layers with larger capacities also ben- eficial for the corresponding high-frequencies to be learned easier. Although this is common design for CNNs, Transformers (Doso- vitskly et al. (020) usually apply the same architecture for all en- coders. For vision Transformer with 12 encoders, with width 2/4/8 to construct three variants. The variants only dif- fer in the arrangement of different encoders, we use W denote the widths, and N to denote the number of each kind of encoders. The configures are summarized below:
Tearspp (8)
8
Considering the layer
WHALKA Ne)U
50
100
160
200
Epoch
encoders
Figure 9: Performance of
three variants of ViTs on Im- ageNet.
deeper encoders wider: = -2, 8), (6,3,3)
vanilla architecture: W (4,4,4), N (4,4,4)
deeper encoders narrower: W = (8, 4, 2), N (3,3,6)
Fig. 9 shows their performances, with the best accuracy of 80.75%, 78.88%, and 75.75%, respec- tively. We find that with the same number of parameters, putting the wider layers deeper results effective way improve the model
in higher training performance. This finding may serve capacity. The causal connection between layer complexity distribution and model performance is discussed in Appendix A.6, And layer convergence bias for ViT is analyzed in Appendix A.7 A.7
8

---

Published as a conference paper at ICLR 2023
6.2 ACCELERATION EFFECT OF TRANSFER LEARNING
Transfer learning (fine-tuning with the pre-trained models) is widely-used technique that can ac- celerate the model convergence (Shao et al. 2018h a Liang Zheng, 2020). We show the layer convergence curves w/o transfer learning
the Flowers dataset (Nilsback & Zisserman, 2006). When training from scratch (Fig. T (a)), the shallower layers converge faster so that the deeper layers extract semantic features based on basic features. Local minima of Stage 4 is sharp in this case. However, with transfer learning (Fig. TO (b)), deeper layers can directly be built on the pre-trained basic features. The Stage 4 shows much higher convergence rate among all layers, its loss landscape also becomes flatter. Two observations that are not consistent with layer convergence bias summarized in the following: 1) the pre-tra pre-trained shallower layers nearly optimal, they don't present fast convergence in transfer learning; 2) although the pre-trained deeper layers are as optimal as the shallower layers do, their loss landscapes are much flatter than training from scratch, which makes them converge much faster.
1000-41
(a) Train from scratch
(b) Fino-tuning
Figure 10: Effects of transfer learning the training process. Left (a.b): The layer convergence
process of ResNet-50. Right (a.b): The loss landscapes Stage w/o transfer learning.
7 RELATED WORK
DNNs with gradient-based training show great potential to fit targets with arbitrary complexities
(Homik et al. 1989 Leshno et al. 1993). given sufficient width. With the advances in the last decade
to verify the capability of the depth of universal approximators (Delallcau & Bengio 2011 [llan &
Shamir 2016 Lu et al. 2017). practitioners tried to reduce the width of neural networks by adding more layers (Simonyan & Zisserman 2014 He al., 2016 Huang et al., (017). We are also inspired
by research on local properties Scarppess//latness of loss functions at minima (Keskar et al. 2017
Li et al. 2018) and relationship between convergence rate and generalization (Hardt et al.) 2016). Furthermore, LARS optimizer (You et al.
2017) shares some valuable insights on layer convergence,
which are discussed in Appendix A.8 In practice, the idea of layer convergence bias had been intuitively applied to accelerate DNN training (Huang et al. [0066] Brock et al., 2017) and mitigating catastrophic forgetting (Ramasesh al. 2020). The arrangement schemes of CNN/Transformer
blocks were explored by (Liu et al. 2022bla)
CONCLUSION
In this work, we empirically studied the phenomenon that the shallower layers of DNNs tend to converge faster than the deeper layers, called layer convergence bias. This phenomenon is natu- ral preference in the process of DNN training: the shallower layers are responsible for extracting low-level features which are more evenly distributed and easier learn, while deeper layers refine these features to do specific tasks. This makes the loss landscapes for shallower layers flatter than the landscapes for deeper layers, making shallower layers converge faster. In addition, this work established connection between layers and learned frequencies. By showing deeper layers tend to components in the target function, we can understand the layer convergence
fit the bias from another perspective. We finally took DNN architecture design and transfer learning as two examples to show how theoretical findings in this work can shed light on the practical applica- tions of deep learning. For progress continue, more in-depth understanding of the properties of neural networks is needed. We also hope that the layer convergence bias can inspire more practical improvements in the DNNs' architecture design and training schemes.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by the Lustgarten Foundation for Pancreatic Cancer Research and the McGovern Foundation.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian Mc MiWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In Interna- tional Conference on Machine Learning, pp. 342-350. PMLR, 2017.
Andrew Brock, Theodore Lim, James Millar Ritchie, and Nicholas J Weston. Freezeout: Accelerate training by progressively freezing layers. In NIPS 2017 Workshop on Optimization: 10th NIPS Workshop Optimization for Machine Learning, 2017.
Olivier Delalleau and Yoshua Bengio. Shallow deep sum-product networks. Advances in neural information processing systems, 24, 2011.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. Conference on learning theory, pp. 907-940. PMLR, 2016.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In 2013 IEEE international conference on acoustics, speech and signal Ieee, 2013.
processing, pp.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pp. 1225-1234, PMLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Sepp Hochreiter.
Untersuchungen dynamischen neuronalen netzen. Diploma, Technische Uni- versität München, 91(1), 1991.
Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber, et al. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural networks, 2(5):359-366, 1989.
Gao Huang, Yu Sun. Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. European conference on computer vision, pp. 646-661. Springer, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 4700-4708, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448-456. PMLR, 2015.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvumakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-566 2021.
10

---

Published as a conference paper at ICLR 2023
Nitish Shirish Keskar, Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy. On large-baud batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representattons, ICLR 2017, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward net- works with nonpolynomial activation function can approximate any function. Neural networks, 6(6):861-667, 1993.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances neural information processing systems, 31, 2018.
Gaobo Liang and Lixin Zheng. transfer learning method with deep residual network for pediatric pneumonia diagnosis. Computer methods and programs in biomedicine, 887:104964, 2020.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin trans- former. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 3202-32222 3,11, 2022a.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976-11986, 2022b.
Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. Advances in neural information processing systems, 30, 2017.
Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87, Springer Science & Business Media, 2003.
Michael Nielsen. Neural networks and deep learning, volume 25. Determination press San Francisco, CA, USA, 2015.
M-E Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In CVPR, volume 2, pp. 1447-1454. IEEE, 2006.
Zhou Pan, Feng Jiashi, Ma Chao, Xiong Caiming, Chu Hong Hoi Steven, and E Weinan. Towards theoretically understanding why sgd generalizes better than adam deep learning. In Advances in Neural Information Processing Systems, pp. 21285-21296, 2020.
Maithra Raghu. Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Confer- ence on Machine Learning, pp. 5301-5310. PMLR, 2019.
Vinay V Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semantics. arXiv preprint arXiv:2007.07400 2020.
Basri Ronen, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural net- works for learned functions of different frequencies. Advances Neural Information Processing Systems, 32, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3)2111252 2015.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? Advances in neural information processing systems, 31, 2018.
11

---

Published as a conference paper at ICLR 2023
Kun Shao, Yuanheng Zhu, and Dongbin Zhao. Starcraft micromanagement with reinforcement learning and curriculum transfer learning. JEEE Transactions on Emerging Topics in Computa- tional Intelligence, 3(1):73-88) 84, 2018a.
Siyu Shao, Stephen McAleer, Ruqiang Yan, and Pierre Baldi. Highly accurate machine fault diagno- sis using deep transfer learning. IEEE Transactions on Industrial Informatics, 15(46:2446.2455 2018b.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484449) 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
1556,
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
Pengxiang Wu, Songzhu Zheng. Mayank Goswami, Dimitris Metaxas, and Chao Chen. A topolog- ical filter for learning with label noise. Advances neural information processing systems, 33: 213833 2020.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pp- 3-19, 2018.
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv: 065233. 2019.
Zhiqin John Xu and Hanxu Zhou Deep frequency principle towards understanding why deeper learning is faster. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 10551-, 2021.
Zhang Yi, Pheng-Ann Heng, and Ada Wai-Chee Fu. Estimate of exponential convergence rate and exponential stability for neural networks. IEEE Transactions on Neural Networks, 10(6):1487- 1493, 1999.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances neural information processing systems, 27, 2014.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint ar Ttt.::170.03888. 2017.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1
EXPERIMENTAL SETTINGS
Datasets. The synthetic and real datasets are summarized in the Tab.
Table Descriptions and statisties of the datasets used in this work.
Dataset
Size (train/test)
Classes
Data description
Sine rogression
5000/5000
I/aa
Function with four sinc composents, domain [-2.2] Photos of common objects
ImageNet
1,281, 167/500000
1000
CIFAR-10
50,000/10,000
10
Photos of common objects, image sizes 32
CIFAR-100
50,000/10,000
100
Photos of common objects, image sizes 32 x 32 Find grained photos Blawers Find-grained photos of aircnafts Photos/paintingskkkcttee
1,088/272
FGVC Aireraft
6,667/3,333
100
Caltech-101
3,060N6,084
common objocts
CUB-200
5.994/5.794
200
Find-gr Find-grained photos of birds Oil Paintings. murals, drawings, latloos
DomainÑei painting
50,416/21.850
345
Network Architectures. The FCNNs, CNNs, and Vision Transformers are summarized in the Tab.
Table 2: Complexities and architectures of DNNs used in this work.
Model
#Panameters
Muli-adds
Architecture description
FCNN (no ses)
2k
layers [1-32-32-32-I]
FCNN (res)
132k
390k
Ic [1- [1-128] - 4 res- ces-blocks [128-128-123 CORV stages [3,4,63]
[128-11
VGG-19
143.7M
19.8G
16 cony layers, fc layers
VIT
9.9M
77.2M
12 Transfomerr encoler blocks (busic width 256), layer
Training uypee-parameters. For the regression task, we train FCNNs with SGD optimizers for 300 epochs. The initial learning rate is 0.1, with learning rate decay (to 0.01) at the 150th epoch. The batch size is 128, no weight decay (L2 regularization) conducted.
For the ImageNet classification task with CNNs, we train ResNet-50 and VGG-19 19 for 120 epochs with SGD optimizers. The initial learning rate is 0.1, with learning rate decays the 50th and 1 100th epoch to 0.01 and 0.001, respectively. The batch size is 256, the input image size is 224 and the weight decay coefficient 10-4
For Vision Transformers IeageNet dataset, we train them for 200 epochs with Adam optimizers. The peak learning rate set 0.0003. We use linear learning rate warm-up for 10,000 iterations, and subsequent cosine learning rate decay. The batch size is 256, the input image size is 224 and the weight decay coefficient 10
For CNN image classification on other datasets, train models for 100 epochs with SGD optimiz- ers. Initial learning rate of 0.01 and cosine learning rate scheduler are applied. The batch size is 128, the input image sizes are 32? (for CIFAR) and 224 (for Flowers, Aircraft, Caltech, CUB, and Net, and the weight decay coefficient 10 10-1
13

---

Published as a conference paper at ICLR 2023
A.2 CONVERGENCE MEASUREMENT USING WEIGHT VARIATION
In Section 2 we have introduced the convergence measurement in this work. This measurement is simple and straightforward, and it show how each layer in a DNN converges during the whole training process (Fig. [ for fully connected networks and Fig. 2 for CNNs) by examining the distance between the training parameters and the converged parameters. However, it has not been verified whether calculating the parameter distance variation to the convergence point between two adjacent epochs is necessary. After all, the measurement highly depend on the convergence point, which can only be obtained after the whole training process.
We come up with simplified convergence measurement. This method uses weight variation metric to examine how fast layer is learning, and whether this layer reaches a state of convergence. a layer is learning actively, reasonable that its weights variate drastically during training. For
(1//t///22 the converged layers, their weights usually keep stable. So we use ||0(()) 0((++1) /12/11/(())) [|2, the normalized weight variation of layer during epoch and + 1, to illustrate how actively it is learning.
Singe
Blago
16"
10'
16
120
opach
opach
[a) ResNes-50 Val Acc 73.24%
(h) VGG-19 Val 71.89%
Figure 11: The convergence processes of ResNet-50 and VGG-19 on ImageNet. The results are illustrated with weight variations. The learning rate decays at epoch 50 and epoch 100.
shown in Fig. From The results of ResNet-50 and VGG-19 training process on ImageNet this plot, we can see that after learning rate decays at epoch 50 and 100, the weight variations drop evidently. However, the weight variations of each layer do not show apparent decreasing trend when the learning rate keeps stable, which indicates that the training of DNNs do not converge as usual convex optimization problems do (e.g., linear programming). Therefore, it is hard for us to compare the convergence rates of different layers by observing their convergence curves. We cannot find a clear clue like what was given by the convergence measurement Section to get the layer convergence bias. All in all, we can safely claim that, it is crucial for the convergence metric to consider direction information to measure how fast different layers are learning towards their convergence points. Our previous convergence measurement really needs to examine convergence by calculating the parameter distance between the current point to convergence point.
14

---

Published as a conference paper at ICLR 2023
A.3 FACTORS AFFECTING LAYER CONVERGENCE BIAS
In Section 5 have shown that the complexity of the datasets is an important factor affecting layer convergence bias. When the fitting target function is complex enough with both low and high frequency components, the shallower layers learn the low low-frequency components while the deeper layers learn the high-freueency components. Here we use the FCNNs with residual connections to show whether some other important factors would affect the layer convergence bias. All following experiments are conducted on the same regression task in Section 3
Model Depth. The default architecture used in previous experiments is the four-blocks FCNN, here we try adding more blocks to make the network deeper and see what change will happen. As shown Fig. [2] all the networks show layer convergence bias. With more and more res-blocks, the overall convergence of the network becomes slightly faster.
Leae"
Lean'
Leee
Bock
Eaudh
(a) Res-Blocks
(b) 8 Res-Blocks
(c) 12 Res-Blocks
(d) 16 Res-Blocks
Figure 12: The convergence process of FCNNs with different number of res-blocks.
Learning Rate. The results with different learning rates are shown in Fig, 13 When the learning rate gets smaller, layer convergence bias becomes weaker. This is because the gradient predictive- ness w.r.f. parameters of all layers get close to 1 (see Fig. D] (b.right) for the predictiveness with the learning rate of 0.01). In this case, a layer is less influenced by the updates of parameters in other layers, only the gradient predictiveness w.t.. data matters for the convergence rate. In addi- tion, smaller learning rates are beneficial for the deeper layers to converge because of their sharper minima.
New
Bost
LR-0.01
(b) LR-0.03
Figure 13: The convergence process of FCNNs with different learning rates.
Weight Decay. The experiments with FCNNs in previous sections are conducted without weight de- cay. is interesting to investigate the sensitivity of the layer-wise model convergence with different weight decay strengths. The results are shown in Fig. [4] We can see that when the weight decay becomes stronger, the residual blocks converge slower in more and more similar convergence rate. We conjecture the reason is that weight decay dominates the total loss when its coefficient is large. In this way, the layer parameters with similar initialization scales tend to converge in similar speed toward zero. Because the residual blocks have identical architectures, they share the same initial parameter distribution, and converge in the same speed when weight decay is strong.
we have discussed the mechanism behind layer convergence bias. The flatter/sharper minimizers of different layers make SGD learn at different speeds. This is because SGD is more good finding flatter minimizers (Pan et al., 2020). In Fig. [5] we compare SGD with three adaptive optimizers: Adagrad, RMSprop. and Adam. It is evident that with adaptive optimizers, layer convergence bias does not hold anymore. We conjecture the reason behind this
Optimizer. In Section 4
15

---

Published as a conference paper at ICLR 2023
10 0.
Corhh
(a) WD--le-7
(b) WD-1e-6
(c) WD-1e-5
(d) WD-1e-4
Figure 14: The convergence process of FCNNs with different weight decay strengths.
Foetions
Mark
5pec
(a) SGD
(b) Adagrad
(c) RMSprop
(d) Adam
Figure 15: The convergence process of FCNNs with different optimizers.
is that the adaptive optimizers heuristically assign different learning rates for different parameters, making their optimization hardly predictable.
Normalization Methods. Like residual connection, batch normalization loffe & Szegedy (2015) is also a common design in modern DNN architectures. As discussed previous literature, normaliza- tion in the neural networks helps make the layer inputs more stable and make the loss landscapes smoother, thus accelerates the model training Santurkar et al. (2018). In Section and Section 4 we mainly use the FCNNs without normalization to verify and explore the layer convergence bias. Here we investigate how the normalization methods (i.e., batch normalization, layer normalization Ba et al. (2016). and group normalization Wu & He (2018)) help the convergence, and whether the shallower layers still converge faster these cases. As shown in Fig. [6]] all layers converge faster when adding batch normalization to them. Particularly. "Res-Block 1 accelerates the most and reach a similar convergence rate as "Layer 1". The layer convergence bias also holds for batch nor- malization. For layer normalization and group normalization, the models show a significantly faster convergence rate than the model using batch normalization. All layers show effective convergence an early stage of training (i.e., the first five epochs). these two cases, different layers have similar convergence rates, thus evident layer convergence bias emerges.
Laper
normalization
Batch normalization
Layer normalization
(d) Group normalizaaiio
Figure 16: The convergence process of FCNNs with different normalization methods. When using group normalization, we set the group number to 8.
16

---

Published as a conference paper at ICLR 2023
A.4 RESULTS ON HARDER DATASETS
For verifying the layer convergence bias more datasets, show convergence results four harder image classification datasets (see Fig. 17p. Most of the classes these datasets only have 100 samples, making them harder learn. Note that the experiments are conducted with the learning rate of 0.01 (learning rate of 0.1 failed in some cases because these datasets have too many classes but not sufficient samples, leading non-decreasing loss), some deeper layers have quite similar convergence rates because of the small learning rate. But roughly speaking, layer convergence bias still holds for these datasets.
Aireraft
CUB200
Caliech101
Domainnel painting
20
13ya
101
10
Epech
Ezech
(a) ResNet--5
Aireraft
CUB200
Caltoch101
DomainNel painting
Lywe
Telu 1000
Epcth
(b) VGG-19
Figure 17: The convergence process of CNNs on four image classification tasks.
A.5 REPEATABILITY OF THE VISUALIZATIONS
Do different ImageNet trained models produce dramatically different loss landscapes? We plot the loss landscapes of different models with different random seeds in Fig. [9 Quite similar patterns of the landscapes for different layers can be observed on both ResNet and VGG with different random seeds.
440
(8) Stage Seed
(b) Stage Soed
(0) Stage 3 Seed
(6) Stago 4 Seod
(d) Stago Seod
Stago Scod
( Stage 3 Seed 2
(x) Stage 4 Seod 2
Figure 18: The loss landscapes of different layers of ResNet-50.
17

---

Published as a conference paper at ICLR 2023
Layer Seed
Layer Seed
Stage
Stage 13 Seed
(e) Layer Seed 2
Layer Seed 2
Stage Seed 2
Stuge 13 Seed 2
Figure 19: The loss landscapes of different layers of VGG-19.
A.6 MODELS OBEYING LAYER CONVERGENCE BIAS PERFORM BETTER
In Section and Section it is discussed that layer convergence bias indicates that the shallower layers are learning low-level features (or low-frequency components of the target function). It is reasonable learning low-level features first have greater potential reach good model performance, since the model can establish its high-level features based on relatively stable low-level feature spaces.
To examine whether the fast establishment of low-level features benefits model performance, we train four different FCNN models with the same amount of parameters, but different architectures, to fit the Sine target with four components. This experiment is based on a finding that residual block with more layers in it tends to converge more slowly. We construct four FCNN models, each of them has four residual blocks (maybe in different sizes). The convergence processes are shown in Fig. 20 We can see that the blocks with the largest complexity always converge the most slowly. As the block with depth=4 being placed shallower in the FCNN, the regression MSE loss goes higher. In other word, if shallower layer converge slowly, the model gets poorer performance. This may due to the vulnerability of deeper layers. If they converge based on changing shallower layers, it is hard for them to learn good features based on their unstable inputs.
Bood
(9) Res-Blocks = (4.1.1.1) Val loss 2.7c-4
(b) Res-Blocks = (1.4.1.1)
(e) Res-Blocks = (1.14.1)
(d) Res-Blocks = (1.1.1.4)
Val loss 2.4c-4
Val loss 1.844
Val loss 1.4e-4
Figure 20: The convergence process of FCNNs with different residual block sizes and their vali- dation performance on the regression task. Each model has a four-layer residual block and three one-layer residual blocks (e.g., "Res- "Res-Blocks- Blocks=(44,,1..)) means the first residual block has four lay- ers, and the rest three blocks have only one layer).
The results can also be understood from another perspective. If the deeper block contains more parameters (with more fully connected layers in it), it would be helpful for this block to lear the corresponding high-frequency components of the target function. Therefore, the model can reach better performance. A similar observation is obtained in Section 6.1 when putting wider layers of the ViT deeper, the model can reach higher performance.
18

---

Published as a conference paper at ICLR 2023
A.7 LAYER CONVERGENCE BIAS FOR VISION TRANSFORMERS
As discussed in Section 6. ViT can benefit from distributing more parameters in the deeper layers. This result comes from one of our main findings about layer convergence bias: the deeper layers tend to learn high-frequency components of the target function, thus converge more slowly. So adding more parameters for the deeper layers is beneficial for these layers learn the high-frequengy components which are usually harder.
When making this claim, we do not verify the layer convergence bias for the ViT. The main diffi- culty for verifying layer convergence bias for ViTs is brought by its typical training scheme. ViT needs adaptive optimizers train, otherwise it converges very slowly. However, adaptive optimizers change the learning rates of different parameters according to their optimization procedures. This leads to unfair convergence comparison between layers, thus affects the layer convergence bias, as try both SGD and Adam optimizers for training ViTs ImageNet,
shown in Fig 15 Therefore, and see whether layer convergence bias holds in some cases. As shown in Fig. [1 (a), the ViT shows roughly trend of layer convergence bias when optimizing with Adam, where the deepest "Encoder Block 12" converges the slowest. However, some other layers do not strictly obey layer convergence bias (e.g., the shallowest "Patch Embedding" does not learn fastest among all blocks). When optimizing with SGD, the ViT shows good layer convergence bias. The results indicate that ViTs approximately share the same rules FCNNs and CNNs, thus supports the discussions in Section
1/
Pirnn Enteecing
Encuder oce
Dicodar
Parin Embeedire
Entodo B ocs
Ercoder ocx
Encoder 8 nec
Encoder ooc
0.0
E Eroodo Box 12
Droodoo Bock
0.7
0.9
08
0.5
555
04
63
(a) VIT Adam
(b) VIT SGD
Figure 21: The convergence curves of ViTs on ImageNet with different optimizers. With Adam op- timizer, the ViT does not obey the layer convergence bias strictly. While SGD can ensure relatively ideal faster convergence processes of shallower layers.
A.8 CONNECTION TO LARS OPTIMIZATION SCHEME
One of the most important factors that affect the optimization procedure is the learning rate. In this work, it is shown that the shallower layers can learn effectively with large learning rates, but the deeper layers only learn fast after learning rate decays. Is there any connection between layers and its suitable learning rate?
LARS optimizer You et al. (2177) made a significant contribution to training DNNs with huge batch sizes and large learning rates. The key observation in the literature that the seight-to-gradieer ratio highly varies in different layers. If a layer has greater gradients and relatively smaller weights, would be hard for to converge due the vigorous parameter update. So LARS considers the scale of the weights and its gradient norms in each layer and assigns a local learning rate for layer to make it converge effectively and stably. For FCNNs in our work, its different hidden layers initialized with the same scale due to their identical architecture, but the deeper layers usually have larger gradients. As a result, the larger gradients may make these layers struggle to converge. Similarly, the CNNs (i.e., ResNet-50 and VGG-19) have wider deeper layers. These layers have smaller initial parameters, so their gradients may lead to drastic weight variations if the learning rate is too large. In this way, we can understand why they cannot get close their optimal points effectively at the early stage of training. It explains layer convergence bias from another perspective.
19


Reviews:
[{"review_id": "fAiFk6j-Wd", "reviewer": "Reviewer_xk3R", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the observation that, during the training of deep neural networks, lower layers (\"shallower\", closer to the input) converge faster towards their final value, compared to upper layers (\"deeper\", further away from the input).\nIt quantifies that tendency on several architectures, on synthetic data and natural images.\nIt then draws links with the \"spectral bias\" of neural networks, showing that lower layers seem to learn functions with lower frequencies (in the input space), and higher layers learn, more slowly, the higher-frequency components (if any).", "summary_of_the_review": "This is a really interesting exploratory work, and the explanations advanced are plausible, but not really convincing. It is not clear how much of these findings can generalize to other architectures, or distributions other than natural images and synthetic 1D data.\nI'm leaning towards accepting it.\n\n**Update after responses**\nThanks for your answers and clarifications.\nWhile I recognize they add insight, I still think the paper is borderline, and I'm keeping my rating.", "strength_and_weaknesses": "Strengths\n-------------\n1. Confirms observations of faster convergence in the first layers during training of a DNN, using a quantitative metric\n2. Establishes a link between learning lower frequencies earlier, and learning them in the first layers, showing that higher layers converge faster if they don't have any high-frequency components to learn\n3. Empirically explores loss landscapes of different layers, for different architectures (FCNN, VGG, ResNets)\n4. Exploits these observations to propose a better repartition of parameters for vision transformers\n\nWeaknesses\n-----------------\n1. I don't think the reason behind these observations have been convincingly proven. The explanations provided are consistent, but may not be the only ones.\n2. Previous work should be more prominently mentioned, as these observations are not new.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "J3UFzyDhhx", "reviewer": "Reviewer_w2MX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Training neural networks on large-scale real-world datasets typically involve gradient-based updates of deep networks. Therefore, understanding the learning dynamics of these models is critical from the perspective of interpretability, as well as designing better architectures/training algorithms. In this work, the authors study the convergence of different layers in a model across training and demonstrate that early layers in canonical deep learning architectures converge faster and learn low-frequency components of a target function. A central claim of the hypothesis is that gradient magnitude is insufficient to characterize the learnability across layers and instead proposes the notion of gradient predictability as a proxy for estimating learning progress. With analysis across simplified target functions and realistic datasets/models (CIFAR/Resnets), the authors show that their notion of measuring learning progress is consistent with existing work in the literature. Further, the insights are used to motivate and explain design choices in the context of transformer architecture and the benefits of transfer learning. ", "summary_of_the_review": "Summarily, the authors consider the learning dynamics of canonical deep learning architectures through the lens of gradient predictability across network layers. Using simplified target functions (mixture of sinusoids at different frequencies), the authors establish earlier network layers tend to learn lower-frequency components (and converge faster). In contrast, the last layers learn high-frequency components. Experiments across multiple model families and datasets suggest the ubiquity of this finding in convolutional architectures. Finally, these insights are used to guide design choices for better architecture design in the context of vision transformers and explain the efficacy of transfer learning in accelerating learning on downstream tasks. ", "strength_and_weaknesses": "Some strengths of the work include the following:\n+ The paper is clearly written, with well-grounded motivations and pedagogical examples on understanding the learning dynamics across different layers of canonical neural network architectures. \n+ The authors highlight the limitations of gradient magnitude as a measure of learning progress, instead proposing a proxy that measures the predictability of gradients as learning progresses.\n+ The insights are leveraged to provide practical design guidance and explain known phenomena in the literature (transfer learning accelerates training on downstream tasks).\n\nSome weaknesses of the current draft are outlined below:\n- The critical observations of the work, i.e., layers converge at different rates and learn different components of the target functions, are well established in the literature. However, it would be helpful to clarify the utility/novelty of the discussion using simplified target functions (Section 3).  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8-2TTMWF4a", "reviewer": "Reviewer_RsgU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an empirical study trying to explain the differences in speed of convergence of different layers of a neural network. To that end, the authors propose to measure the speed of convergence by tracking the rate of change in distance to the optimum of different layers within a time span. They argue that, contrary to the gradient magnitude, this metric can capture a more relevant notion of convergence speed consistent with different phenomena. In particular, ther experiments seem to indicate that on many training tasks, deeper layers converge slower than shallower ones, and that they fit higher frequency functional terms. The authors then argue that this inherent difficulty of training deeper layers can explain the success of using wider layers deeper in CNNs. Inspired by this intuition they then suggest and, partially test, using this design choice on modern ViTs.", "summary_of_the_review": "Overall, I believe this work can be of interest to the community as it provides intuitive insights on an intriguing phenomenon. This being said, I believe these insights coulld be better grounded on certain experiments and isolated with respect to different hyperparameters. If the experiments where more thorough this paper could be a strong contribution. In the current stage, though, I believe this is a borderline paper, slightly above the threshold for acceptance.\n\n----\n\n**Post-rebuttal udpate** : After the detailed rebuttal by the authors where a new improved version of the manuscript has been shared, I have decided to increase my score to an 8, as most of my concerns have been alleviated.", "strength_and_weaknesses": "# Strengths\n1. **Interesting empirical study based on an intriguing phenomenon**: The observation that deeper layers have a slower speed of convergence when training on many tasks is interesting and worth investigating. In particular, I find the following observations and experiments noteworthy:\n   a. Connection between rate of convergence of deeper layers and frequency of target function. \n   b. Consistent observation that deeper layers have a rougher loss landscape.\n   c. Faster convergence rates on shallower layers and frequency specialization is not an architectural bias, as it changes during training.\n2. **Consistent observations on synthetic and real tasks**: I really appreciate the alternation between experiments on synthetic tasks and real tasks in which the controllable experiments are used to isolate a described effect which can then be observed on the more realistic training runs.\n3. **Clear writing**: The paper is clearly written and easy-to-follow. The different sections follow naturally from the previous ones and create a coherent storyline.\n\n# Weaknesses\n1. **Lack of causal connection to support arguments in Sec. 6.1**: The explanationss about the usefulness of having wider layers at deeper stages of a neural network are not proprely backed by experiments. In particular, no experiment shows that increasing/decreasing the width of a given layer increases/decreases its speed of convergence. Similarly, the claimed benefits of the proposed architectural changes to ViTs are only superficially investigated. To support these claims, a much further investigation than the one provided should be performed.\n2. **Results based on single runs**: Most results are based on observations made for a single run and lack studies of sensitivity to different hyperparameters. In this regard, some of the results presented in this study might be a bit anecdotal.\n3. **No normalization of gradients**: While the authors soundly argue that their proposed convergence rate metric needs proper normalization to be comparable among layers, they do not seem to care about normalizing the gradients by number of parameters so that they become comparable among layers. If deeper layers have many more parameters than shallower ones, then it is not surprising that the unnormalized gradient norm of the shallower layers is smaller.\n4. (Minor) **Convergence rate metric is not strongly motivated**: The reasons behind using this specific convergence rate metric and not the standard convergence rate used in optimization theory are not very strongly motivated. In particular, the short footnote explaining this on page 2 is a bit unclear.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3NALzGxJa0B", "reviewer": "Reviewer_ogf3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study a specific notion of convergence and compare the convergence of different layers. It is found that gradient norm does not correlate to the convergence, early layers converge faster and learn low frequency components ", "summary_of_the_review": "The paper proposes some original observations regarding layerwise convergence. The paper is built on a specific definition which is not fully analyzed. Additionally the experimental results focus on CNN and image data, while the claims suggest it is more general. The practical applications of the results as shown by the authors are not completely clear. ", "strength_and_weaknesses": "*Strengths*\n\n-The observations are intuitive and may help provide insights on the training dynamics of NNs \n\n-To the best of the reviewers knowledge the layer convergence bias has not been pointed out in the literature\n\n\n\n*Weakness*\n\n-Article uses a specific definition of convergence that normalizes across layers. Although it seems sensible this is the basis of the entire article so it would be good to fully justify it and discuss other potential notions of comparing convergence across different sets of parameters. Are there limitations to this notion of convergence?\n\n-Some of the conclusions are made on rather limited examples\n   a)The main results are shown for a single task and with convolutional networks, how does the architecture affect these conclusions? Would MLP or ViT have similar results?\n   b) Gradient “predictiveness”  is concluded to be higher based on 2 layers of one specific model in one specific training scenario analyzed. -The difference between the layers does not really look significant on the shown figures especially with the noise in the measurements\n-The novelty to some existing work is not strictly made clear. For example the authors note the observations of the classic Zeiler et al. \n-The practical applications for DNN architecture design is not completely convincing. First of all the authors have not even confirmed the results on ViT. Secondly the connection to the observations and making the deeper layers wider is a bit unclear.\n- It was not clear what the practical observations of 6.2 were\n- It seems a more natural application of such results would be in the realm of optimization, driving new layerwise adaptive optimization schemes. It would be interesting if the authors can also comment on the connection of their work to the popular LARS optimization scheme (You et al \"Large Batch Training of Convolutiona Networks\")\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster