Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors study a specific notion of convergence and compare the convergence of different layers. It is found that gradient norm does not correlate to the convergence, early layers converge faster and learn low frequency components 
strength_and_weaknesses: *Strengths*

-The observations are intuitive and may help provide insights on the training dynamics of NNs 

-To the best of the reviewers knowledge the layer convergence bias has not been pointed out in the literature



*Weakness*

-Article uses a specific definition of convergence that normalizes across layers. Although it seems sensible this is the basis of the entire article so it would be good to fully justify it and discuss other potential notions of comparing convergence across different sets of parameters. Are there limitations to this notion of convergence?

-Some of the conclusions are made on rather limited examples
   a)The main results are shown for a single task and with convolutional networks, how does the architecture affect these conclusions? Would MLP or ViT have similar results?
   b) Gradient “predictiveness”  is concluded to be higher based on 2 layers of one specific model in one specific training scenario analyzed. -The difference between the layers does not really look significant on the shown figures especially with the noise in the measurements
-The novelty to some existing work is not strictly made clear. For example the authors note the observations of the classic Zeiler et al. 
-The practical applications for DNN architecture design is not completely convincing. First of all the authors have not even confirmed the results on ViT. Secondly the connection to the observations and making the deeper layers wider is a bit unclear.
- It was not clear what the practical observations of 6.2 were
- It seems a more natural application of such results would be in the realm of optimization, driving new layerwise adaptive optimization schemes. It would be interesting if the authors can also comment on the connection of their work to the popular LARS optimization scheme (You et al "Large Batch Training of Convolutiona Networks")

clarity,_quality,_novelty_and_reproducibility: -The paper is overall clear however some parts do not emphasize the takeaway message making it challenging to follow some of the authors more subtle observations (e.g. 6.2). Some key figure's such as 1 is are bit small hurting readability.

-The work is original however some of the ideas have been described in related work 

-The paper seems possible to reproducible 
summary_of_the_review: The paper proposes some original observations regarding layerwise convergence. The paper is built on a specific definition which is not fully analyzed. Additionally the experimental results focus on CNN and image data, while the claims suggest it is more general. The practical applications of the results as shown by the authors are not completely clear.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Training neural networks on large-scale real-world datasets typically involve gradient-based updates of deep networks. Therefore, understanding the learning dynamics of these models is critical from the perspective of interpretability, as well as designing better architectures/training algorithms. In this work, the authors study the convergence of different layers in a model across training and demonstrate that early layers in canonical deep learning architectures converge faster and learn low-frequency components of a target function. A central claim of the hypothesis is that gradient magnitude is insufficient to characterize the learnability across layers and instead proposes the notion of gradient predictability as a proxy for estimating learning progress. With analysis across simplified target functions and realistic datasets/models (CIFAR/Resnets), the authors show that their notion of measuring learning progress is consistent with existing work in the literature. Further, the insights are used to motivate and explain design choices in the context of transformer architecture and the benefits of transfer learning. 
strength_and_weaknesses: Some strengths of the work include the following:
+ The paper is clearly written, with well-grounded motivations and pedagogical examples on understanding the learning dynamics across different layers of canonical neural network architectures. 
+ The authors highlight the limitations of gradient magnitude as a measure of learning progress, instead proposing a proxy that measures the predictability of gradients as learning progresses.
+ The insights are leveraged to provide practical design guidance and explain known phenomena in the literature (transfer learning accelerates training on downstream tasks).

Some weaknesses of the current draft are outlined below:
- The critical observations of the work, i.e., layers converge at different rates and learn different components of the target functions, are well established in the literature. However, it would be helpful to clarify the utility/novelty of the discussion using simplified target functions (Section 3).  
clarity,_quality,_novelty_and_reproducibility: The manuscript is well-written and communicates the core principles and hypotheses clearly. The authors use pedagogical examples of simplified target functions to establish the differences in learning dynamics across layers of a neural network, with observations that are consistent with empirical evidence in the existing literature. The work provides sufficient information to replicate core results from a reproducibility standpoint. Furthermore, the paper presents sufficiently novel results, particularly the notion of gradient predictability that could provide useful diagnostic tools to debug learnability in future architecture designs. 
summary_of_the_review: Summarily, the authors consider the learning dynamics of canonical deep learning architectures through the lens of gradient predictability across network layers. Using simplified target functions (mixture of sinusoids at different frequencies), the authors establish earlier network layers tend to learn lower-frequency components (and converge faster). In contrast, the last layers learn high-frequency components. Experiments across multiple model families and datasets suggest the ubiquity of this finding in convolutional architectures. Finally, these insights are used to guide design choices for better architecture design in the context of vision transformers and explain the efficacy of transfer learning in accelerating learning on downstream tasks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper investigates the observation that, during the training of deep neural networks, lower layers ("shallower", closer to the input) converge faster towards their final value, compared to upper layers ("deeper", further away from the input).
It quantifies that tendency on several architectures, on synthetic data and natural images.
It then draws links with the "spectral bias" of neural networks, showing that lower layers seem to learn functions with lower frequencies (in the input space), and higher layers learn, more slowly, the higher-frequency components (if any).
strength_and_weaknesses: Strengths
-------------
1. Confirms observations of faster convergence in the first layers during training of a DNN, using a quantitative metric
2. Establishes a link between learning lower frequencies earlier, and learning them in the first layers, showing that higher layers converge faster if they don't have any high-frequency components to learn
3. Empirically explores loss landscapes of different layers, for different architectures (FCNN, VGG, ResNets)
4. Exploits these observations to propose a better repartition of parameters for vision transformers

Weaknesses
-----------------
1. I don't think the reason behind these observations have been convincingly proven. The explanations provided are consistent, but may not be the only ones.
2. Previous work should be more prominently mentioned, as these observations are not new.
clarity,_quality,_novelty_and_reproducibility: Clarity
----------
The paper is overall clear, describing in an understandable way the phenomenon observed and the investigative steps taken.

A few potential improvements:
1. The convergence rate in Def 2.1 does not seem to correspond to the convergence curves in most of the following figures.
2. The difference between "first layer" and "first hidden layer" in Sec 3.1 is confusing, usually when a difference is made, that's because the "first layer" is the input one, which does not have any parameters.
3. Figure clarity could benefit from having the setup explicitly stated in the caption (which dataset, which experiment is being performed...), as well as the conclusions and points of interest (e.g., what's happening at Epoch 150 in Fig. 3?)
4. In Def 4.1, is it intended to have $G''_{l,t}$ defined with $\theta^{(t+1)}_L$? Why is $L$ updated, in addition to $1, \ldots, l-1$?


Quality
-----------
The experiments are well designed, and are consistent with the conclusions made. The experiments on frequency (in Sec. 5) and transfer learning (Sec. 6.2) are interesting and insightful. But I'm not sure they support it to the extent being claimed.

1. I don't really agree with naming the investigated phenomenon "bias", although it shares similarities with the "spectral bias" (and seems related to it), it is fundamentally different as it does not directly affect the behavior or predictive power of the network at any given time, it's just related to internal training dynamics. I don't think we can say that the network exhibits a bias (in its abilities, or performance, or predictions) due to this phenomenon.
2. The paper assumes that weights of each layer are converging to a value, but it does not report whether it actually does before training stops (at epoch 100 or so depending on the experiment).
3. The observations are made on typical architectures, known for working well experimentally. It is not clear they are intrinsic to neural network training, correlated to good models and training conditions, or if there is any causality link: are there models where the output layers converge first, but that don't perform well? Fig. 8 (e-h) shows that changing the input distribution can change the convergence behavior. Section 6.1 might be affecting the convergence behavior by changing the architecture, but it's not clear if it actually changes, or this is what causes the performance improvements.
4. The visualization of loss landscape "slices" is interesting, and consistent with the idea of smoother, wider minima in the lower layers, but I don't think it's a definite proof, let alone a causal explanation. I don't think that this being "the fundamental reason behind this phenomenon" is being adequately supported.
5. It would be good for figures 1(b), 8, and 11-13 to show the convergence all the way to $\theta_l^*$, in addition to the zoomed-in portion at the beginning.


Novelty
-----------
The paper clearly states that faster convergence in the first layers has been observed and reported in cited work earlier. I think it would have been better to state more prominently towards the beginning of the paper that it builds on these observations, rather than in the last paragraph before the conclusion.

It is good to have an explicit consideration for skip-connections, given that they're often used in modern deep architectures, but it would have been nice to also investigate batch normalization or layer normalization, at least in the context of ResNets, to see if the findings still hold.

Reproducibility
--------------------
The paper, formulas, and description of experiments are defined clearly enough that someone with knowledge of the field could reimplement and reproduce the results.
However, the source code is not provided with the paper.
summary_of_the_review: This is a really interesting exploratory work, and the explanations advanced are plausible, but not really convincing. It is not clear how much of these findings can generalize to other architectures, or distributions other than natural images and synthetic 1D data.
I'm leaning towards accepting it.

**Update after responses**
Thanks for your answers and clarifications.
While I recognize they add insight, I still think the paper is borderline, and I'm keeping my rating.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents an empirical study trying to explain the differences in speed of convergence of different layers of a neural network. To that end, the authors propose to measure the speed of convergence by tracking the rate of change in distance to the optimum of different layers within a time span. They argue that, contrary to the gradient magnitude, this metric can capture a more relevant notion of convergence speed consistent with different phenomena. In particular, ther experiments seem to indicate that on many training tasks, deeper layers converge slower than shallower ones, and that they fit higher frequency functional terms. The authors then argue that this inherent difficulty of training deeper layers can explain the success of using wider layers deeper in CNNs. Inspired by this intuition they then suggest and, partially test, using this design choice on modern ViTs.
strength_and_weaknesses: # Strengths
1. **Interesting empirical study based on an intriguing phenomenon**: The observation that deeper layers have a slower speed of convergence when training on many tasks is interesting and worth investigating. In particular, I find the following observations and experiments noteworthy:
   a. Connection between rate of convergence of deeper layers and frequency of target function. 
   b. Consistent observation that deeper layers have a rougher loss landscape.
   c. Faster convergence rates on shallower layers and frequency specialization is not an architectural bias, as it changes during training.
2. **Consistent observations on synthetic and real tasks**: I really appreciate the alternation between experiments on synthetic tasks and real tasks in which the controllable experiments are used to isolate a described effect which can then be observed on the more realistic training runs.
3. **Clear writing**: The paper is clearly written and easy-to-follow. The different sections follow naturally from the previous ones and create a coherent storyline.

# Weaknesses
1. **Lack of causal connection to support arguments in Sec. 6.1**: The explanationss about the usefulness of having wider layers at deeper stages of a neural network are not proprely backed by experiments. In particular, no experiment shows that increasing/decreasing the width of a given layer increases/decreases its speed of convergence. Similarly, the claimed benefits of the proposed architectural changes to ViTs are only superficially investigated. To support these claims, a much further investigation than the one provided should be performed.
2. **Results based on single runs**: Most results are based on observations made for a single run and lack studies of sensitivity to different hyperparameters. In this regard, some of the results presented in this study might be a bit anecdotal.
3. **No normalization of gradients**: While the authors soundly argue that their proposed convergence rate metric needs proper normalization to be comparable among layers, they do not seem to care about normalizing the gradients by number of parameters so that they become comparable among layers. If deeper layers have many more parameters than shallower ones, then it is not surprising that the unnormalized gradient norm of the shallower layers is smaller.
4. (Minor) **Convergence rate metric is not strongly motivated**: The reasons behind using this specific convergence rate metric and not the standard convergence rate used in optimization theory are not very strongly motivated. In particular, the short footnote explaining this on page 2 is a bit unclear.
clarity,_quality,_novelty_and_reproducibility: - **Clarity**: The paper is well-written and easy to follow.
- **Quality**: Some of the arguments of this work could be better supported through experiments that test the causal connections between the different argumentss, and investigated deeper using proper ablation studies with several seeds.
- **Novelty**: As far as I know, this study is novel.
- **Reproducibility**: The lack of studies showing the sensitivity of the observations to different hyperparameters, e.g., learning rate schedule, weight decay, etc, might make some of the observations not fully reproducible in other settings.
summary_of_the_review: Overall, I believe this work can be of interest to the community as it provides intuitive insights on an intriguing phenomenon. This being said, I believe these insights coulld be better grounded on certain experiments and isolated with respect to different hyperparameters. If the experiments where more thorough this paper could be a strong contribution. In the current stage, though, I believe this is a borderline paper, slightly above the threshold for acceptance.

----

**Post-rebuttal udpate** : After the detailed rebuttal by the authors where a new improved version of the manuscript has been shared, I have decided to increase my score to an 8, as most of my concerns have been alleviated.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the convergence rate during training for different network layers and finds that earlier layers converge more quickly. They also include experiments indicating that the loss landscape is flatter in directions corresponding to early layer weights than late layer weights and suggest that this is connected to the faster convergence and stability. The paper reviewers found these observations compelling and sufficiently novel. 

That being said, an essential shortcoming of the manuscript as it stands (and echoing reviewer xk3R) is the lack of a thorough related works section. The observation that early layers converge first and are most stable during training has a long history in the literature. Some references for this point include:

https://arxiv.org/abs/1706.05806 – (Main contribution 2 is bottom up learning of DNNs)
https://arxiv.org/abs/2007.07400 – Find that later layers move the most during task adaptation, while earlier layers are stable

I would strongly recommend modifying the paper to expand section 7 and potentially include this in the introduction, clarifying the distinguishing features of this work!

With this modification I am happy to accept the publication!
justification_for_why_not_higher_score: The bottom up convergence has missing prior work: eg https://arxiv.org/abs/1706.05806
justification_for_why_not_lower_score: Quoting: This paper studies the convergence rate during training for different network layers and finds that earlier layers converge more quickly. They also include experiments indicating that the loss landscape is flatter in directions corresponding to early layer weights than late layer weights and suggest that this is connected to the faster convergence and stability. The paper reviewers found these observations compelling and sufficiently novel.
Decision: Accept: poster