Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes setting different learning rates for each layer of a deep neural network such that the first layers (closer to the input) have a much higher learning rate than the last layers. Learning rates are all gradually increased until they reach the same value $\eta_0$ (the optimal learning rate of no-curriculum models). 
strengths_and_weaknesses: Strengths:
- Simple, novel, and effective method. 
- Good review of related work and reasoning for using model-level curriculum learning.
- Consistent performance improvements across eight audio, image, and text datasets and seven convolutional, recurrent, and transformer architectures. 


Weaknesses:
- Insufficient analysis on why the method works. 
- The motivation for setting different learning rates per layer is not sound. The authors state that random initialization of parameters causes the propagation of noise during the forward pass, and that assigning higher learning rate to first layers and lower learning rates to last layers prevents this propagation of noise. But there is no proof for this argument.  

questions: - What is the method for optimizing $\eta_j$, j = {1 ... n}?
- How sensitive is the model to different settings of $\eta_j$?
- How do the training dynamics compare to standard training where all layers are trained at the same rate? Would the earlier layers generate qualitatively different features maps than under-trained later layers?
limitations: - The interaction of the model with adaptive optimizers that scale the learning rate of each parameter is not well justified. 
- Learning rate schedulers may disturb the proposed model. If the base learning rate is reduced before the learning rate of the last layer $\eta_n$ is sufficiently increased, the last layer may never be trained enough, training may take much longer or may become unstable.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The manuscript proposes a model-level curriculum learning framework that uses variable learning rates across a DNN to jump-start the learning process. The core idea revolves around compensating for the difference in learning across the depth of a DNN by using larger learning rates closer to the input and gradually decreasing their value towards the output layers. As intended, this curriculum is data-free and removes the need to assess the difficulty of samples, similar to how data-level curriculum operates, and instead can be applied across a range of architectures and tasks. The proposed idea improves on the performance of standard training regimes across vision, language and audio tasks, using residual, recurrent and transformer-based architectures.
strengths_and_weaknesses: Strengths
- The writing offers a clear and understandable representation of the curriculum learning approach.  More importantly, the categorization of various levels of curriculum learning provides important context to the ideas being discussed.
- The depth of experimental evaluation, stretching across datasets, tasks and architectures is commendable.

Weaknesses
- Data augmentation has proven valuable to the training process, be it commonly used approaches like random cropping/flipping or more intricate mixed approaches. They offer substantial improvement in performance and as well as alternative properties of a DNN. The proposed method is not evaluated alongside any common preprocessing techniques (vision datasets), thus offering a gap in the exploration of the behavior of the proposed algorithm alongside them.
- The set of results posted across various tables in the manuscript is commendable. However, in certain cases the the average performance and standard deviation across multiple methods are extremely close. While this does not take away from the final conclusions, the use of significance tests/metrics could help further distinguish improvements.
- The descriptions of the preprocessing and training regimes offer extremely important context to the results presented throughout the paper.
However, an exploration of the two key parameters that make up the proposed method, various update setting for the learning rate and the choice of iteration/epoch at which to converge to the standard learning rate would offer more insight into the proposed method.
questions: - Specific to the vision datasets and results, could the authors provide insights into (a) how the use of common preprocessing techniques affects the behavior of the proposed approach, and (b) the balance in peak performance achievable in standard training vs. the proposed method when preprocessing is included. While preprocessing techniques add some element of stochasticity, it is important to understand an algorithm in that context.
- The addition of significance tests/metrics would be beneficial to further emphasize the benefits of the proposed algorithm as well as other areas for potential improvement. Could the authors reassess their results using significance metrics?
- Moving a small portion of the experimental setup to the appendices/supplementary material and including a study of how k (iterations/epochs), from small to absurdly large, affects the improvement in performance offered by the proposed algorithm would add another dimension to the evaluations presented in this work. The results themselves should be a by-product of the suite of experiments already executed so hopefully there is minimal overhead.
- Given the variable levels of improvement, even within the scope of the same task, could the authors discuss plausible schemes that take into account other factors (like data, gradient flow, etc.) that could provide more consistent gains?
- At a more abstract level could the authors address the potential negative implications/limitations of their approach.
limitations: While there aren't glaring limitations or issues with the proposed method, the authors can consider potential negative effects in alternative properties of the DNN, as a by-product.
Since there isn't an evaluation of the calibration quality or adversarial robustness of the final solutions, these could be potential issues or limitations to the proposed algorithm. 
Keeping robust/adversarial training or alternative objective functions to classification in mind could lead to unexpected outcomes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a kind of model-level curriculum learning strategy, which assigns higher initial learning rates to shallow layers than deep ones and continues increasing all learning rates until they reach the same value during the training process. It is a model- and task-agnostic approach. To verify its effectiveness, the authors apply it to multiple domains and different neural networks.
strengths_and_weaknesses: Strengths:

1.The proposed strategy concerning layer-wise learning rate is important.

2.The empirical experiments are sufficient and comprehensive to prove the generality and effectiveness of the strategy.

3.This paper is well written and easy to read.

Weaknesses:

1.The relation between this work and curriculum learning is vague.  Curriculum learning is a training strategy that learns from easy to hard, or more generally, in a certain kind of meaningful order. For example, CBS, which is mentioned in the paper, anneals the standard deviation of Gaussian kernels to pass more high-frequency information. However, the relation between this work and CL is not so clear.

2.The discussed related work is not sufficient. Since this work focuses on the layer-wise learning rate, it should be discussed more clearly about how this work is different from the works about layer-wise(or adaptive) learning rate[1][2][5], which is a common and widely adopted strategy. Additionally, training with increasing learning rate in the early stages is a kind of warmup[3][4], the relations between this work and warmup strategies are required to discussed.  The vague position of this work makes its contribution vague.

3.This paper lacks theoretical analysis and does not explain why increasing the learning rates until reaching the same value is reasonable. It is not convincing only with the empirical results. The designed mechanism is neither intuitive nor well theoretically supported, making its justification questionable. The only way for justification is through experiments, which may be tricky, because the proposed method needs more hyper-parameters than that of the conventional methods.

[1] Singh, B., De, S., Zhang, Y., Goldstein, T., & Taylor, G. (2015, December). Layer-specific adaptive learning rates for deep networks. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA) (pp. 364-368). IEEE.

[2] Ginsburg, B., Gitman, I., & You, Y. (2018). Large batch training of convolutional networks with layer-wise adaptive rate scaling.

[3] Gotmare, A., Keskar, N. S., Xiong, C., & Socher, R. (2018). A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. 

[4] You, Y., Gitman, I., & Ginsburg, B. (2017). Large batch training of convolutional networks. 

[5] Adam:A Method for Stochastic Optimization. ICLR 2015

questions: 1. There are many hyperparameters of this strategy, so I’d like to know how to choose proper initial learning rates for all layers and how does the sensitivity of hyperparameters affect the experimental results?
2. For pre-trained models, this paper states that “The same issue can occur if the weights are pre-trained on a distinct task, where the misalignment of the weights with a new task is likely higher for the high-level feature layers. ” However, it is generally believed that the learning rates of the layers close to the outputs are relatively high and the learning rates of the layers close to the inputs are low or even equals to zero in the process of fine-tuning, which seems to be the opposite of the strategy proposed in this paper. How to explain the difference between them?
3. It may be easier to understand Eq. (8), if the authors can plot a figure for μ(j).
4. Whether assigning higher learning rate to the deeper layers can also obtain better performance through hyper-parameter tuning ? The learning landscape of the models is not clear.
limitations: The authors do not discuss the limitation. However, there is a lack of both an intuitive explanation or theoretical analysis for the proposed method, making it hard to be a convincing work. A large line of related works is missed, making the contributions vague.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to modify neural net training by applying a different learning rate to each layer: the deeper layers are assigned a lower learning rate that progressively grows to match the first layer learning rate. 
strengths_and_weaknesses: The paper is clear, the approach is simple and can be implemented easily. The experiments shows better generalization when adopting the proposed learning rate schedule. The paper could be improved on two main axes:
(i) presenting the approach as an optimization method seem more appropriate than the curriculum learning angle, (ii) more analysis and justifications of why the method works seem necessary.
Overall, I feel that the paper does not explain how/if the proposed differentiated learning rate schedule is different from the effective learning rates reached by different settings of adaptive optimizers (momentum, Adadelta, Adam, SM3...).

questions: == Optimization vs curriculum learning ==

The proposed method uses a different learning rate per layer. This endeavor has been widely explored in the literature on optimization of neural nets while the name curriculum learning is usually employed for methods modifying the ordering of the examples during training. I agree that the name curriculum could make sense in your context but the paper needs to spend more time on previous work on optimization. This means expanding the related work section and the empirical analysis.

The Adadelta paper (Zeiler 2012) observes the benefit to use different learning rates for different layers and Adadelta results in "effective learning rates [...] are larger for the lower layers of the network and much
smaller for the top layer at the beginning of training." Similar observations can be made for other adaptive learning rate algorithms like Adagrad, Adam, Adafactor, Adamax and AdamW or even later algorithms like e.g. SM3 (Memory-Efficient Adaptive Optimization, NeurIPS'19, see Figure 1.).

To justify your approach it seems necessary to compare your method at least with Adam and report the effective average learning rate (e.g. see Zeiler 2012's definition) per layer for both methods and how their parameters control the differentiation of learning rates across layers during training. The fact that your method is used in conjunction with Adam (or variants) could mean that either (i) the same type of learning rate schedule could be attained by changing Adam's parameters or (ii) your method's benefit comes from schedules unreachable by Adam. It seems very important to explore both methods' parameters to distinguish these two cases.

== Benefit is faster training or better generalization? ==

Modifying the training algorithm requires reporting training curves with *both* validation and training loss over time. The paper does not report any training loss which left us wonder if the generalization benefit comes from better optimization (lower training and validation loss) versus a better generalization trajectory (reaching a lower validation loss at the same training loss). 

== Schedule Tuning ==

Could you explain how the exponential scheduler of Eq. 8 has been selected? Is it the only option that you tried? How is the training speed affected by the choice of the additional hyperparameters (c, \nu^{(0)}_n, k)? In particular, compared to a robust baseline like Adam, would most hyperparameter choices yield faster or slower training? I do not understand how c can be set to 10 without validation.

== Number of training steps ==

The experimental section should spend more time describing how the number of training steps was selected. In which cases the baselines would reach the same or better validation loss with more steps? E.g. Figure 1.d. seems to suggest that better validation loss with CBS can be reached with more training. This improvement does not seem to agree with the sentence "all models are trained with early stopping", does it?

== Justification of the noise argument ==

Adaptive methods (Adam etc) justify higher learning rates for first layers with a curvature argument: in laymans term, to get the same impact on the output the first layers' updates require a larger learning rates than the last ones. Adadelta (Zeiler 2012) also mentions the larger learning rate in light of the vanishing gradient effect (Bengio 2014). Your motivation (from [21]) is to reduce the effect of "noise due to untrained parameters". This argument is unclear to the reader and it would help if you could find a way to measure/characterize that "noise" and show how your method reduces its effect.

limitations: This work does not raises concerns on societal impact.
review_title: null

Meta-review: metareview: The paper proposes a model-level curriculum learning strategy, which assigns higher initial learning rates to shallow layers than deep ones and continues increasing all learning rates until they reach the same value during the training process. It is a model- and task-agnostic approach.

Reviewers appreciated the simplicity of the approach, as well as its effectiveness on a multiple domains and different neural networks.

Main concerns which remains after the rebuttal are:
- There is an insufficient analysis on why the method works. Some intuition has been given in the rebuttal, but all reviewers felt more analysis should be given to reach NeurIPS standards.
- There is insufficient comparison with previous work on optimization.

In addition, a more minor concern (in AC's opinion) remains:
- On vision data, evaluations using augmentations are missing. While it can be argued that the effect of augmentations and LeRaC might be orthogonal, it remains unclear how one can improve on strong baselines there.

confidence: Less certain
Decision: Reject