Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the parameter redundancy issue in large transformer architectures. Instead of pruning redundant parameters, it strengthens training them to make them contribute better performance. To this end, it proposes an adaptive learning rate algorithm SAGE, which automatically scales the learning rate for each parameter based on its sensitivity. The sensitivity is approximated by the dot product between parameters and their gradients. An exponential moving average is used to track the sensitivity scores to reduce uncertainty in mini-batch training. The algorithm is applied to fine-tuning pre-trained transformer models in benchmarks for natural language understanding (NLU), neural machine translation (NMT), and image classification.
main_review: Strengths
1. The paper is well-written and easy to follow. The method section provides helpful intuitions behind the algorithm design.
2. The idea is interesting since it explores a different direction in dealing with redundant parameters.
3. Experiments use multiple benchmarks, including both language and vision, and results show noticeable performance improvements.
4. SAGE is orthogonal to the existing adaptive gradient methods. Jointly using them can bring more gains.

Weaknesses
1. Table 1 shows that models with different percentages of redundant parameters have similar performance, implying that performance may not be proportional to "well-trained" parameters. It means that making more parameters "well-trained" does not necessarily improve the performance. This seems not totally in line with the paper's motivation.
2. Figure 5 is not straightforward to visualize the difference. The curve is drawn only on the right subfigure.
3. Sufficiently training all parameters seems a double-edged sword. According to Figure 3, models trained with SAGE are susceptible to parameter pruning. In general, we want efficient and compact models in deployment. Will SAGE be harmful to developing efficient deployment models?
summary_of_the_review: This paper proposes SAGE, an adaptive learning rate schedule to train redundant parameters more sufficiently to improve model generalization. SAGE, together with existing adaptive optimizers, shows effectiveness in a wide range of downstream tasks. The main concern is whether SAGE has adverse effects in getting efficient deployment models.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper argues that redundant or "useless" parameters are not an axiom we should take for granted, but rather a symptom of current optimization settings. The authors propose an adaptive learning-rate schedule that specifically aims to eliminate redundant parameters. Through extensive experiments on fine-tuning transformers, they show that indeed this method (SAGE) does reduce redundancy and also slightly improves results.
main_review: I think that overall the paper is good as it raises and studies an important point: redundancy in parameters is not an axiom we have to accept.
However, after this introduction and motivation, it reads more like a typical optimizer paper introducing a new optimizer based on hand-wavy intuitions. This is unfortunate and not rigorous.

What follows are detailed comments:
1) The use of Taylor approximation is only good close to the operating point for nonlinear functions. However Theta_j,-j may be very large, and thus the approximation may be very bad.
2) I do not agree that the memory and computational costs of SAGE are "marginal". The EMA "I-hat" is a full copy of the model since we need one EMA per parameter, especially for large models this is substantial overhead. I would further appreciate timings to verify that indeed the computational overhead is "marginal".
3) It is not clear to me in what scale the quantities U and I-hat are, and hence, the multiplier to the learning-rate. Because of this, it is not clear how one needs to change the original learning-rate "eta" when one turns on SAGE.
4) I am confused by Fig5, it does not look like the two-moon dataset at all to me? Please image-search "two moon dataset" and compare.

minor nitpicks:
5) Fig1: Percent should be [0,100].
6) I do not agree that one can conclude from the results that SAGE is more effective for small datasets than for large ones, since the "points" betwen the datasets/tasks do not live on the same scale.
7) All experimental evaluation is about fine-tuning of already pre-trained transformer models. This should be more accurately reflected in the title, for example by replacing the word "training" by "fine-tuning". It would further be interesting to see how well it works when training from scratch; even if it does not work, it is still a valuable fine-tuning method, and stating this may save a lot of people a lot of resources and time.
8) In Figure 6, since we have per-parameter learning-rates, which one is shown for SAGE?
9) Typo: "smoothier" -> "smoother"
summary_of_the_review: For an "optimizer" paper, it is weak: hand-wavy motivation, no real derivation of update rule or even convergence proofs, and experiments only in one very specific domain: transfer of pre-trained transformer models. I am not convinced that it will be generally useful at all.

However, the paper raises a very important point: we should not take redundant parameters as a necessity, and it does propose a method to avoid them. I find this point important enough to still suggest acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method for scaling the learning rate during training in order to encourage all parameters in a neural net to be fully used. Specifically, the updates for parameters are scaled in inverse proportion to how much they affect the loss (their sensitivity); the scaling factor also depends on how stable the estimate of sensitivity is, so that high-sensitivity parameters whose role changes rapidly aren't down-weighted as much. This technique is shown to improve the performance of Transformer models on several different problems, even when used in combination with other adaptive learning rate methods (Adam, Adamax). Analysis experiments verify that the method has the intended effect: compared to the baseline, more parameters have higher sensitivity, and pruning is less effective.

main_review: Strengths:

- The idea is novel, straightforward, well motivated, and easy to implement. Computational cost is low.
- Potential for large impact on model training best practice.
- The experiments are very thorough, demonstrating gains across different settings, and showing that the technique achieves its goal.
- The paper is extremely clearly and carefully written.

Weaknesses:

- The scaling formula seems somewhat ad hoc and hard to characterize. In particular, if the sensitivity of some parameter spikes on a given iteration, it will get a larger update than another parameter with the same moving-average sensitivity that has not spiked.
- There are some hints that the gains might be partly related to regularization (eg, better results on IWSLT than WMT). It would have been good to test the combination with dropout or something similar.
- The hyper-parameters for SAGE are exhaustively tuned, more so than for the baseline adaptive optimizers, so there’s a potential for bias. Figure 7 counters this possibility, but only by showing heatmaps, so it's hard to gauge the actual numbers.

Details:

- Table 3 should show results for your implementation of Adamax that’s comparable to Adamax-SAGE, in addition to the Devlin et al numbers.
- Figure 2: Consider flipping these plots to the standard convention, and stacking them.
- Figure 4: Why do we care that local temporal variation is lower in SAGE than the baselines?
summary_of_the_review: A simple adaptive learning-rate formula that seems to work really well, even on top of other optimizers, as demonstrated by very thorough experiments. Downsides are that the formula isn't particularly well justified, and there are a few potential experimental weaknesses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes the SAGE method. The idea is that neural networks have redundant parameters. Some approaches will prune these parameters which has the effect of not decreasing the performance but to decrease the number of parameters. The paper studies if it is possible to learn better these parameters in order to make them more useful for the network. The paper proposes a method that will allow to train differently the parameters of a network in order to have a better use of the weights. The paper is evaluated in NLP and image classification with transformers.

main_review: Strengths:

- **Writing:** The paper is well written and easy to follow.
- **Tasks:** The paper evaluates its method with different optimisers and on different tasks in NLP and image classification.
- **Results:**  The results are quite good; the proposed method surpasses the baseline each time.

Weakness:

- **Architecture:**  The method is evaluated only with transformer architectures. In the context of image classification it would be interesting to evaluate it with CNN and others architecture than ViT.

- **Optimisation:** In image classification, the pre-training procedure used is quite sub-optimal since the paper of Dosovitsky et al. [1] many improvements have been proposed such as the DeiT approach [2]. It would be interesting to see if the proposed method still works when the model is trained with more regularisation and data-augmentation which may also lead to a better use of weights.

- **Image Classification:** Having only results of models pre-trained on ImageNet-21k and fine-tuned with the proposed method on downstream tasks is not very usual in image classification. It would be interesting to have results on ImageNet only where the SAGE method is used during the training.

[1] Dosovitskiy et al, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021

[2] Touvron et al, Training data-efficient image transformers & distillation through attention, ICML 2021
summary_of_the_review: The idea of the paper is interesting and the proposed method seems effective. Nevertheless, more complete experiments in image classification would allow to better evaluate the interest of the proposed method.
review_title: null

Meta-review: The paper observes that the number of redundant parameters is a function of the training procedure and proposes a training strategy that encourages all parameters in the model to be trained sufficiently and become useful. The method adaptively adjusts the learning rate for each individual parameter according to its sensitivity (a proxy for the parameter's contribution to the model performance). The approach encourages the use of under fitted parameters while preventing overfitting in the well-fitted ones. Experimental results are presented covering a wide range of tasks and in combination with several optimisers, showing improvements in model generalization.

The paper is very well written and easy to follow (as mentioned by Reviewers NSqH, 4pzE and sSHP). 

The authors provided a strong rebuttal including new experiments, like training using CNN based architectures (as requested by Reviewers sSHP and MzBV). Reviewer sSHP requested these results to be reported with STD, the AC encourages the authors to do so for the camera ready.

Reviewer MzBV points out that the paper could be improved by giving a motivation of the update rule and proving convergence. However, still recommends accepting the paper due to the novelty in the idea of not taking redundant parameters as something inevitable and devising an effective strategy to improve it. This idea was also appreciated by the other reviewers. While the AC agrees that adding these points would improve the work, it takes as valid the point made by the authors. Namely, that the intuition behind the update rule is quite clear, and many other reasonable variants were ablated (in Appendix A.4.4). Furthermore, the empirical evidence shows that the method improves generalization.

Reviewer NSqH points out that while SAGE improves the model’s generalization performance for lightly compressed models, its performance becomes more susceptible to pruning when the model is compressed heavier. While the authors responded with good points, the AC encourages them to follow the reviewer’s advice and incorporate further experiments studying this issue (e.g. other datasets).

In sum, the paper proposes a simple and effective method that is able to improve generalization of large scale models. All four reviewers recommend accepting the paper. The AC agrees and encourages the authors to incorporate the requests mentioned above.
Decision: Accept (Poster)