Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the performance gap between Adam and SGD at different batch sizes. It is found that the gap still exists with large batches or even full batches, which suggests the distribution of random noise may not explain this gap. By contrast, it is shown that Adam with large batches is similar to sign descent.
strength_and_weaknesses: This paper studies the important problem of why Adam performs better than SGD (especially on language tasks), and provides interesting results. Prior work argues that the performance gap might be due to heavy tail of the noise distribution; however, this paper shows that heavy-tailed-ness  may not be enough to explain the performance gap, as it still exists even with full batches. Moreover, as the batch size increases, the SGD performance almost always degrades, while Adam can sometimes do better. Finally, with full batches, it is shown that sign descent can sometimes match the Adam performance. These findings can help us understand Adam better and design more efficient algorithms.

On the other hand, I have the following suggestions:
1. I think more hyperparameter tweaking is needed (e.g., some simple learning rate schedule). The reason is to make sure the results still hold when we have satisfactory test accuracies. For example, in Figure 11, the best test accuracy obtained for ResNet18 on CIFAR10 is below 80%, but I believe it should be easy to get 90%.
2. In Figure 5, to support the claim that Adam is similar to sign descent, it is also nice to compare the test error for Adam and sign descent.
2. For sign descent, in addition to moving averages of signs, it might also be interesting to consider signs of moving averages.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written.
summary_of_the_review: This paper provides interesting results, that the performance gap between Adam and SGD still exists with full batches, and that Adam is similar to sign descent. On the other hand, I think more experiments can be tried to make the story more complete. Therefore currently I put this paper marginally above the acceptance threshold.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work studies whether the heavy-tailed nature of gradient is the true cause of Adam outperforming SGD, as suggested by prior work (J. Zhang et. al., 20). The paper conducts an extensive set of experiments involving decreasing noise level of SGD/Adam via increasing the batchsize from small batch all the way to fullbatch GD. The paper makes three main observations: 1. Noise does not explain the gap between SGD and Adam since Adam still has lower training loss with full batch gradient, 2. SGD’s performance does not improve as the batchsize increases, which is a more granular version of claim 1, and 3. Adam’s success might be attributed to its behavioral similarities with signed gradient descent.
strength_and_weaknesses: ### Strength
- The paper’s motivation to establish the correct assumptions for theoretical work on optimization is refreshing and extremely important.
- The experimental procedure is carefully designed

### Weakness
In my opinion, the major weakness of the paper is overclaiming, especially with regard to the claim about refuting “long-tailed gradient error is why Adam does better than SGD”. I will comment on each of the claims made individually.

**Claim 1**: comparing figure 1 and figure 2, it does seem Adam benefits from the full batch more than SGD. This claim seems well supported.

**Claim 2**: This claim only seems to hold for a subset of the experiments and does not say anything about the long-tailed nature of SGD. Further, even in Figure 3, I am not sure how I should reach the conclusion “SGD does not take advantage of the reduction in noise while Adam does”. On Squad, it seems that SGD is able to take advantage of the increased batchsize. Similarly, on PTB, SGD seems to do better when the batch size increases. The argument seems strenuous at best. Finally, in Figure B.2, it seems that Mnist and Cifar10 exhibit very different behavior even though both are supposed to not have long-tailed gradient noise according to figure 1. This casts further doubt on the claim about long-tailed gradient error.

**Claim 3**: This claim does not seem supported by the figures. In figure 4, I am not sure how to get “signed GD scales better than normalized GD” from the plots. In particular, it seems to me that signed GD since the larger batch sizes have higher training loss. If I am misinterpreting the plots, please correct me and add appropriate instructions about how to read the plots.

Overall, I find the plots to be difficult to read even with the help of the text. The addition of two sets of experiments for w/ and w/o momentum makes the matter worse. The momentum and no momentum plots (e.g., figure 3) should ideally be separate from each other. The dashed lines look really bad presumably because of the fact that the lines are sometimes not smooth. I would rather see a plot of the final performance against the number of each batch size rather than seeing their progress which doesn’t add anything. I think it would be more helpful to quantify how much the batchsize is correlated with training loss rather than commenting on it qualitatively.

More importantly, I believe that the paper did not fully disprove the hypothesis about heavy-tailed gradient error. In particular, I believe that what the paper showed is that “gradient noise does not explain the gap between Adam and SGD '' by showing that there is still a gap between the two even using fullbatch GD. This is somewhat a subtle point, because the paper didn’t convincingly address the issue of heavy-tailedness noise but instead removed it altogether. There exist alternative hypotheses. “Full-batch improving adam” and “heavy tail worsening sgd” can coexist. For example, noise helps gradient descent in general but SGD performs worse when the noise is heavy-tailed, or gradient noise worsens Adam, but when there is noise, Adam performs better if the noise is heavy tail. If I am wrong about this, I would be more than happy to learn how the current set of experiments do not support these possibilities.

Given how much the paper focuses on the heavy-tailed gradient noise, I believe the experimental results do not fully support the claim. The findings are still interesting and it supports the claim that the difference between Adam and SGD is deterministic, but I don’t think they fully preclude the heavy-tailed hypothesis. I would like to emphasize that I do not believe in the hypothesis myself but I believe that it is important to be careful and rigorous in scientific claims. As such, I would suggest the authors rephrase the introduction and title accordingly to reflect subtleties.


clarity,_quality,_novelty_and_reproducibility: **Clarity** Some part of the paper are not clear (e.g., plots)

**Quality** the paper has some problems such as overclaiming

**Novelty** the paper is novel

**Reproducibility** Seems good.

summary_of_the_review: The paper makes interesting findings but makes some overclaims. I would increase my score to 6 or 8 if the authors can correct these overclaims and fix other issues that I mentioned above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors show that across a range of tasks, the performance of Adam improves more as the batch size rises than the performance of SGD. In the full batch limit, Adam continues to significantly outperform SGD on language tasks. Based on this observation, they argue that the benefits of Adam are best studied in the full-batch/deterministic limit, and that key benefit of Adam is therefore not better robustness in the presence of heavy tailed gradient noise.

Additionally, they observe that sign descent also improves considerably as the batch size rises, and that while it performs poorly for small batch sizes it achieves comparable performance to Adam on some tasks in the full batch limit. They therefore propose that studying sign descent in the full batch limit may be a useful toy model for understanding the benefits of Adam.
strength_and_weaknesses: Strengths:
1) The paper tells a clear story, and makes a convincing case that the key differences between Adam and SGD are best studied in the large batch/low noise regime.
2) The observation that sign descent outperforms gradient descent in the full batch limit is quite interesting, and provides a plausible intuition for the success of Adam.

Weaknesses:
1) The sign descent experiments are not very convincing. There is still quite a significant gap between sign descent and Adam in the large batch regime on some datasets (eg Figure 5 wikitext/Squad).
2) The authors note that the gap between SGD and Adam primarily arises on language/transformer tasks, however this observation is not explored/discussed.
clarity,_quality,_novelty_and_reproducibility: Clarity:
The overall layout of the paper is very clear, and I appreciated that the paper has a clear narrative. However I would encourage the authors to improve the presentation of the figures. I found it difficult to quickly determine which curve corresponded to which batch size/algorithm. The text is also quite long-winded in places, and I'd encourage the authors to make the writing more succinct.

The performance of Adam and Sign Descent is compared in Figures 3/4 without plotting both curves on the same plot.

Quality:
I think the paper does a good job of conveying a simple core message with well chosen experiments. It was a shame that the authors did not study sign descent in more detail.

Novelty:
The core ideas have mostly appeared in prior work, but since the paper is primarily empirical I think that is fine.

Reproducibility:
I think the paper could be reproduced

Other comments:

1) Why does the "Bad starting assumptions can lead us astray" paragraph appear in the intro? It didn't feel very relevant to the paper to me.
2) On heavy tailed noise: note that heavy tailed noise likely arises when the Hessian is poorly conditioned, and this may also explain why sign descent outperforms gradient descent. Ie heavy tailed gradients and sign descent outperforming gradient descent may well have a common cause.
3) Zhang et al. (https://arxiv.org/abs/1907.04164) previously showed that Adam scales to larger batch sizes than SGD, while Zhang et al., Smith et al. (https://arxiv.org/pdf/2006.15081.pdf) and Shallue et al. (https://arxiv.org/abs/1811.03600) all showed that Momentum scales to larger batch sizes.
4) Related to point 2, Zhang et al. and Smith et al. both argued that batch size scaling is connected to the conditioning of the Hessian. It would be nice to see some discussion of this/other explanations for the success of sign descent, especially with regards to the differences between vision and language tasks.

summary_of_the_review: I think this paper makes a valuable contribution to the field. However I think it could be significantly improved by improving/extending the experiments and discussion of sign descent. I therefore will score it weak accept for now.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: **Overview of this paper**

This paper studies why adaptive methods perform better than SGD in terms of convergence. Specifically, this paper aims to explore the hypothesis proposed by (Zhang et al., 2019): Adaptive methods including Adam converge faster than SGD due to the more robust gradient estimate of adaptive methods (abbreviated as the adaptivity-to-noise hypothesis latter). Through extensive experiments, it is observed that the convergence gap between Adam and SGD even gets larger when the batch size increases (and thus the noise decreases), and thus the adaptivity to the noise of adaptive optimizers can not fully explain the gap.

**Summary of the experiment observations**

(1). Over language tasks, the training loss gap between SGD and Adam (both best tuned) gets larger when the batch size increases.

(2). Adding momentum improves the performance of optimizers, mainly in the case when the batch size is large.

(3). The performance of SignGD in the small-batch regime, but improves significantly with respect to batch. Over some tasks (e.g., PTB), the performance of full-batch SignGD is comparable to that of full-batch Adam.

(4). Normalized gradient descent also scales better with batch size than plain gradient descent, but less so than sign descent. Full-batch normalized gradient descent performs worse than full-batch SignGD.



**References**

Zhang et al.,  Why are adaptive methods good for attention models?, 2019
strength_and_weaknesses: **Strength**

1. The question that the paper aims to study may be of interest to the optimization community. The adaptivity-to-noise hypothesis is impactful and its correctness is worth studying. Ruling out this hypothesis also indicates that new hypotheses need to be considered.

2. The experiments are extensive and solid, thus convincing.

**Weakness**

1. While disproving the adaptivity-to-noise hypothesis, this paper does not provide a new hypothesis/conjecture. It is unclear why signGD works well in the large batch regime and why signGD works badly in the small batch regime.

**Questions**

In (Zhang et al. 2019), it is hypothesized that adaptive methods perform better than SGD since the smoothness is not uniformly bounded but can be controlled by the gradient norm. It also proves that normalized GD can adapt to such a landscape and converge arbitrarily faster than SGD. If this was the key mechanism of the acceleration effect of adaptive methods, then clipped SGD should perform as well as Adam in the full-batch case. However, this paper shows that normalized GD performs much worse than Adam in the full-batch case. Does this mean this hypothesis is also wrong?

**References**

Zhang et al. Why gradient clipping accelerates training: A theoretical justification for adaptivity, 2019

clarity,_quality,_novelty_and_reproducibility: **Clarity**

The writing of this paper is clear and I am able to the key information without much effort.

**Novelty**

To the best of my knowledge, this is the first work investigating the correctness of the adaptivity-to-noise hypothesis.
summary_of_the_review: This work is the first to investigate the correctness of the adaptivity-to-noise hypothesis, which is an impactful hypothesis for why Adam converges faster than SGD. While this paper does not provide a new hypothesis, it disproves the adaptivity-to-noise hypothesis through extensive and sound experiments. I believe this result is of interest to the optimization-in-deep-learning community, and lean on acceptance of this work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper is on borderline. All the reviewers felt the paper considered a challenging problem of figuring out the gap between SGD and Adam optimizers and did detailed experiments to challenge the heavy tail noise hypothesis. However reviewers felt the claims of the paper are over the top and not well supported by the current results, 1- Experiments suggest other factors beside heavy tail noise for gap between the optimizers, they don't completely rule out this hypothesis, 2- SignSGD results are not consistent with Adam at all batch sizes. Authors indeed agree and promised to tone down their claims. I am suggesting acceptance assuming authors will honor their response and tone down their claims in the final version.

Pros -
1. The paper presents detailed set of experiments comparing SGD and Adam for larger batch sizes to separate the mini batch noise effects.
2. Results show other factors at play besides the noise in the difference between SGD and Adam

Cons -
1. Over the top claims that are not supported by the experiments.
2. Unclear results around relation between SignSGD and Adam
justification_for_why_not_higher_score: No explanations or new hypothesis that can explain the gap between the optimizers.
justification_for_why_not_lower_score: Interesting set of experiments trying to understand the gap between SGD and Adam
Decision: Accept: poster