Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes an adaptive optimizer, ACProp, which combines centering of second momentum and asynchronous update. The authors show that ACProp has a convergence rate of $O(\frac{1}{\sqrt{T}})$ in a stochastic non-convex setting, outperforming the  $O(\frac{\log T}{\sqrt{T}})$ of RMSProp and Adam in the same setting. Experiments on a range of tasks, including image classification, training GANs, reinforcement learning and machine translation, show the effectiveness of ACProp.
main_review: This is good paper with both theoretical results and comprehensive experiments. However, I still wonder how much ACProp could help in practice. To be detailed below.

Strengths:
1. ACProp converges under a wide range of hyperparameter settings on the toy problem in Eq. 1 and Eq. 2. The convergence rate is better than Adam and RMSProp in the same stochastic non-convex setting.

2. Empirical results are verified on a range of different tasks.

Weaknesses:
1. Lack of comparisons with AMSGrad. AMSGrad already stabilizes Adam for Eq. 1. I wonder if it also solves Eq. 2. It would be even better if AMSGrad is also considered in other experiments.

2. The main improvement of convergence rate in the stochastic non-convex setting seems to be from the asynchronous update scheme proposed in AdaShift, rather than a combination of the asynchronous update scheme and centering second momentum. The rate of ACProp is only better than AdaSfhit by a constant.

3. I feel it is quite difficult to provide a 100% fair comparison for optimizers (finding the best hyperparameter for each optimizer), but this is important unless the proposed optimizer already shows a significant improvement over the best result of a well-established, challenging benchmark. Therefore, I have doubts about many experimental results. For example, in Figure 7, the learning rate schedule does not seem to be optimal for every optimizer. What if a cosine annealing schedule is used? Is SGD still not as good as adaptive methods? In Table 4, why is AdaShift much worse than ACProp? The only difference seems to be the second-momentum centering. However, with this technique, AdaBelief did not make such a huge improvement over Adam (e.g., ACProp's 99.32 to 43.43 vs. AdaBelief's 49.29 to 47.25).


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposed a new adaptive optimizer, ACProp, which combines centering of second momentum and asynchronous update. In essence, ACProp modifies an existing method AdaBelief by using s_{t-1} instead of s_{t} as the update denominator. It has been shown that ACProp has a weaker convergence condition in the example by Reddi et al. (2018). They also proved that ACProp has the same convergence rate of O(1\\sqrt{T}), which matches the oracle rate for the stochastic nonconvex case.
main_review: The proposed method is very similar to AdaBelief. The only difference is that ACProp uses s_{t-1} instead of s_{t} as the update denominator. It’s not clear how much difference there is between s_{t-1} and s_{t} during most period of the training procedure (i.e., when training loss curve starts to enter the flat part and only slowly decreases), especially given that \beta_2 is normally set to a value that is very close to 1 (e.g., 0.99 or 0.999). 

What’s more, although the theoretical analysis of the proposed method ACProp looks interesting, I am concerned with how significant the change from AdaBelief to ACProp is in terms of the practical performance. The theoretical analysis has shown that ACProp is better than AdaBelief in some cases (e.g., weaker convergence condition in the example by Reddi et al. (2018)). However, in practice these cases might not be common enough to set ACProp clearly apart from AdaBelief. Actually, from the experiments in this paper, I tend to have the impression that ACProp has very close performance to AdaBelief in many cases (sometimes even worse than AdaBelief: see the row JA-EN in Table 3 and the row RLGAN in Table 4). For example, in Figure 7 (left and middle), blue curve (ACProp) and orange curve (AdaBelief) reach identical final test accuracy. In Table 2, the difference between AdaBelief and ACProp is only 0.38% (not sure how much of it might come from random noise), while the difference between AdaBelief and Adam is much larger, i.e., around 3.5%. Results from Table 3 and Table 4 also seem to suggest that ACProp has very similar performance with AdaBelief in most cases. 

Given the above reasons, I am concerned that ACProp is just an incremental improvement over AdaBelief in practice.

limitations_and_societal_impact: Limitations have been discussed in Section 3 and 4. 
No negative societal impact is applicable.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work studies the problem of adaptive gradient methods and it is relevant to the conference. The paper provides momentum centering and asynchronous update for adaptive gradient methods that combines several existing works. The paper prove its algorithm the standard $1/\sqrt{T}$ convergence rate for stochastic nonconvex case. There are some experiments on VGG + CIFAR10 associated with the paper, thus the practical performance of the approach is also validated.



main_review: The main contribution of the paper is the offering of momentum centering and asynchronous update for adaptive gradient methods. The problem is well-motivated and the writing is clear. There are some simulation results associated with the paper thus the proposed algorithm seems work from both the theory and simulations.
 
In short, the paper is technically sound and the developments are clear. The derived algorithm and rates seems to be correct and may be a useful contribution to the literature on adaptive gradient methods, showing a modest improvement over the state of the art. ​ 
limitations_and_societal_impact: There is no discussion on the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to combine centering of second momentum with asynchronous update for adaptive learning rate optimizer, showing that the proposed optimizer has: (1) a weaker convergence condition on sequence online convex optimization problem than its counterparts; (2) a convergence rate that matches the oracle; (3)  strong empirical performance in terms of generalization and training stability.
main_review: This paper is overall well written and has a moderate theoretical contribution. However, the experiments can be conducted and/or explained better to help understand the results. Besides, some points need to be clarified.

Issues:

- It should be mentioned that ACProp in the experiments does not use momentum, i.e., mt is only used for the estimation of st. 

- Given the previous point, the comparison between ACProp and other optimizers can be done better, e.g., disable momentum for other optimizers. Otherwise, it is hard to conclude whether centering is the key to better empirical performance.


- The learning rate used In Figure 6 (a) and (b) should be given. Given that centered version generally has a large effective learning rate at the beginning, using a same learning rate for centered and uncentered would be inappropriate. If it is using a same learning rate, it might be better to tune the learning rate such that the curves in the initial stage basically overlap.


- It is unclear why the proposed optimizer leads to better generalization. It is better to have some investigations and/or explanations in the paper.

- It is interesting to show how rectification (RAdam) and/or decoupled weight decay (AdamW) work when combined with ACProp and other optimizers. It might help to understand the experiments, e.g., what are the fundamental issues in these optimizers and how the issues are solved.


Details:

- Figure 8, there are typos for the range of eps. It is also not inconsistent with the correlated figure in the Appendix.


- It is better to mention that AdaShift assumed the independence of gradients, which holds if minibatches are randomly sampled and is violated in the sequence online convex optimization problem.

**Score increased from 6 to 7, after rebuttal.**
limitations_and_societal_impact: N/A
review_title: null

Meta-review: I recommend acceptance. The work build on top of existing work and clearly (even if moderately) improves some theoretical and empirical aspects.

I applaud the authors for their great communication with the reviewers that were responsive during the discussion period. Their responses have been extensive and convincing. They have gone to great lengths to respond to all timely requests for extra experiments and hyper parameter tuning.

I acknowledge that last-minute requests for heavy experiments cannot be reasonably accommodated and I accept the provided experimental evidence as sufficient for the paper’s main claims.
Decision: Accept (Poster)