Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes to study some weaknesses of Adam and AMSGrad and propose a new method called AdamAL that is evaluated on CIFAR10.

I am not from this area, but unfortunately I find this paper to not be rigorously written or organized. Section 3.3 for instance which discusses the non-alignment projection issue with AMSGrad is not rigorously written. There are no proofs to any theorems and even some of the theorems/corollaries are not written rigorously. 

Organization-wise I feel it is difficult to see where the paper is going and some sort of outline / notation box would help. 

Empirically, there are results on one dataset CIFAR10 that shows the author's proposed variant AdamAL works better. However, it is not the detailed types of experiments I was expecting of a paper that points out specific issues in AMSGrad. Shouldn't the experiments be showing these weaknesses in some sort of controlled setting? 

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper analyzed a few issues of Adam, and proposed a new variant of Adam called AdamAL.  

There are quite a few issues: 
--It is not clear whether the heuristic observations are useful. 
--Theorem 3.1 does not even lead to the convergence of the algorithm. 
--The simulation does not show the advantage. 
--There are too many typos and grammar errors.

1) Heuristic observations.
One main observation is that v_{t+1} = beta_2 v_t + (1 - beta_2) g_{t+1}^2 in the original Adam will be different from \hat{v}_t defined in AMSGrad. The paper states "...will accumulate this small error into each step" ..."will lead this model to find a suspicious local optimal".  This claim makes little sense. Even if AMSGrad uses a different v from the original Adam, it is not necessarily bad. In addition, why is this related to "suspicious local optimal" (I suppose this paper intends to say "spurious local optima")? I do not have any intuition why this is related to spurious local minima. 



2) Too many typos and grammar errors. 

Just give one example. In the first paragraph of Sec. 3.3, there are at least 15 typos, and a few sentences are hard to read: 
   "They derive it mainly from an unrealistic objective function" --what does "objective function" mean? 
   "Does it really solve the problem or dose this design violate the intuition of Adam-Type algorithm"?  --what is the "intuition" to be violated? I roughly get the point of this sentence, but "intuition" is not a good word here.
   "To be more specifically, we present a simple one-step AMSGrad swapping at iteration t and figure out the ill-condition problem". --What is "ill-condition problem"? It is not mentioned in this paragraph, and I don't know where it comes from.
   In the whole paper, there are too many problematic sentences to enumerate (a very rough estimate: at least 30?)  It makes the paper very difficult to read.

  

   
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary:

This work proposed a framework to analyze both Adam-type algorithms and SGD-type algorithms. The authors considered both of them as specialized cases of mirror descent algorithms and provided a new algorithm AdamAL. The authors showed experiments to backup their theoretical results. 

Pros:

The authors provided a novel framework to analyze Adam-type algorithms by using standard FTRL framework. It provides a unified viewpoint to consider a broad class of algorithms. The authors also provided a new algorithm AdamAL to overcome some shortcomings in previous algorithms. 

Cons:

- The novelty of this paper is limited. The authors provided a framework to analyze Adam-type algorithms. However, it seems that contribution is more conceptual rather than practical. I suggest the authors add more examples or theorems to show the superiority of their mirror descent framework. 
- Some of the explanations in this paper may be wrong. For instance, around equation 17, the authors suggested that if the swap happens, then v_{t+1} = v_t. That is not correct since the swap happens for each coordinate. Meanwhile, the explanation about why AdamAL is better than AMSgrad is quite poor since the update rule of AMSgrad can also guarantee the coordinate decreasing of v_t. I suggest the authors explain more on the algorithm design. 
- The authors should complete the proof of Theorem 3.1. 
- The settings of experiments are limited. The authors should at least compare AdamAL with other baseline algorithms on some modern deep learning tasks including Imagenet.  

Minor comments:

- Below equation 8, detail -> details.
- The authors should add the definition of 1:t in subscript for g_{1:t} or \phi_{1:t}.
- Page 6, the first paragraph, logt-> \log t
- This paper lacks some references in this area. 

J. Chen and Q. Gu. Closing the generalization gap of adaptive gradient methods in training
deep neural networks. arXiv preprint arXiv:1806.06763, 2018.
Ward, R., Wu, X. and Bottou, L. (2018). Adagrad stepsizes: Sharp convergence over nonconvex
landscapes, from any initialization. arXiv preprint arXiv:1806.01811 .
Li, X. and Orabona, F. (2018). On the convergence of stochastic gradient descent with adaptive
stepsizes. arXiv preprint arXiv:1805.08114 .
review_title: Official Blind Review #3

Meta-review: In this paper, the authors draw upon online convex optimization in order to derive a different interpretation of Adam-Type algorithms, allowing them to identify the functionality of each part of Adam. Based on these  observations, the authors derive a new Adam-Type algorithm,  AdamAL and test it in 2 computer vision datasets using 3 CNN architectures. The main concern shared by all reviewers is the lack of novelty but also rigor both on the experimental and theoretical justification provided by the authors. After having read carefully the reviews and main points of the paper, I will side with the reviewers, thus not recommending acceptance of this paper. 
Decision: Reject