Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors found that using a variant of DEMA to replace the EMA used in past optimizers and using a dynamic lookahead optimizer can achieve a better result. Motivated by that, the author proposed a novel optimizer framework Admeta, and implement it on SGD and Radam. Empirical results demonstrate the superior performance of Admeta compared to other optimizers.
strength_and_weaknesses: The strengths of the paper include an interesting framework to analyze the optimizer with modifications to each part, and a nice visalization figure show the comparison between DEMA and EMA. In my views, the limitations of the paper are that some results are not so good, especially in some relatively large models or datasets, such as MNLI with large BERT model. (But it should not be a big problem for the optimizer.) 
My detailed comments are listed as below.

1) Theorems 1-4 provide convergence proof on AdmetaR and AdmetaS, the two implementation versions of Admeta based on RAdam and SGD respectively. Before the main proof in the appendix, the authors make many assumptions and omit many things and argue that this argument can be attended to these cases. For example, the authors omit the forward-looking part in the convergence proof.  Are these assumptions and neglections reasonable? More discussion should be put in this part.

2) As in some related works, the author should do some convergence rate analysis of AdmetaR in the convex and nonconvex settings to demonstrate its high convergence speed?

3) The authors should introduce the hyperparameters tuning with more details since it is very important for the application of this optimizer in other tasks.

4) In the ablation study part, I notice that AdmetaS – DEMA achieves worse result than SGDM, which may indicate that DEMA is not always a good method compared to EMA unless combined with a dynamic method. What may be the reason for this? The author needs to explain more.

5) Separating optimizers into forward-looking and backward-looking parts is not a novel view. In fact, it is already used in Ranger ([1]). What is the difference of Admeta compared to Ranger optimizer. 


6) I notice that authors use SGD with Nesterov momentum in the experiment. It would be more interesting to see Nadam, Adam with Nesterov momentum in the comparisons. How about incorporating Nesterov’s approach in DEMA?

[1] Less Wright. Ranger - a synergistic optimizer. https://github.com/lessw2020/ Ranger-Deep-Learning-Optimizer, 2019.
clarity,_quality,_novelty_and_reproducibility: The paper is clear for readers to follow with a nice quality and interesting novelty. The reproducibility is good.
summary_of_the_review: This work has introduced a new optimizer with modifications on two parts. Despite some deficiencies in theory, the paper is well written.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors give a clear introduction to current optimizers in deep neural networks and propose a new framework for improvement, which divides the optimizer into two parts, forward-looking and backward-looking. 

To replace the traditional EMA method used in backward-looking part, a variant of double exponential moving average (DEMA) is used. The authors demonstrate its advantage over EMA by analyzing the structure of the formula and attaching a comparison figure, which visually presents the convergence trend in the appendix.

And to further improve the Lookahead optimizer, an asymptotic weight is used, which achieves faster training than constant lookahead around the early stage and maintains the advantage of better convergence around the final stage. The authors also provide two alternatives to compute the asymptotic weight.

strength_and_weaknesses: Strengths:

The ideas are reasonable, and this work gives a novel view in the optimizer field.

The authors conduct several experiments in Image Classification, Natural Language Understanding and Audio Classification, and experiments show the effectiveness of the proposed method. For all but a few tasks, this method shows advantages over other recent optimizers.

This paper follows a good logic and is easy to understand.

Concerns:

On the other hand, I also have some concerns and questions toward this paper:

 (1) The theoretical explanation for the advantage of DEMA is insufficient. For example, the authors mention that the past gradients in EMA follow a fixed proportionality which is incompatible with actual optimization. Why?

 (2) In Image Classification, why the improvement for ResNet-110 is larger than PyramidNet?

 (3) The authors introduce a bias term in the DEMA method, but omit it in the rest of the paper and point out that this does not make difference. Why?

(4) The choice of $\mu$ and $\kappa$ seems strange. What is the reason for constructing such a function? 

(5) How do you choose the hyper-parameter of the Admeta optimizer? It seems that the hyper-parameters chosen for each task are different.  

 (6) There are some deficiencies in the theoretical analysis, although I understand the space is limited. If more theoretical analysis is added, it will be more convincing and interesting. 

 (7) Other works on optimizers like [1] tend to give an interval on experiment results in the appendix. I suggest that the results need to be tested for statistical significance.

(8) Although training on large datasets is not necessary for the optimizer, if Admeta can be used in the large-scale dataset ImageNet or BERT pre training stage, the advantages of the proposed optimizer will be better demonstrated.

(9) Though the related work is organized well and summarized into SGD Family, Adam Family, Stochastic Second Order Family and Other Optimizers, the authors should clearly explain the relation between the proposed method and related works.

[1] Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization 

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well organized and clear.

Quality: See contents above.

Novelty: This article is innovative as a conference paper.

Reproducibility: The paper has released the code for reproducing.

summary_of_the_review: The paper proposes a novel framework for optimizer. It seems that this work is quite original and the experimental results are relatively good. The main weaknesses are that some points are not made clear and extended experiments are needed to demonstrate the effectiveness of the proposed method. In general, the paper is well-written and comprehensible, and the idea is impressive.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a framework for stochastic optimizers named Admeta. Based on SGD and R-Adam, the authors provide two implementations, AdmetaS and AdmetaR. 

Specifically, the author contributes a new gradient decent algorithm framework in the style of SGD and Adam optimizer. First, the author identifies the problem of the commonly used EMA method: the increasing lag time and overshoot problem. Then the author identifies the problem of the Lookahead optimizer: a slow stepsize around the early stage of training. The paper is well written.

strength_and_weaknesses: Pros:
- The authors identify the problem of the traditional used EMA and Lookahead method and give improvements to modify them.
- The introduction of the method is straightforward and easy to follow and the motivation is very interesting which comes from stock markets.
- It is simple but effective. I am positively surprised by the results achieved in some tasks brought by the modifications. 

Concerns:
- I notice that in the CIFAR-100 task trained on PyramidNet, the results of R-Adam, Ranger, and Adabelief are even worse than AdamW. It is very interesting. What hyper-parameters do you use? More discussion of this phenomenon is welcomed.
- Can you explain why SGD-family achieves awful results in Finetune task? Additional experiments are needed to explore this point. How will AdmetaS perform in Finetune tasks? Will all SGD family optimizers behave like this? What if the model chooses the Deep RNN/CNN architecture instead of the Deep Transformer architecture? 
- The authors simply changed EMA to a DEMA variant and turn lookahead into a dynamic one. I think innovation is a bit incremental.
- There are some problems in the organization of the paper. Many important parts have been placed in the appendix, which increases the difficulty for readers.

clarity,_quality,_novelty_and_reproducibility: Clarity:
- The introduction is straightforward and the method is stated clearly.

Quality:
-This work is OK.

Novelty:
- The innovation is a bit incremental.

Reproducibility:
-Checked
summary_of_the_review: I believe that the paper has a lot of potentials but requires some additional discussion. My main concern lies in the results and innovation of the proposed optimizer. In general, this paper is worth publishing if concerns can be well addressed in the rebuttal stage.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This article shows that exponential moving averages have several weaknesses, namely momentum and Lookahead optimizers. The authors then propose modifications corresponding to these weaknesses and propose a framework, Admeta, to analyze optimizers. Theoretical results are provided to show the convergence and the reasons for such modifications. Empirical results on various tasks are provided to demonstrate that the proposed method outperforms the baseline optimizer and the currently proposed optimizer.

In the implementation, the authors propose new variants of Adam and SGD using the DEMA method and a dynamic look-ahead strategy to avoid the potential negative effects of traditionally used EMA and look-ahead optimizers. The paper also provides evidence that both implementations converge in the convex and non-convex cases.
strength_and_weaknesses: Strength:
1. The paper provides a point of view that is often overlooked by researchers.
2. This paper provides experimental results on various tasks and datasets to demonstrate the advantages of the proposed method.
3. The paper is easy to read.

Weaknesses:

1. The proof in the paper, although valid, is not sufficient to demonstrate the advantages of the proposed method, as it only guarantees convergence, which is the most basic requirement of an optimizer in deep learning.

2. The paper can be improved by adding more theoretical explanations. What's more, the paper could add more and larger datasets.

3. The improvements of the proposed method on some tasks are small in the experiments, so error bars are needed to verify that these improvements are not due to randomness. However, this paper does not provide error bars for the empirical results.

4. AdaBelief's report in CIFAR seems bad. However, in the original paper by Adabelief ([1]), it achieves state-of-the-art results, albeit with a different backbone model. I don't know if there is a problem with the parameter settings of AdaBelief.

5. SGDM in PyramidNet tried a learning rate of 0.5. Why not try it in ResNet110?
clarity,_quality,_novelty_and_reproducibility: The article is clearly written and presents an enlightening method. I think the attached code can be reproduced.
summary_of_the_review: The paper is well written and easy to read. To clearly explain these methods, the authors combine text, formulas, and figures. For this, I thank the author for his hard work. I also believe that the Admeta framework provides an impressive perspective on the future work of the optimizer. Experiments demonstrate its good results in many tasks. All in all, it's an interesting piece of work that I'm sure would be of interest to researchers in related fields.

[1] AdaBelief optimizer: adjust step size based on observed gradients
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents an variant of the Adam algorithm, replacing the exponential moving average of the gradients with a double exponential moving average and using elements from the  Lookahead optimizer. Convergence results as well as an empirical validation are presented.

The reviewers found this paper mildly interesting and promising. However, they also pointed out some shortcomings. In particular, as in many similar papers, the theoretical analysis is so bad to be almost a negative point of the paper. For example, 1) the convergence results do not show any advantage over SGD; 2) the theory does not offer any insights in the working of the algorithm; 3) the iterates are assumed to be bounded instead of proving it; 4) the analysis treats all the modifications of the algorithms as "disturbances" to SGD. Moreover, the proofs techniques are very standard. While some reviewers always tend to ask for theory, such low quality results are not really useful in any sense.
This means that the empirical comparison is the only way we have to validate the claim of the gains over the baseline algorithms.

The empirical results show some small gain, but it is very difficult to understand how significant they are. In particular, it seems extremely difficult to distinguish these gains from all the ones in all the other similar papers (accepted and rejected) from the past few years that did not survive to the test of time. Also, there are a number of hyperparameters that are not tuned for each algorithm in the experiments that might completely change the results, for example, learning rate schedule, weight decay, momentum terms, etc.

Overall, given the tepid reception from the reviewers (even after the changes done to the paper by the authors), the very weak theory, and the unclear significance of the results, I believe this paper is not ready to be published at ICLR in its current form.
justification_for_why_not_higher_score: None of the reviewers replied when asked *why* to accept this paper, all of them gave a borderline score.
Decision: Reject