Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper, the author propose a decaying momentum rule to improve algorithm. Furthermore, he apply this rule in momentum SGD and Adam, then use experiment to prove the algorithm.

In the experiment, it training on many different dataset and compare with many baseline. The algorithm with decaying momentum rule get much better result than all other algorithm. Furthermore, the paper is well written and east to follow.

However, I still have a question about this paper. 

Even the algorithm get a good performance, the intuition of the algorithm is not clear. In fact, it is reasonable to decay the total contribution of a gradient to all future updates; it is still unclear why you choose such $\beta$. As you say, the contribution of previous item is $ \sum \beta^i=\frac \beta {(1-\beta)} $, however, this is only correct why $\beta$ is constant, and when the $\beta_t$ is change like your definition, what will the contribution become? Furthermore, even the contribution of previous item is $ \frac \beta {(1-\beta)} $, why you want the contribution is equal to $ (1-\frac t T)\beta_{init}/(1-\beta_{init})$, it may need more comment about why you choose the value.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposed a new decaying momentum rule to further improve neural network training. The idea is motivated by decaying the total contribution of a gradient to all future updates. The authors also extend this idea on to Adam and show that it improved upon the vanilla Adam.

- The intuition of using momentum decaying scheme is not quite clear to me. Why having a linear momentum decay schedule is better? This question is not answered in the paper. I hope the authors could provide more illustrative explanations regarding this.

- This paper focused essentially on empirical evaluations. I do appreciate that the authors conduction various experiments using different dataset and architecture but I do not understand why the authors want to separate the experiments into parts: adaptive methods and adaptive momentum methods. Not to mention the confusing names, it makes no sense to say the same words again for different optimizers. I would suggest the authors to combine the results together for better comparison.

- CM formulation mentioned in the paper is actually different from the commonly refer SGD with momentum implementation, which is the optimizer that is widely used in the community. Therefore, it is important to at least add SGD with momentum as one baseline in the experiments. 

- According to https://github.com/kuangliu/pytorch-cifar, Resnet18 on CIFAR10 can achieve at least 93% accuracy using simple SGD with momentum. It seems that the current reported results are not fully optimized. I would suggest the authors to check the parameter settings and make sure all the hyperparameters for baseline methods are fully tuned.

- Aside from SGD with momentum, the experiments part also lacks several important baselines. The author should also consider comparing the AdamW, Padam mentioned in the paper.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The authors propose Demon schedule that automatically decays momentum. The paper is very well written, and the experiments are performed on an, extensive, compared to typical papers in the domain, suite of tasks. Unfortunately, while the paper is technically well executed, I have fundamental issues with novelty. Based on the current state-of-the-art understanding of optimization in deep learning, it is quite expected that decaying momentum has an analogous effect to decaying learning rate or increasing batch size (see [1,2]). Given this "Similarly, applying DEMON to momentum SGD rivals momentum SGD with learning rate decay" is not surprising, and "and in many cases leads to improved performance" warrarnts a bit of scepticism.

More precisely, a very similar analysis as in "Motivation and interpretation." can be already found in [1]. Similarly, [2] already suggests decreasing momentum. Based on this, experiments in Table. 2 (and the other similar Tables) should compare to decaying learning rate or batch size in Adam and other adaptive methods using an analogous schedule. It might be surprisng, but adaptive methods do benefit from learning rate schedules. Analogously, could you please compare performance of Demon to the recommendation in [2] to decay momentum together with increasing learning rate initially? 

Unless a more detailed experiments demonstrate that tuning momentum brings additional benefit on top of its effect on the effective learning rate, there isn't unfortunately in my onion enough practical value in the proposed method.

References:

[1] Smith et al, Don't Decay the Learning Rate, Increase the Batch Size, https://arxiv.org/abs/1711.00489
[2] Fast.ai documentation on one cycle method, https://docs.fast.ai/callbacks.one_cycle.html
review_title: Official Blind Review #2

Meta-review: This paper proposes a new decaying momentum rule to improve existing optimization algorithms for training deep neural networks, including momentum SGD and Adam. The main objections from the reviewers include: (1) its novelty is limited compared with prior work; (2) the experimental comparison needs to be improved (e.g., the baselines might not be carefully tuned, and learning rate decay is not applied, while it usually boosts the performance of all the algorithms a lot). After reviewer discussion, I agree with the reviewers’ evaluation and recommend reject.
Decision: Reject