A New Variant of Stochastic Heavy ball Optimization Method for Deep Learning

Abstract: Stochastic momentum optimization methods, also known as stochastic heavy ball (SHB) methods, are one of the most popular optimization methods for deep learning. These methods can help accelerate stochastic gradient descent and dampen oscillations. In this paper we provide a new variant of the stochastic heavy ball method, called stochastic Euler’s heavy ball (SEHB). The proposed SEHB method modifies the steepest descent direction to achieve acceleration, and combines Euler‘s method to adaptively adjust learning rates as well. A convergence analysis of the regret bound is discussed under the online convex optimization framework. Furthermore, we conduct experiments on various popular datasets and deep learning models. Empirical results demonstrate that our SEHB method shows comparable or even better generalization performance than state-of-the-art optimization methods such as SGD and Adam. 

Reviews and author responses:

Based on the idea of Euler’s method, this paper proposes an algorithm that adaptively adjusts the directional step sizes. The algorithm also incorporates Heavy Ball momentum with a tunable momentum parameter. A convergence analysis of the algorithm is provided in the case of a decaying learning rate. The proposed method outperforms other adaptive methods, including Adam, in the experiments shown in this paper.

Comments/Questions:

> Like the other adaptive methods, e.g., Adam, the proposed SEHB adaptively adjusts the (directional) step sizes according to the stochastic gradient history. However, a major distinction I can see is: while other algorithms “additively” accumulate the gradients, the proposed algorithm “multiplicatively” accumulates the gradients; see Eq. (4). This makes it different from all the other well-known adaptive methods, such as Adam, Adadelta, RMSprop etc. I am curious to see whether this difference brings new benefits in theory.

> In SEHB, the single hyper-parameter $\eta$ controls both the overall learning rate (as in Eq.(5) and line 6 in Algorithm 1) and the gradient accumulation level (as in Eq.(4) and line 4 in Algorithm 1). Note that all the other adaptive methods, such as Adam, Addelta, PMSprop etc, use two independent hyper-parameters (e.g., step size and beta_1 in Adam) to control the two seemingly independent aspects of the algorithms. Although it seems that this degeneration of the hyper-parameters in SEHB arises from the Euler’s method, I am curious whether this degeneration restricts the performance of SEHB in practice.

> As the denominator of Eq (4) is always greater than 1, and that hyper-parameter c is set to 1 in practice, we can see that the actual step sizes (the multiplicative factor before $\nabla f$ in Eq.(5)) is monotonically decreasing, and potentially decrease too fast resulting in a slow convergence later. Also note that, unlike Adam and Adadelta etc which decays the effects/weights of early gradients, the SEHB seems not to decay any of its gradients in the accumulation, which may result in a long term effect of a bad gradient. Both of the above effects are commonly believed to be not beneficial in practice.

> The experimental results demonstrated, especially the ones on CIFAR-10 dataset, are super impressive. SEHB clearly outperforms all the others and achieves <5.0% test error on ResNet-50. I am interested to see the performance of SEHB on architectures that achieve SOTA performance on CIFAR-10. Could SEHB achieve new SOTA performance? I also would like to see a comparison of SEHB with SGD (and SGD + Nesterov) which is believed to have better generalization performance than the adaptive methods.

Minor:

Title of Section 3.2: convergence analyze --> convergence analysis


#### Summary
This paper proposes a new SGD algorithm with heavy-ball momentum and adaptive coordinate-wise stepsizes, called SEHB. It is based on a very recent algorithm called AEGD, and when the momentum parameter gamma_t is set to 0, SEHB becomes identical to AEGD. 

I find the paper to be of quite low quality for several reasons. Firstly, the paper does not motivate the algorithm at all: it simply lists the ideas that the authors combine in this work, and the derivation of the stepsizes is done under the assumption that $\gamma_t=0$, which corresponds to the case with no momentum. Secondly, it proposes a momentum modification and tries to analyze it, but the theoretical momentum parameter has to decrease exponentially. Finally, the work claims superior performance to other adaptive and non-adaptive algorithms on nonconvex problems, but the tests are insufficient to draw this conclusion. I elaborate on these points below.

#### Motivation
I can see how the authors derive the stepsizes, and every step of the derivation is clear, but the motivation behind the steps is missing. For instance, why was the function $g()$ introduced? Why are we trying to estimate its values and based on that produce stepsizes? Similarly, why the stepsizes are not changed when the authors introduce momentum? 

#### Theory
The theory is disconnected from practice since the momentum parameter in the analysis assumed to be decreasing exponentially. The authors argue that this is similar for Adam and AMSGrad, but in a recent work [1] there is an analysis of AMSGrad with constant momentum ($\beta_1$). Therefore, I find the decreasing momentum schedule of this submission to be a significant flaw, especially since introducing momentum to AEGD is its main contribution. In addition, as far as I can see, the authors show no advantage of the proposed bound compared to that of AEGD.

The nature of Theorem 1 is also concerning: before proving anything about the iterates, the authors assume that the iterates difference are bounded, $\|\theta_n - \theta_m\|\le \mathrm{const}$ for any pair of the indices $(n, m)$. This is equivalent to assuming that the iterates themselves are bounded and the only way it can be guaranteed is by introducing a projection step in the algorithm, which the authors didn't do. Of course, this issue is common in regret analysis, but it is an issue nevertheless. Moreover, convergence of many stochastic algorithms such as SGD can be shown without assuming bounded iterates or gradients, see for example [2].

#### Experiments
Even if the theory is not good, the paper still had a chance if it showed a solid justification of improvement over state-of-the-art optimization methods. Unfortunately, I see no code attached to this submission, which makes me a bit concerned about reproducibility, especially as so many works have claimed to be better than both SGD and Adam but have not become commonly used in practice. What makes me additionally concerned about the experiments is that only one seed seems to be used per each method, which makes the results random. Even SGD may converge to different accuracy with a different random seed when everything else, including the stepsize and the architecture, is fixed. 

The next issue is that only deep networks for vision were tested, while some methods fail to outperform Adam on NLP tasks, GANs, etc. A more diverse selection of problems is required to guarantee that the proposed algorithm is practical.

Ideally, more than a single minibatch size needs to be tested, but this is a smaller issue. To improve the experiments, I would recommend the authors to consult a paper concerned with benchmarking optimization methods, for instance [3].

The first experiment is on Rosenbrock function, which is rarely used these days as a benchmark since no one uses it in practice. I suggest the authors remove this experiment from the paper and use the space for other things such as additional experiments on real-world applications, proof sketch, etc.

In the experiment details, the authors wrote numerical constants as "1e−2". I suggest writing it as $10^{-2}$ or 0.01, which is more readable.

#### Minor issues

I recommend writing in the algorithm caption that G^2 denotes the coordinate-wise square. Currently, the notation is explained in a separate section, which is convenient but might be overlooked by someone who just wants to know what algorithm you propose.

Lemmas 3 and 4 have no commentary around the equations. Please add at least some minimal explanations of the steps performed in the proofs and their motivation.

##### Typos
page 1, "SGD havs"
Equation (7), missing a period at the end.
page 4, Remark cites (Nesterov, 1983) for the convergence of SGD. This paper does not have any result about SGD and instead is his work with the accelerated gradient algorithm. Moreover, the citation for the paper of (Nesterov, 1983) has the incorrect title: it mentions $1/\sqrt{k}$ convergence instead of $1/k^2$.
page 6, "default γ often have" -> "has"
page 7, "performance on the test set have" -> "has"
page 7, "has a excellent" -> "an"
page 9, Equation (9) and other equations, $\theta_{,i}^*$ should be replaced with $\theta_i^*$
page 9, Equation (9) and other equations, when writing the $i$-th coordinate of the gradient of $f_t$ at $\theta_t$, you probably meant to write $\nabla_i f_t(\theta_t)$ or $(\nabla f_t(\theta_t))_i$ rather than $\nabla f_t(\theta_{t,i})$, which doesn't make any sense unless $f_t$ is a linear function

[1] Alacaoglu et al., "A new regret analysis for Adam-type algorithms", ICML 2020
[2] Gower et al., "SGD: General Analysis and Improved Rates", ICML 2019
[3] Sivaprasad et al., "Optimizer Benchmarking Needs to Account for Hyperparameter Tuning", ICML 2020

Summary: 

This paper proposes a new variant of Stochastic Heavy Ball method, combining Euler's method to adjust learning rates adaptively. They give a convergence analysis of the regret bound under the online convex optimization framework and conduct experiments on MNIST and CIFAR10. 

Overall, this method is simple to use and novel to me. But the motivation of this work and how it related to the previous ones like AEGD are not clearly explained. Also, I think the theoretical analysis is weak, and more empirical evidences are expected to back up their claims. 

- Though the author write the derivation of their method, I find it vague that why to introduce g($\theta$) as a ~square root of f($\theta$) and why to update the gradient of g using heavy ball method instead of gradient of f. The motivation should be more clearly illustrated. 

- In Theorem 1, the bounded domain assumption is strong and unjustified: It can not be assumed since there is no projection in the algorithm. Also, the bounded gradients assumptions is kind of strong.  

- In Theorem 1, there is no discussion about $\sum_{i=1}^d (1/\gamma_{T+1, i} + 1/\gamma_{2, i})$. What is it expected to be in general?  The authors claim that when $\sum_{i=1}^d (1/\gamma_{T+1, i} + 1/\gamma_{2, i})<< \sqrt{d}$, the regret bound is better than SGD. Yet, there is no reason nor justification on why that condition should happen. So, the theoretical result is weak even with the strong assumptions. 

- There are experiments on two data sets. The authors should also consider more tasks to justify the better generalization of their method, like task on ImageNet or NLP task. 


Meta-review: The paper presents  a new variant of the Stochastic Heavy Ball method with coordinate-wise stepsizes. They prove a regret upper bound in the online convex optimization setting and validate the algorithm on few deep learning tasks.

The reviewers found the paper severely lacking on many aspects. In particular, the formulation appears not motivated at all, the regret upper bounds relies on an unverified assumption of boundedness of the iterates, the momentum parameter must decrease exponentially over time. Note that it is known how to analyze the momentum algorithm under much more general conditions.
The empirical evaluation was also judged not sufficient, with only 2 datasets (one of them being MNIST).

Overall, the paper was judged not suited for publication at ICLR.

Decision: withdrawn-rejected-submissions