SlowGuess/NeurIPS_2023_OCR huggingface dataset:
with  $\beta \geq 2$ . This update reduces to (23) when  $e_k = 0, s_k = \sqrt{s} = \beta / \sqrt{L}, t_k = k\sqrt{s}$ . We will refer to (31) as the Noisy NAG (NNAG) algorithm. NNAG is interesting due to its capability of dealing with perturbed gradients. This is the case in practical methods e.g. SGD [Bottou, 2010], SAG [Schmidt et al., 2017], SAGA [Defazio et al., 2014], SVRG [Johnson and Zhang, 2013], and etc. The following convergence result holds for NNAG, and its proof is in Appendix A.6.
Theorem 5.1. Suppose  $f \in \mathcal{F}_L$  and consider the NNAG method detailed in (31) with the following parameter choices:
$$
\beta \geq 2, s _ {k} = \frac {c}{k ^ {\alpha}}, a n d t _ {k} = \sum_ {i = 1} ^ {k} s _ {i} \quad f o r s o m e \quad c \leq \frac {1}{\sqrt {L}} a n d \frac {3}{4} \leq \alpha <   1. \tag {32}
$$
We define the critical iteration  $k_{0}$  as the smallest positive integer that satisfies
$$
k _ {0} \geq \left(\frac {\beta}{\frac {1}{c \sqrt {L}} + \frac {c \sqrt {L}}{8} \left(\sum_ {i = 1} ^ {k _ {0}} \frac {1}{i ^ {\alpha}}\right) ^ {2}}\right) ^ {1 / \alpha}. \tag {33}
$$
Then, the following bounds hold for all  $k \geq k_0$ :
$$
\mathbb {E} [ f (x _ {k}) ] - f (x ^ {*}) \leq \frac {\mathbb {E} [ \varepsilon (k _ {0}) ] + \frac {\sigma^ {2} c ^ {4}}{(1 - \alpha) ^ {2}} \left[ k _ {0} ^ {3 - 4 \alpha} - k ^ {3 - 4 \alpha} \right] + \frac {\sigma^ {2} c ^ {3} \beta}{2 \sqrt {L} (1 - \alpha) (3 \alpha - 2)} \left[ k _ {0} ^ {2 - 3 \alpha} - k ^ {2 - 3 \alpha} \right] + \frac {\beta^ {2} c ^ {2} \sigma^ {2}}{2 L (2 \alpha - 1)} \left[ k _ {0} ^ {1 - 2 \alpha} - k ^ {1 - 2 \alpha} \right]}{\frac {c ^ {2}}{4 (1 - \alpha) ^ {2}} ((k ^ {1 - \alpha} - 1) ^ {2}) + \frac {c \beta}{2 \sqrt {L} (1 - \alpha)} \left(k ^ {(1 - \alpha)} - 1\right)}
$$
if  $\alpha > 3/4$ , and
$$
\mathbb {E} \left[ f \left(x _ {k}\right) \right] - f \left(x ^ {*}\right) \leq \frac {\mathbb {E} \left[ \varepsilon \left(k _ {0}\right) \right] + 2 \sigma^ {2} c ^ {4} \left[ \log \left(\frac {k}{k _ {0}}\right) \right] + \frac {8 \sigma^ {2} c ^ {3} \beta}{\sqrt {L}} \left[ k _ {0} ^ {- 1 / 4} - k ^ {- 1 / 4} \right] + \frac {\beta^ {2} c ^ {2} \sigma^ {2}}{L} \left[ k _ {0} ^ {- 1 / 2} - k ^ {- 1 / 2} \right]}{4 c ^ {2} \left((k ^ {1 / 4} - 1) ^ {2}\right) + \frac {2 c \beta}{\sqrt {L}} \left(k ^ {1 / 4} - 1\right)} \tag {34}
$$
$$
i f \alpha = 3 / 4 w i t h \varepsilon (k) = \left(\frac {t _ {k} ^ {2}}{4} + \frac {t _ {k} \beta}{2 \sqrt {L}}\right) \left(f \left(x _ {k}\right) - f \left(x ^ {*}\right)\right) + \frac {1}{2} \| v _ {k} - x ^ {*} \| ^ {2}.
$$
Next, we show that slight modifications to the NNAG method gives rise to another stochastic method with a similar convergence rate as the NNAG algorithm, but more transparent proof (see Appendix A.7). This proof results in a convergence rate for  $\mathbb{E}\left[\min_{0\leq i\leq k - 1}\| \nabla f(x_i)\| ^2\right]$  with a rate of  $O(\log (k) / k^{(3 / 4)})$ . It remains a future work to show similar result for the NNAG update.
Theorem 5.2. Suppose  $f \in \mathcal{F}_L$  and consider the following modification of the NNAG method
$$
\left\{ \begin{array}{l} x _ {k + 1} = x _ {k} + \frac {2 s _ {k}}{t _ {k}} \left(v _ {k} - x _ {k + 1}\right) - \frac {s _ {k}}{\sqrt {L}} \left(\nabla f \left(x _ {k}\right) + e _ {k}\right), \\ v _ {k + 1} = v _ {k} - \frac {1}{2} \left(\left(t _ {k}\right) s _ {k}\right) \left(\nabla f \left(x _ {k + 1}\right) + e _ {k + 1}\right) - s _ {k} ^ {2} \left(\nabla f \left(x _ {k + 1}\right) + e _ {k + 1}\right). \end{array} \right. \tag {35}
$$
with the same parameter choices as in (32). Then, the following convergence bounds hold:
$$
\mathbb {E} \left[ f \left(x _ {k}\right) \right] - f \left(x ^ {*}\right) \leq \left\{ \begin{array}{l l} \frac {\mathbb {E} \left[ \varepsilon (0) \right] + \frac {c ^ {4} \sigma^ {2}}{8} [ 1 6 (1 + \log (k)) + 3 2 + 6 ]}{2 c ^ {2} \left[ 2 \left(k ^ {\frac {1}{4}} - 1\right) ^ {2} + k ^ {- \frac {3}{4}} \left(k ^ {\frac {1}{4}} - 1\right) \right]} & \alpha = \frac {3}{4} \\ \frac {\mathbb {E} \left[ \varepsilon (0) \right] + \frac {c ^ {4} \sigma^ {2}}{8} \left[ \frac {(4 \alpha - 2)}{(1 - \alpha) ^ {2} (4 \alpha - 3)} + \frac {4 (4 \alpha - 1)}{(1 - \alpha) (4 \alpha - 2)} + \frac {4 (4 \alpha)}{(4 \alpha - 1)} \right]}{\frac {c ^ {2}}{2 (1 - \alpha)} \left[ \frac {(k ^ {1 - \alpha} - 1) ^ {2}}{2 (1 - \alpha)} + k ^ {- \alpha} (k ^ {(1 - \alpha)} - 1) \right]} & 1 > \alpha > \frac {3}{4} \end{array} , \right. \tag {36}
$$
with  $\mathbb{E}[\varepsilon(0)] = \frac{1}{2}\|v_0 - x^*\|^2$ . In addition, for  $\alpha = 3/4$  we have
$$
\mathbb {E} \left[ \min  _ {0 \leq i \leq k - 1} \| \nabla f (x _ {i}) \| ^ {2} \right] \leq \frac {2 \sqrt {L} \mathbb {E} [ \varepsilon (0) ] + \left(2 c ^ {4} \sigma^ {2} \sqrt {L}\right) \left(2 \log (k) + 6 + \frac {3}{4}\right)}{1 6 c ^ {3} \left(\frac {k ^ {3 / 4} - 1}{3} + k ^ {1 / 4} - \frac {3}{2} + k ^ {1 / 2}\right)}. \tag {37}
$$
Remark 5.2.1. The algorithm in (35) reduces to (23) when  $e_k = 0$ ,  $s_k = \sqrt{s} = 1 / \sqrt{L}$ ,  $t_k = k\sqrt{s}$ . Remark 5.2.2 (Connection to NAG). When there is no noise ( $\sigma = 0$ ) we can also allow parameter  $\alpha$  to be zero. This is because we do not need to alleviate the effect of the noise with a decreasing step-size. Therefore, we recover the convergence rate of  $O(1 / k^2)$  for the NAG method when  $c = 1 / \sqrt{L}$ . Remark 5.2.3 (Comparison with [Laborde and Oberman, 2020]). Laborde et al., proposed a stochastic method with noisy gradients. Their method uses another presentation of the NAG algorithm (the presentation from EE discretization). Our rate (36) has the same order of convergence as [Laborde and Oberman, 2020]. However, their analysis did not achieve the bound (37) (see [Laborde and Oberman, 2020] Appendix C.4).
Remark 5.2.4 (Comparison between Theorems 5.1 and 5.2). The rate in (34) is asymptotically similar to (36). However, the transient behavior of (34) is faster than both (36) and the rate in [Laborde and Oberman, 2020] when  $L$  is large (see Figure 1 top row). This is due to the tuning parameter  $\beta$  which is usually set to  $L$  or higher. This scenario (Large  $L$ ) often happens in practice, e.g. in training a two-layer Convolutional Neural Network (CNN) [Shi et al., 2022].
Remark 5.2.5 (Limitations and future directions). One limitation of our theoretical analysis is that our convergence result for NNAG holds only for a large enough number of iterations  $k \geq k_0$ . However, in our numerical experiments we observed that the same bounds hold also for  $k \leq k_0$ , so we believe our result can be improved. Additionally, our proposed forces are currently defined only in Euclidean space, and we see potential for extending the framework to non-Euclidean spaces.
In this section, we present our empirical results, divided into three parts: Theoretical upper bounds, binary classification with logistic regression, and classification with neural networks.
Upper Bounds. First, we compare the bounds in (34) and (36) with the Proposition 4.5 in [Laborde and Oberman, 2020]. The results are depicted in Figure 1. In practical scenarios where  $L$  is large [Shi et al., 2022] the bound (34) is lower than the other two for large enough iterations. This observation has motivated us to analyze the behavior of NNAG in practical scenarios such as binary classification and CNN training tasks.
Binary Classification. For this task we considered  $d = 1000$  randomly generated samples of dimension  $n = 10$  and labels. The problem can be written as
$$
\min  _ {x \in \mathbb {R} ^ {n}} \quad \frac {1}{d} \sum_ {i = 1} ^ {d} \log \left(1 + e ^ {- y _ {i} \langle x _ {i}, x \rangle}\right), \tag {38}
$$
where  $y_{i}$  and  $x_{i}$  denote the  $i$ th label and sample. For comparison, we considered the perturbed gradient descent method with Gaussian noise and decreasing step-size  $s_{k} = 1 / (\sqrt{L} k^{2 / 3})$  (Proposition 3.4 in [Laborde and Oberman, 2020]) together with accelerated noisy gradient descent (Per-FE-C) in [Laborde and Oberman, 2020]. For NNAG we used  $s_{k} = 1 / (\sqrt{L} k^{3 / 4})$  and  $\beta = L / 10$ . All the perturbation was done using i.i.d. Gaussian noise with unit variance, and we conducted 100 Monte-Carlo runs. The results are presented in Figure 2 right panel. As shown, NNAG outperforms all the other methods in this case.
In a related experiment, we combined NNAG with SGD and SVRG. For the SGD-mixing, we replaced the noisy gradients with SGD-like gradients, while for the SVRG-mixing we evaluated all the gradients at the beginning of each epoch (essentially like taking a snapshot in SVRG) and set  $t_k = 0$ . The step-sizes for SVRG (in accordance with the original implementation [Johnson and Zhang, 2013]), SGD, NNAG+SVRG, and NNAG+SGD were set as  $1/(10L), 1/L, c = 1/\sqrt{L}, \beta = L/10$ ,
and  $c = 1 / \sqrt{L}, \beta = L$ , respectively. We conducted 100 Monte-Carlo runs, and the results are displayed in Figure 2 left panel. Notably, when mixed with either SGD or SVRG, NNAG outperforms the original methods. This highlights NNAG's flexibility in terms of gradient noise and demonstrates its potential to accelerate various methods.
Classification on CIFAR10. Finally, we tackled the non-convex optimization problem of training a CNN with CIFAR10 dataset [Krizhevsky et al., 2009] using the SGD, SVRG, NNAG, and NNAG+SVRG methods. The network consisted of two convolutional layers each followed by max pooling and 3 fully connected linear layers each followed by ReLU activation function. The step-sizes for SGD and SVRG were set as 0.01, and for the NNAG and NNAG+SVRG algorithms we had  $c = 0.05$ ,  $\beta = 150^2$  and  $c = 0.001$ ,  $\beta = 100^2 / 10$ , respectively. The division by 10 is due to step-size division by 10 in the SVRG method. The results for 20 Monte-Carlo simulations are depicted in Figure 3. Notably, SVRG+NNAG outperforms the other methods in terms of minimizing training error. Additionally, NNAG exhibits slightly better validation accuracy, hinting at its convergence toward a different local solution.
It is worth noting that faster convergence rates of NNAG+SVRG do not pose problems of overfitting. If necessary, one can terminate the algorithm earlier to achieve optimal performance. In Figure 3, for instance, NNAG+SVRG reaches its peak "validation accuracy" after approximately 20 epochs, with a validation accuracy of roughly 0.6 and a training error of around 0.66. After this point, a slow overfitting phase begins. Similarly, for SGD and SVRG, their peak "validation accuracy" is achieved after about 50 epochs, with a validation accuracy of approximately 0.6 and a training error of about 0.66, followed by a slow overfitting phase. Finally, NNAG achieves comparable results after approximately 100 epochs.
Polyak's Heavy-Ball (HB) method was one of the first momentum-based methods which could accelerate relative to the gradient descent method [Polyak, 1963]. However, this was not the case for every smooth function [Lessard et al., 2016]. Nesterov modified the HB and introduced the NAG method. This method achieved global convergence with a rate of  $\mathcal{O}(1 / k^2)$  for smooth convex functions [Nesterov, 1983]. Nesterov used estimate sequence technique to show the convergence of the NAG method. This technique does not provide immediate insights toward the success of the NAG algorithm in acceleration. Thus, many have tried to understand the essence of acceleration.
On a similar line of work, Wibisono et al. [2016] introduced a variational perspective on accelerated methods, leading to a general (non-Euclidean) ODE containing the ODE found by Su et al. [2016] as a special case. Their approach is based on the choice of a Lagrangian and its corresponding parameters. Since the choice of Lagrangian is not unique, Wilson et al. [2021] provided a variational perspective on different accelerated first-order methods using another Lagrangian. Fazlyab et al. [2017] developed a family of accelerated dual algorithms for constrained convex minimization through a similar variational approach. In a more recent development, Zhang et al. [2021] showed that the second-variation also plays an important role in optimality of the ODE found by Su et al. [2016]. Specifically, they showed that if the time duration is long enough, then the mentioned ODE for the NAG algorithm is the saddle point to the problem of minimizing the action functional.
The dynamical system perspective on NAG was studied in [Muehlebach and Jordan, 2019]. They showed that the NAG algorithm is recovered from the SIE discretization of an ODE. The mentioned ODE was not the result of a vanishing step-size argument. They found that a curvature-dependent damping term accounts for the acceleration phenomenon. Interestingly, [Chen et al., 2022] also used similar ODE without the SIE discretization. They showed that implicit-velocity is the reason of the acceleration. In a recent analysis, [Muehlebach and Jordan, 2023] explores the connections between non-smooth dynamical systems and first-order methods for constrained optimization.
In this work, we tackled the problem of unconstrained smooth convex minimization in Euclidean space. Through a variational analysis of HR-ODEs, we achieved improved convergence rates for NAG in terms of gradient norm. In addition, we showed that NAG can be viewed as an approximation of the rate-matching technique when applied on a specific ODE. Our analysis was then extended to stochastic scenarios. In particular, we proposed a method with both constant and varying step-sizes which performed comparable and sometimes better than state of the art methods.
This work entails multiple future directions. Nesterov's oracle complexity lower bound on gradient norm minimization is  $\mathcal{O}(k^{-4})$  [Nesterov, 2003]. It remains an open question to see if the NAG method can achieve this rate of convergence for gradient norm minimization. In this work, we noticed that the HR-ODEs follow the same external force structure. In the smooth-strongly convex case, Triple Momentum (TM) method is the fastest known globally convergent method [Van Scoy et al., 2018]. However, the HR-ODE associated with the TM method is not shown to achieve the similar convergence rate as the TM method [Sun et al., 2020]. One could use the external force structure proposed here to find a better convergence rate for the HR-ODE associated with the TM algorithm. In addition, our analysis was confined to the Euclidean space. We believe it is possible to explore non-Euclidean forces using a Bregman Lagrangian as in [Wibisono et al., 2016]. Finally, we blended our noisy stochastic scheme with other known stochastic methods (e.g. SGD and SVRG). This technique improved the performance of those methods. As a future work, one can apply the same technique to other practical methods like ADAM, RMSprop, etc, and study the behavior of the final algorithm.
Alp Yurtsever and Hoomaan Maskan received support from the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. Konstantinos C. Zygalakis acknowledges support from a Leverhulme Research Fellowship (RF/2020-310), and the EPSRC grant EP/V006177/1. We gratefully acknowledge the support of NVIDIA Corporation with the donation of 2 Quadro RTX 6000 GPUs used for this research.
Y. Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2003.
B. Polyak. Gradient methods for the minimisation of functionals. Ussr Computational Mathematics and Mathematical Physics, 3:864-878, 1963.
J. M. Sanz Serna and K. C. Zygalakis. The connections between lyapunov functions for some optimization algorithms and differential equations. SIAM Journal on Numerical Analysis, 59(3): 1542-1565, 2021.
M. Schmidt, N. Le Roux, and F. Bach. Minimizing finite sums with the stochastic average gradient. Mathematical Programming, 162:83-112, 2017.
B. Shi, S. S. Du, W. J. Su, and M. I. Jordan. Acceleration via symplectic discretization of high-resolution differential equations. arXiv preprint arXiv:1902.03694, 2019.
B. Shi, S. S. Du, M. I. Jordan, and W. J. Su. Understanding the acceleration phenomenon via high-resolution differential equations. ArXiv, abs/1810.08907, 2021.
Z. Shi, Y. Wang, H. Zhang, J. Z. Kolter, and C.-J. Hsieh. Efficiently computing local lipschitz constants of neural networks via bound propagation. Advances in Neural Information Processing Systems, 35:2350-2364, 2022.
J. W. Siegel. Accelerated first-order methods: Differential equations and lyapunov functions. arXiv preprint arXiv:1903.05671, 2019.
W. Su, S. Boyd, and E. J. Candès. A differential equation for modeling nesterov's accelerated gradient method: Theory and insights. Journal of Machine Learning Research, 17(153):1-43, 2016. URL http://jmlr.org/papers/v17/15-084.html.
B. Sun, J. George, and S. S. Kia. High-resolution modeling of the fastest first-order optimization method for strongly convex functions. In 59th IEEE Conference on Decision and Control, CDC 2020, Jeju Island, South Korea, December 14-18, 2020, pages 4237-4242. IEEE, 2020. doi: 10.1109/CDC42340.2020.9304444. URL https://doi.org/10.1109/CDC42340.2020.9304444.
B. Van Scoy, R. A. Freeman, and K. M. Lynch. The fastest known globally convergent first-order method for minimizing strongly convex functions. IEEE Control Systems Letters, 2(1):49-54, 2018. doi: 10.1109/LCSYS.2017.2722406.
A. Wibisono, A. C. Wilson, and M. I. Jordan. A variational perspective on accelerated methods in optimization. Proceedings of the National Academy of Sciences, 113 (47):E7351-E7358, 2016. ISSN 0027-8424. doi: 10.1073/pnas.1614734113. URL https://www.pnas.org/content/113/47/E7351.
A. C. Wilson, B. Recht, and M. I. Jordan. A lyapunov analysis of accelerated methods in optimization. J. Mach. Learn. Res., 22:113-1, 2021.
P. Zhang, A. Orvieto, and H. Daneshmand. Rethinking the variational interpretation of accelerated optimization methods. Advances in Neural Information Processing Systems, 34:14396-14406, 2021.
Z. A. Zhu and L. Orecchia. Linear coupling: An ultimate unification of gradient and mirror descent. In Information Technology Convergence and Services, 2014.