SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ahmed Khaled  
Princeton University
Konstantin Mishchenko  
Samsung AI Center
Chi Jin
Princeton University
This paper proposes a new easy-to-implement parameter-free gradient-based optimizer: DoWG (Distance over Weighted Gradients). We prove that DoWG is efficient—matching the convergence rate of optimally tuned gradient descent in convex optimization up to a logarithmic factor without tuning any parameters, and universal—automatically adapting to both smooth and nonsmooth problems. While popular algorithms following the AdaGrad framework compute a running average of the squared gradients to use for normalization, DoWG maintains a new distance-based weighted version of the running average, which is crucial to achieve the desired properties. To complement our theory, we also show empirically that DoWG trains at the edge of stability, and validate its effectiveness on practical machine learning tasks.
We study the fundamental optimization problem
$$
\min  _ {x \in \mathcal {X}} f (x), \tag {OPT}
$$
where  $f$  is a convex function, and  $\mathcal{X}$  is a convex, closed, and bounded subset of  $\mathbb{R}^d$ . We assume  $f$  has at least one minimizer  $x_* \in \mathcal{X}$ . We focus on gradient descent and its variants, as they are widely adopted and scale well when the model dimensionality  $d$  is large (Bottou et al., 2018). The optimization problem (OPT) finds many applications: in solving linear systems, logistic regression, support vector machines, and other areas of machine learning (Boyd and Vandenberghe, 2004). Equally important, methods designed for (stochastic) convex optimization also influence the intuition for and design of methods for nonconvex optimization— for example, momentum (Polyak, 1964), AdaGrad (Duchi et al., 2010), and Adam (Kingma and Ba, 2015) were all first analyzed in the convex optimization framework.
As models become larger and more complex, the cost and environmental impact of training have rapidly grown as well (Sharir et al., 2020; Patterson et al., 2021). Therefore, it is vital that we develop more efficient and effective methods of solving machine learning optimization tasks. One of the chief challenges in applying gradient-based methods is that they often require tuning one or more stepsize parameters (Goodfellow et al., 2016), and the choice of stepsize can significantly influence a method's convergence speed as well as the quality of the obtained solutions, especially in deep learning (Wilson et al., 2017).
The cost and impact of hyperparameter tuning on the optimization process have led to significant research activity in designing parameter-free and adaptive optimization methods in recent years, see e.g. (Orabona and Cutkosky, 2020; Carmon and Hinder, 2022) and the references therein.
We say an algorithm is universal if it adapts to many different problem geometries or regularity conditions on the function  $f$  (Nesterov, 2014; Levy et al., 2018; Grimmer, 2022). In this work, we focus on two regularity conditions: (a)  $f$  Lipschitz and (b)  $f$  smooth. Lipschitz functions
have a bounded rate of change, that is, there exists some  $G > 0$  such that for all  $x, y \in \mathcal{X}$  we have  $|f(x) - f(y)| \leq G \| x - y \|$ . The Lipschitz property is beneficial for the convergence of gradient-based optimization algorithms. They converge even faster on smooth functions, which have continuous derivatives; that is, there exists some  $L > 0$  such that for all  $x, y \in \mathcal{X}$  we have  $\| \nabla f(x) - \nabla f(y) \| \leq L \| x - y \|$ . Smoothness leads to faster convergence of gradient-based methods than the Lipschitz property. Universality is a highly desirable property because in practice the same optimization algorithms are often used for both smooth and nonsmooth optimization (e.g. optimizing both ReLU and smooth networks).
The main question of our work is as follows:
Can we design a universal, parameter-free gradient descent method for (OPT)?
Existing universal variants of gradient descent either rely on line search (Nesterov, 2014; Grimmer, 2022), bisection subroutines (Carmon and Hinder, 2022), or are not parameter-free (Hazan and Kakade, 2019; Levy et al., 2018; Kavis et al., 2019). Line search algorithms are theoretically strong, achieving the optimal convergence rates in both the nonsmooth and smooth settings with only an extra log factor. Through an elegant application of bisection search, Carmon and Hinder (2022) design a parameter-free method whose convergence is only double-logarithmically worse than gradient descent with known problem parameters. However, this method requires resets, i.e. restarting the optimization process many times, which can be very expensive in practice. Therefore, we seek a universal, parameter-free gradient descent method for (OPT) with no search subroutines.
Our contributions. We provide a new algorithm that meets the above requirements. Our main contribution is a new universal, parameter-free gradient descent method with no search subroutines. Building upon the recently proposed Distance-over-Gradients (DoG) algorithm (Ivgi et al., 2023), we develop a new method, DoWG (Algorithm 1), that uses a different stepsize with adaptively weighted gradients. We show that DoWG automatically matches the performance of gradient descent on (OPT) up to logarithmic factors with no stepsize tuning at all. This holds in both the nonsmooth setting (Theorem 3) and the smooth setting (Theorem 4). Finally, we show that DoWG is competitive on real machine learning tasks (see Section 4).
Algorithm	No search	Parameter-free	Universal	GD framework
Polyak stepsize (Polyak, 1987; Hazan and Kakade, 2019)	✓	X	✓	✓
Coin betting with normalization (Orabona and Pál, 2016; Orabona and Cutkosky, 2020; Orabona, 2023)	✓	✓	✓(*)	X
Nesterov line search (Nesterov, 2014)	X	✓	✓	✓
AdaGrad (Duchi et al., 2010; Levy et al., 2018; Ene et al., 2021)	✓	X	✓	✓
Adam (Kingma and Ba, 2015; Li et al., 2023)	✓	X	✓	✓
Bisection search (Carmon and Hinder, 2022)	X	✓	✓	✓
D-Adaptation (Defazio and Mishchenko, 2023)	✓	✓	X	✓
DoG (Ivgi et al., 2023)	✓	✓	✓(*)	✓
DoWG (new, this paper!)	✓	✓	✓	✓
(*) Result appeared after the initial release of this paper. Table 1: A comparison of different adaptive algorithms for solving (OPT). "Universal" means that the algorithm can match the rate of gradient descent on both smooth and nonsmooth objectives up to polylogarithmic factors. "No search" means the algorithm does not reset. "GD framework" refers to algorithms that follow the framework of Gradient Descent.
There is a lot of work on adaptive and parameter-free approaches for optimization. We summarize the main properties of the algorithms we compare against in Table 1. We enumerate some of the major approaches below:
Polyak stepsize. When  $f_{*} = f(x_{*})$  is known, the Polyak stepsize (Polyak, 1987) is a theoretically-grounded, adaptive, and universal method (Hazan and Kakade, 2019). When  $f_{*}$  is not known, Hazan and Kakade (2019) show that an adaptive re-estimation procedure can recover the optimal convergence rate up to a log factor when  $f$  is Lipschitz. Loizou et al. (2021) study the Polyak stepsize in stochastic non-convex optimization. Orvieto et al. (2022) show that a variant of the Polyak stepsize with decreasing step sizes can recover the convergence rate of gradient descent, provided the stepsize is initialized properly. Unfortunately, this initialization requirement makes the method not parameter-free.
The doubling trick. The simplest way to make an algorithm parameter-free is the doubling-trick. For example, for gradient descent for  $L$ -smooth and convex optimization, the stepsize  $\eta = \frac{1}{L}$  results in the convergence rate of
$$
f (\hat {x}) - f _ {*} = \mathcal {O} \left(\frac {D _ {0} L ^ {2}}{T}\right), \tag {1}
$$
where  $D_0 = \| x_0 - x_*\|$ . We may therefore start with a small estimate  $L_{0}$  of the smoothness constant  $L$ , run gradient descent for  $T$  steps, and return the average point. We restart and repeat this for  $N$  times, and return the point with the minimum function value. So long as  $N \geq \log \frac{L}{L_0}$ , we will return a point with loss satisfying eq. (1) at the cost of only an additional logarithmic factor. This trick and similar variants of it appear in the literature on prediction with expert advice and online learning (Cesa-Bianchi et al., 1997; Cesa-Bianchi and Lugosi, 2006; Hazan and Megiddo, 2007). It is not even needed to estimate  $N$  in some cases, as the restarting can be done adaptively (Streeter and McMahan, 2012). In practice, however, the performance of doubling trick suffers from restarting the optimization process and throwing away useful that could be used to guide the algorithm.
Parameter-free methods. Throughout this paper, we use the term "parameter-free algorithms" to describe optimization algorithms that do not have any tuning parameters. We specifically consider only the deterministic setting with a compact domain. As mentioned before, Carmon and Hinder (2022) develop an elegant parameter-free and adaptive method based on bisection search. Bisection search, similar to grid search, throws away the progress of several optimization runs and restarts, which may hinder their practical performance. Ivgi et al. (2023); Defazio and Mishchenko (2023) recently developed variants of gradient descent that are parameter-free when  $f$  is Lipschitz. However, D-Adaptation (Defazio and Mishchenko, 2023) has no known guarantee under smoothness, while DoG (Ivgi et al., 2023) was only recently (after the initial release of this paper) shown to adapt to smoothness. We compare against the convergence guarantees of DoG in Section 3.2. For smooth functions, Malitsky and Mishchenko (2020) develop AdGD, a method that efficiently estimates the smoothness parameter on-the-fly from the training trajectory. AdGD is parameter-free and matches the convergence of gradient descent but has no known guarantees for certain classes of Lipschitz functions. A proximal extension of this method has been proposed by Latafat et al. (2023).
Parameter-free methods in online learning. In the online learning literature, the term "parameter-free algorithms" was originally used to describe another class of algorithms that adapt to the unknown distance to the optimal solution (but can still have other tuning parameters such as Lipschitz constant). When the Lipschitz parameter is known, approaches from online convex optimization such as coin betting (Orabona and Pál, 2016), exponentiated gradient (Streeter and McMahan, 2012; Orabona, 2013), and others (McMahan and Orabona, 2014; Orabona and Cutkosky, 2020; Orabona and Pál, 2021; Orabona and Tommasi, 2017) yield rates that match gradient descent up to logarithmic factors. Knowledge of the Lipschitz constant can be removed either by using careful restarting schemes (Mhammedi et al., 2019; Mhammedi and Koolen, 2020), or adaptive clipping on top of coin betting (Cutkosky, 2019). For optimization in the deterministic setting, it is later clarified that, by leveraging the normalization techniques developed in (Levy, 2017), the aforementioned online learning algorithms can be used without knowing other tuning parameters (i.e., achieve "parameter-free" in the sense of this paper) for optimizing both Lipschitz functions (Orabona and Pál, 2021) and smooth functions (Orabona, 2023). Concretely, as shown in Orabona (2023) (which appears after the initial release of this paper), combining algorithms in Streeter and McMahan (2012); Orabona and Pál (2016) with normalization techniques (Levy, 2017) yields new algorithms that are also search-free, parameter-free (in the sense of this paper), and universal. However, these algorithms are rather different from DoWG in algorithmic style: these algorithms only use normalized gradients while DoWG does use the magnitudes of the gradients; DoWG falls in the category of gradient descent algorithms with adaptive learning rate, while these algorithms do not.
Line search. As mentioned before, line-search-based algorithms are universal and theoretically grounded (Nesterov, 2014) but are often expensive in practice (Malitsky and Mishchenko, 2020).
AdaGrad family of methods. Li and Orabona (2019) study a variant of the AdaGrad stepsizes in the stochastic convex and non-convex optimization and show convergence when the stepsize is tuned to depend on the smoothness constant. Levy et al. (2018) show that when the stepsize is tuned properly to the diameter of the domain  $\mathcal{X}$  in the constrained convex case, AdaGrad-Norm adapts to smoothness. Ene et al. (2021) extend this to AdaGrad and other algorithms, and also to variational inequalities. Ward et al. (2019); Traore and Pauwels (2021) show the convergence of AdaGrad-Norm for any stepsize for non-convex (resp. convex) optimization, but in the worst case the dependence on the smoothness constant is worse than gradient descent. Liu et al. (2022) show that AdaGrad-Norm converges in the unconstrained setting when  $f$  is quasi-convex, but their guarantee is worse than gradient descent. We remark that all AdaGrad-style algorithms mentioned above require tuning stepsizes, and are thus not parameter-free.
Alternative justifications for normalization. There are other justifications for why adaptive methods work outside of universality. Zhang et al. (2020a) study a generalized smoothness condition and show that in this setting tuned clipped gradient descent can outperform gradient descent. Because the effective stepsize used in clipped gradient descent is only a constant factor away from the effective stepsize in normalized gradient descent, (Zhang et al., 2020a), also show that this improvement holds for NGD. Zhang et al. (2020b) observe that gradient clipping and normalization methods outperform SGD when the stochastic gradient noise distribution is heavy-tailed. However, Kunstner et al. (2023) later observe that adaptive methods still do well even when the effect of the noise is limited.
3 Algorithms and theory
In this section we first review the different forms of adaptivity in gradient descent and normalized gradient descent, and then introduce our proposed algorithm DoWG. The roadmap for the rest of the paper is as follows: we first review the convergence of gradient descent in the Lipschitz and smooth settings, and highlight the problem of divergence under stepsize misspecification, and how normalization fixes that. Then, we introduce our main new algorithm, DoWG, and give our main theoretical guarantees for the algorithm. Finally, we evaluate the performance of DoWG on practical machine learning problems.
3.1 Baselines: gradient descent and normalized gradient descent
We start our investigation with the standard Gradient Descent (GD) algorithm:
$$
x _ {t + 1} = \Pi_ {\mathcal {X}} \left(x _ {t} - \eta \nabla f \left(x _ {t}\right)\right), \tag {GD}
$$
where  $\Pi_{\mathcal{X}}$  is the projection on  $\mathcal{X}$  (when  $\mathcal{X} = \mathbb{R}^d$ , this is just the identity operator). The iterations (GD) require specifying the stepsize  $\eta > 0$ . When  $f$  is  $G$ -Lipschitz, gradient descent achieves the following standard convergence guarantee:
Theorem 1. Suppose that  $f$  is convex with minimizer  $x_*$ . Let  $f_* = f(x_*)$ . Let  $D_0 \stackrel{\text{def}}{=} \|x_0 - x_*\|$  be the initial distance to the optimum. Denote by  $\hat{x}_T = \frac{1}{T} \sum_{t=0}^{T-1} x_t$  the average iterate returned by  $GD$ . Then:
(Bubeck, 2015) If  $f$  is  $G$ -Lipschitz, the average iterate satisfies for any stepsize  $\eta > 0$ :
$$
f \left(\bar {x} _ {T}\right) - f _ {*} \leq \frac {D _ {0} ^ {2}}{\eta T} + \frac {\eta G ^ {2}}{2}, \tag {2}
$$
(Nesterov, 2018) If  $f$  is  $L$ -smooth, then for all  $\eta < \frac{2}{L}$  the average iterate satisfies
$$
f \left(\hat {x} _ {T}\right) - f _ {*} \leq \frac {2 L D _ {0} ^ {2}}{4 + T \eta L (2 - L \eta)}. \tag {3}
$$
Minimizing eq. (2) over  $\eta$  gives  $f(\bar{x}_T) - f_* = \mathcal{O}\left(\frac{D_0G}{\sqrt{T}}\right)$  with  $\eta = \frac{D_0}{G\sqrt{T}}$ . We have several remarks to make about this rate for gradient descent. First, the optimal stepsize depends on both the distance
to the optimum  $D_0$  and the Lipschitz constant  $G$ , and in fact, this rate is in general optimal (Nesterov, 2018, Theorem 3.2.1). Moreover, if we misspecify  $D_0$  or  $G$  while tuning  $\eta$ , this does not in general result in divergence but may result in a slower rate of convergence. On the other hand, for the smooth setting the optimal stepsize is  $\eta = \frac{1}{L}$  for which  $f(x_{T}) - f_{*} \leq \mathcal{O}\left(\frac{LD_{0}^{2}}{T}\right)$ . Unfortunately, to obtain this rate we have to estimate the smoothness constant  $L$  in order to choose a stepsize  $\eta < \frac{2}{L}$ , and this dependence is hard: if we overshoot the upper bound  $\frac{2}{L}$ , the iterations of gradient descent can diverge very quickly, as shown by Figure 1. Therefore, GD with a constant stepsize cannot be universal: we have to set the stepsize differently for smooth and nonsmooth objectives.
(a) Convergence with  $\eta = \frac{1.9}{L}$
(b) Divergence with  $\eta = \frac{2.1}{L}$
Figure 1: Two trajectories of gradient descent on the one-dimensional quadratic  $f(x) = \frac{Lx^2}{2}$ , with  $L = 100$ .
Normalized Gradient Descent (NGD) (Shor, 2012) consists of iterates of the form
$$
x _ {t + 1} = \Pi_ {\mathcal {X}} \left(x _ {t} - \eta \frac {\nabla f \left(x _ {t}\right)}{\| \nabla f \left(x _ {t}\right) \|}\right). \tag {NGD}
$$
The projection step above is not necessary, and the results for NGD also hold in the unconstrained setting where the projection on  $\mathcal{X} = \mathbb{R}^d$  is just the identity. NGD has many benefits: it can escape saddle points that GD may take arbitrarily long times to escape (Murray et al., 2019), and can minimize functions that are quasi-convex and only locally Lipschitz (Hazan et al., 2015). One of the main benefits of normalized gradient descent is that normalization makes the method scale-free: multiplying  $f$  by a constant factor  $\alpha > 0$  and minimizing  $\alpha f$  does not change the method's trajectory at all. This allows it to adapt to the Lipschitz constant  $G$  in nonsmooth optimization as well as the smoothness constant  $L$  for smooth objectives, as the following theorem states:
Theorem 2. Under the same conditions as Theorem 1, the iterations generated by generated by (NGD) satisfy after  $T$  steps satisfy:
(Nesterov, 2018) If  $f$  is  $G$ -Lipschitz, the minimal function suboptimality satisfies
$$
\min  _ {k \in \{0, 1, \dots , T - 1 \}} \left[ f \left(x _ {k}\right) - f _ {*} \right] \leq G \left[ \frac {D _ {0} ^ {2}}{2 \eta T} + \frac {\eta}{2} \right], \tag {4}
$$
where  $D_0 \stackrel{\text{def}}{=} \|x_0 - x_*\|$ .
Levy, 2017; Grimmer, 2019) If  $f$  is  $L$ -Lipschitz, the minimal function suboptimality satisfies
$$
\min  _ {k = 0, \dots , T - 1} \left[ f \left(x _ {k}\right) - f _ {*} \right] \leq \frac {L}{2} \left[ \frac {D _ {0} ^ {2}}{2 \eta T} + \frac {\eta}{2} \right] ^ {2}. \tag {5}
$$
Tuning eq. (4) in  $\eta$  gives  $\eta = \frac{D_0}{\sqrt{T}}$ , and the stepsize is also optimal for eq. (5). This gives a convergence rate of  $\frac{D_0G}{\sqrt{T}}$  when  $f$  is Lipschitz and  $\frac{D_0^2L}{T}$  when  $f$  is smooth. Observe that NGD matches the dependence of gradient descent on  $G$  and  $L$  without any knowledge of it. Furthermore that, unlike GD where the optimal stepsize is  $\frac{1}{L}$  in the smooth setting and  $\frac{D_0}{G\sqrt{T}}$  in the nonsmooth
setting. The optimal stepsize for NGD is the same in both cases. Therefore, NGD is universal: the same method with the same stepsize adapts to nonsmooth and smooth objectives. Moreover, misspecification of the stepsize in NGD does not result in divergence, but just slower convergence. Another interesting property is that we only get a guarantee on the best iterate: this might be because NGD is non-monotonic, as Figure 2 (a) shows.
Figure 2: NGD iterations on  $\ell_2$ -regularized linear regression on the mushrooms dataset from LibSVM (Chang and Lin, 2011) with  $\eta = 0.1$ . Top (a) shows the function suboptimality over time. Observe that as the number of iterations grow, the method becomes non-monotonic. Bottom (b) shows the effective stepsize  $\eta_{\mathrm{eff},t} = \frac{0.1}{\|\nabla f(x_t)\|}$  over time.
Edge of Stability Phenomena. We may reinterpret NGD with stepsize  $\eta$  as simply GD with a time-varying "effective stepsize"  $\eta_{\mathrm{eff},t} = \frac{\eta}{\|\nabla f(x_t)\|}$ . We plot this effective stepsize for an  $\ell_2$ -regularized linear regression problem in Figure 2 (b). Observe that the stepsize sharply increases, then decreases until it starts oscillating around  $\frac{2}{L}$ . Recall that  $\frac{2}{L}$  is the edge of stability for gradient descent: its iterates diverge when the stepsize crosses this threshold. Arora et al. (2022) observe this phenomenon for NGD, and give a detailed analysis of it under several technical assumptions and when the iterates are close to the manifold of local minimizers.
Theorem 2 offers an alternative, global, and less explicit explanation of this phenomenon: NGD matches the optimal gradient descent rate, and in order to do so it must drive the effective stepsize to be large. Specifically, suppose that we use the optimal stepsize  $\eta = \frac{D_0}{\sqrt{T}}$ , and call the best iterate returned by NGD  $x_{\tau}$ . Then  $x_{\tau}$  satisfies  $f(x_{\tau}) - f_{*} \leq \frac{D_0^2 L}{T}$  and therefore by smoothness
$$
\| \nabla f (x _ {\tau}) \| \leq \sqrt {2 L (f (x _ {\tau}) - f _ {*})} \leq \sqrt {\frac {L ^ {2} D _ {0} ^ {2}}{T}} = \frac {L D _ {0}}{\sqrt {T}} = L \eta .
$$
This implies  $\eta_{\mathrm{eff},\tau} = \frac{\eta}{\|\nabla f(x_t)\|} \geq \frac{1}{L}$ . Therefore the effective stepsize at convergence is forced to grow to  $\Omega \left(\frac{1}{L}\right)$ . But if the effective stepsize increases too much and crosses the threshold  $\frac{2}{L}$ , the gradient norms start diverging, forcing the effective stepsize back down. Thus, NGD is self-stabilizing. We note that Edge of Stability phenomenon is not unique to NGD, and GD itself trains at the edge of stability for more complicated models where the smoothness also varies significantly over training (Cohen et al., 2021; Damian et al., 2023).
3.2 DoWG
We saw in the last section that NGD adapts to both the Lipschitz constant  $G$  and the smoothness  $L$ , but we have to choose  $\eta$  to vary with the distance to the optimum  $D_0 = \| x_0 - x_*\|$ . In this section, we develop a novel algorithm that adaptively estimates the distance to the optimum, and attains the optimal convergence rate of gradient descent for constrained convex and smooth optimization up to a logarithmic factor. Our algorithm builds upon the recently proposed Distance over Gradients (DoG)
algorithm developed by Ivgi et al. (2023). We call the new method DoWG (Distance over Weighted Gradients), and we describe it as Algorithm 1 below.
Algorithm 1: DoWG: Distance over Weighted Gradients
1 Input: initial point  $x_0\in \mathcal{X}$  Initial distance estimate  $r_\epsilon >0$    
2 Initialize  $v_{-1} = 0,r_{-1} = r_{\epsilon}$    
3 for  $t = 0,1,2,\ldots ,T - 1$  do   
4 Update distance estimator:  $\bar{r}_t\gets \max (\| x_t - x_0\|$ $\bar{r}_{t - 1})$    
5 Update weighted gradient sum:  $v_{t}\gets v_{t - 1} + \bar{r}_{t}^{2}\| \nabla f(x_{t})\|^{2}$    
6 Set the stepsize:  $\eta_t\gets \frac{\bar{r}_t^2}{\sqrt{v_t}}$    
7 Gradient descent step:  $x_{t + 1}\gets \Pi_{\mathcal{X}}(x_t - \eta_t\nabla f(x_t))$    
8 end