Title: DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method
Venue: NeurIPS 2023
Accepted: yes
Keywords: normalized gradient descent

gradient descent

adagrad

adaptive optimization

parameter-free

smooth optimization

convex optimization

edge of stability
Abstract: This paper proposes a new easy-to-implement parameter-free gradient-based optimizer: DoWG (Distance over Weighted Gradients). We prove that DoWG is efficient---matching the convergence rate of optimally tuned gradient descent in convex optimization up to a logarithmic factor without tuning any parameters, and universal---automatically adapting to both smooth and nonsmooth problems. While popular algorithms following the AdaGrad framework compute a running average of the squared gradients, DoWG maintains a new distance-based weighted version of the running average, which is crucial to achieve the desired properties. To complement our theory, we also show empirically that DoWG trains at the edge of stability, and validate its effectiveness on practical machine learning tasks.

DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method

Abstract

This paper proposes a new easy-to-implement parameter-free gradient-based optimizer: DoWG (Distance over Weighted Gradients). We prove that DoWG is efficient -matching the convergence rate of optimally tuned gradient descent in convex optimization up to a logarithmic factor without tuning any parameters, and universal -automatically adapting to both smooth and nonsmooth problems. While popular algorithms following the AdaGrad framework compute a running average of the squared gradients to use for normalization, DoWG maintains a new distance-based weighted version of the running average, which is crucial to achieve the desired properties. To complement our theory, we also show empirically that DoWG trains at the edge of stability, and validate its effectiveness on practical machine learning tasks.

1 Introduction

We study the fundamental optimization problem

where f is a convex function, and X is a convex, closed, and bounded subset of R d . We assume f has at least one minimizer x ∗ ∈ X . We focus on gradient descent and its variants, as they are widely adopted and scale well when the model dimensionality d is large (Bottou et al., 2018). The optimization problem (OPT) finds many applications: in solving linear systems, logistic regression, support vector machines, and other areas of machine learning (Boyd and Vandenberghe, 2004). Equally important, methods designed for (stochastic) convex optimization also influence the intuition for and design of methods for nonconvex optimization- for example, momentum (Polyak, 1964), AdaGrad (Duchi et al., 2010), and Adam (Kingma and Ba, 2015) were all first analyzed in the convex optimization framework.

As models become larger and more complex, the cost and environmental impact of training have rapidly grown as well (Sharir et al., 2020; Patterson et al., 2021). Therefore, it is vital that we develop more efficient and effective methods of solving machine learning optimization tasks. One of the chief challenges in applying gradient-based methods is that they often require tuning one or more stepsize parameters (Goodfellow et al., 2016), and the choice of stepsize can significantly influence a method's convergence speed as well as the quality of the obtained solutions, especially in deep learning (Wilson et al., 2017).

The cost and impact of hyperparameter tuning on the optimization process have led to significant research activity in designing parameter-free and adaptive optimization methods in recent years, see e.g. (Orabona and Cutkosky, 2020; Carmon and Hinder, 2022) and the references therein.

We say an algorithm is universal if it adapts to many different problem geometries or regularity conditions on the function f (Nesterov, 2014; Levy et al., 2018; Grimmer, 2022). In this work, we focus on two regularity conditions: (a) f Lipschitz and (b) f smooth. Lipschitz functions

have a bounded rate of change, that is, there exists some G > 0 such that for all x, y ∈ X we have | f ( x ) -f ( y ) | ≤ G ∥ x -y ∥ . The Lipschitz property is beneficial for the convergence of gradient-based optimization algorithms. They converge even faster on smooth functions, which have continuous derivatives; that is, there exists some L > 0 such that for all x, y ∈ X we have ∥∇ f ( x ) -∇ f ( y ) ∥ ≤ L ∥ x -y ∥ . Smoothness leads to faster convergence of gradient-based methods than the Lipschitz property. Universality is a highly desirable property because in practice the same optimization algorithms are often used for both smooth and nonsmooth optimization (e.g. optimizing both ReLU and smooth networks).

The main question of our work is as follows:

Can we design a universal, parameter-free gradient descent method for (OPT)?

Existing universal variants of gradient descent either rely on line search (Nesterov, 2014; Grimmer, 2022), bisection subroutines (Carmon and Hinder, 2022), or are not parameter-free (Hazan and Kakade, 2019; Levy et al., 2018; Kavis et al., 2019). Line search algorithms are theoretically strong, achieving the optimal convergence rates in both the nonsmooth and smooth settings with only an extra log factor. Through an elegant application of bisection search, Carmon and Hinder (2022) design a parameter-free method whose convergence is only double-logarithmically worse than gradient descent with known problem parameters. However, this method requires resets, i.e. restarting the optimization process many times, which can be very expensive in practice. Therefore, we seek a universal, parameter-free gradient descent method for (OPT) with no search subroutines.

Our contributions. We provide a new algorithm that meets the above requirements. Our main contribution is a new universal, parameter-free gradient descent method with no search subroutines. Building upon the recently proposed Distance-over-Gradients (DoG) algorithm (Ivgi et al., 2023), we develop a new method, DoWG (Algorithm 1), that uses a different stepsize with adaptively weighted gradients. We show that DoWG automatically matches the performance of gradient descent on (OPT) up to logarithmic factors with no stepsize tuning at all. This holds in both the nonsmooth setting (Theorem 3) and the smooth setting (Theorem 4). Finally, we show that DoWG is competitive on real machine learning tasks (see Section 4).

Table 1: A comparison of different adaptive algorithms for solving (OPT). "Universal" means that the algorithm can match the rate of gradient descent on both smooth and nonsmooth objectives up to polylogarithmic factors. "No search" means the algorithm does not reset. "GD framework" refers to algorithms that follow the framework of Gradient Descent.

2 Related Work

There is a lot of work on adaptive and parameter-free approaches for optimization. We summarize the main properties of the algorithms we compare against in Table 1. We enumerate some of the major approaches below:

Polyak stepsize. When f ∗ = f ( x ∗ ) is known, the Polyak stepsize (Polyak, 1987) is a theoreticallygrounded, adaptive, and universal method (Hazan and Kakade, 2019). When f ∗ is not known, Hazan and Kakade (2019) show that an adaptive re-estimation procedure can recover the optimal convergence rate up to a log factor when f is Lipschitz. Loizou et al. (2021) study the Polyak stepsize in stochastic non-convex optimization. Orvieto et al. (2022) show that a variant of the Polyak stepsize with decreasing stepsizes can recover the convergence rate of gradient descent, provided the stepsize is initialized properly. Unfortunately, this initialization requirement makes the method not parameter-free.

The doubling trick. The simplest way to make an algorithm parameter-free is the doubling-trick. For example, for gradient descent for L -smooth and convex optimization, the stepsize η = 1 L results in the convergence rate of

where D 0 = ∥ x 0 -x ∗ ∥ . We may therefore start with a small estimate L 0 of the smoothness constant L , run gradient descent for T steps, and return the average point. We restart and repeat this for N times, and return the point with the minimum function value. So long as N ≥ log L L 0 , we will return a point with loss satisfying eq. (1) at the cost of only an additional logarithmic factor. This trick and similar variants of it appear in the literature on prediction with expert advice and online learning (Cesa-Bianchi et al., 1997; Cesa-Bianchi and Lugosi, 2006; Hazan and Megiddo, 2007). It is not even needed to estimate N in some cases, as the restarting can be done adaptively (Streeter and McMahan, 2012). In practice, however, the performance of doubling trick suffers from restarting the optimization process and throwing away useful that could be used to guide the algorithm.

Parameter-free methods. Throughout this paper, we use the term 'parameter-free algorithms' to describe optimization algorithms that do not have any tuning parameters. We specifically consider only the deterministic setting with a compact domain. As mentioned before, Carmon and Hinder (2022) develop an elegant parameter-free and adaptive method based on bisection search. Bisection search, similar to grid search, throws away the progress of several optimization runs and restarts, which may hinder their practical performance. Ivgi et al. (2023); Defazio and Mishchenko (2023) recently developed variants of gradient descent that are parameter-free when f is Lipschitz. However, D-Adaptation (Defazio and Mishchenko, 2023) has no known guarantee under smoothness, while DoG (Ivgi et al., 2023) was only recently (after the initial release of this paper) shown to adapt to smoothness. We compare against the convergence guarantees of DoG in Section 3.2. For smooth functions, Malitsky and Mishchenko (2020) develop AdGD, a method that efficiently estimates the smoothness parameter on-the-fly from the training trajectory. AdGD is parameter-free and matches the convergence of gradient descent but has no known guarantees for certain classes of Lipschitz functions. A proximal extension of this method has been proposed by Latafat et al. (2023).

Parameter-free methods in online learning. In the online learning literature, the term 'parameterfree algorithms' was originally used to describe another class of algorithms that adapt to the unknown distance to the optimal solution (but can still have other tuning parameters such as Lipschitz constant). When the Lipschitz parameter is known, approaches from online convex optimization such as coin betting (Orabona and Pál, 2016), exponentiated gradient (Streeter and McMahan, 2012; Orabona, 2013), and others (McMahan and Orabona, 2014; Orabona and Cutkosky, 2020; Orabona and Pál, 2021; Orabona and Tommasi, 2017) yield rates that match gradient descent up to logarithmic factors. Knowledge of the Lipschitz constant can be removed either by using careful restarting schemes (Mhammedi et al., 2019; Mhammedi and Koolen, 2020), or adaptive clipping on top of coin betting (Cutkosky, 2019). For optimization in the deterministic setting, it is later clarified that, by leveraging the normalization techniques developed in (Levy, 2017), the aforementioned online learning algorithms can be used without knowing other tuning parameters (i.e., achieve 'parameterfree' in the sense of this paper) for optimizing both Lipschitz functions (Orabona and Pál, 2021) and smooth functions (Orabona, 2023). Concretely, as shown in Orabona (2023) (which appears after the initial release of this paper), combining algorithms in Streeter and McMahan (2012); Orabona and Pál (2016) with normalization techniques (Levy, 2017) yields new algorithms that are also search-free, parameter-free (in the sense of this paper), and universal. However, these algorithms are rather different from DoWG in algorithmic style: these algorithms only use normalized gradients while DoWG does use the magnitudes of the gradients; DoWG falls in the category of gradient descent algorithms with adaptive learning rate, while these algorithms do not.

Line search. As mentioned before, line-search-based algorithms are universal and theoretically grounded (Nesterov, 2014) but are often expensive in practice (Malitsky and Mishchenko, 2020).

AdaGrad family of methods. Li and Orabona (2019) study a variant of the AdaGrad stepsizes in the stochastic convex and non-convex optimization and show convergence when the stepsize is tuned to depend on the smoothness constant. Levy et al. (2018) show that when the stepsize is tuned properly to the diameter of the domain X in the constrained convex case, AdaGrad-Norm adapts to smoothness. Ene et al. (2021) extend this to AdaGrad and other algorithms, and also to variational inequalities. Ward et al. (2019); Traoré and Pauwels (2021) show the convergence of AdaGrad-Norm for any stepsize for non-convex (resp. convex) optimization, but in the worst case the dependence on the smoothness constant is worse than gradient descent. Liu et al. (2022) show that AdaGrad-Norm converges in the unconstrained setting when f is quasi-convex, but their guarantee is worse than gradient descent. We remark that all AdaGrad-style algorithms mentioned above require tuning stepsizes, and are thus not parameter-free.

Alternative justifications for normalization. There are other justifications for why adaptive methods work outside of universality. Zhang et al. (2020a) study a generalized smoothness condition and show that in this setting tuned clipped gradient descent can outperform gradient descent. Because the effective stepsize used in clipped gradient descent is only a constant factor away from the effective stepsize in normalized gradient descent, (Zhang et al., 2020a), also show that this improvement holds for NGD. Zhang et al. (2020b) observe that gradient clipping and normalization methods outperform SGD when the stochastic gradient noise distribution is heavy-tailed. However, Kunstner et al. (2023) later observe that adaptive methods still do well even when the effect of the noise is limited.

3 Algorithms and theory

In this section we first review the different forms of adaptivity in gradient descent and normalized gradient descent, and then introduce our proposed algorithm DoWG. The roadmap for the rest of the paper is as follows: we first review the convergence of gradient descent in the Lipschitz and smooth settings, and highlight the problem of divergence under stepsize misspecification, and how normalization fixes that. Then, we introduce our main new algorithm, DoWG, and give our main theoretical guarantees for the algorithm. Finally, we evaluate the performance of DoWG on practical machine learning problems.

3.1 Baselines: gradient descent and normalized gradient descent

We start our investigation with the standard Gradient Descent (GD) algorithm:

where Π X is the projection on X (when X = R d , this is just the identity operator). The iterations (GD) require specifying the stepsize η > 0 . When f is G -Lipschitz, gradient descent achieves the following standard convergence guarantee:

Theorem 1. Suppose that f is convex with minimizer x ∗ . Let f ∗ = f ( x ∗ ) . Let D 0 def = ∥ x 0 -x ∗ ∥ be the initial distance to the optimum. Denote by ˆ x T = 1 T ∑ T -1 t =0 x t the average iterate returned by GD. Then:

(Bubeck, 2015) If f is G -Lipschitz, the average iterate satisfies for any stepsize η > 0 :

(Nesterov, 2018) If f is L -smooth, then for all η < 2 L the average iterate satisfies

Minimizing eq. (2) over η gives f (¯ x T ) -f ∗ = O ( D 0 G √ T ) with η = D 0 G √ T . We have several remarks to make about this rate for gradient descent. First, the optimal stepsize depends on both the distance

to the optimum D 0 and the Lipschitz constant G , and in fact, this rate is in general optimal (Nesterov, 2018, Theorem 3.2.1). Moreover, if we misspecify D 0 or G while tuning η , this does not in general result in divergence but may result in a slower rate of convergence. On the other hand, for the smooth setting the optimal stepsize is η = 1 L for which f ( x T ) -f ∗ ≤ O ( LD 2 0 T ) . Unfortunately, to obtain this rate we have to estimate the smoothness constant L in order to choose a stepsize η < 2 L , and this dependence is hard : if we overshoot the upper bound 2 L , the iterations of gradient descent can diverge very quickly, as shown by Figure 1. Therefore, GD with a constant stepsize cannot be universal : we have to set the stepsize differently for smooth and nonsmooth objectives.

0.75

0.50

0.25

0.00

0.25

0.50

0.75

x

0

5

10

15

20

25

30

35

40

f(x)

f(x)

Initial Point

Trajectory

Final Point

(a) Convergence with

η

=

1

.

9

L

1.0

0.5

0.0

0.5

1.0

1.5

x

0

20

40

60

80

100

f(x)

f(x)

Initial Point

Trajectory

Final Point

(b) Divergence with

η

=

2

.

1

L

Normalized Gradient Descent (NGD) (Shor, 2012) consists of iterates of the form

The projection step above is not necessary, and the results for NGD also hold in the unconstrained setting where the projection on X = R d is just the identity. NGD has many benefits: it can escape saddle points that GD may take arbitrarily long times to escape (Murray et al., 2019), and can minimize functions that are quasi-convex and only locally Lipschitz (Hazan et al., 2015). One of the main benefits of normalized gradient descent is that normalization makes the method scale-free: multiplying f by a constant factor α > 0 and minimizing αf does not change the method's trajectory at all. This allows it to adapt to the Lipschitz constant G in nonsmooth optimization as well as the smoothness constant L for smooth objectives, as the following theorem states:

Theorem 2. Under the same conditions as Theorem 1, the iterations generated by generated by (NGD) satisfy after T steps satisfy:

(Nesterov, 2018) If f is G -Lipschitz, the minimal function suboptimality satisfies

where D 0 def = ∥ x 0 -x ∗ ∥ .

(Levy, 2017; Grimmer, 2019) If f is L -Lipschitz, the minimal function suboptimality satisfies

Tuning eq. (4) in η gives η = D 0 √ T , and the stepsize is also optimal for eq. (5). This gives a convergence rate of D 0 G √ T when f is Lipschitz and D 2 0 L T when f is smooth. Observe that NGD matches the dependence of gradient descent on G and L without any knowledge of it. Furthermore that, unlike GD where the optimal stepsize is 1 L in the smooth setting and D 0 G √ T in the nonsmooth

setting. The optimal stepsize for NGD is the same in both cases. Therefore, NGD is universal : the same method with the same stepsize adapts to nonsmooth and smooth objectives. Moreover, misspecification of the stepsize in NGD does not result in divergence, but just slower convergence. Another interesting property is that we only get a guarantee on the best iterate: this might be because NGD is non-monotonic, as Figure 2 (a) shows.

0

50

100

150

200

250

Steps

10

1

10 0

f

(

x

)

f

*

Function suboptimality

0

50

100

150

200

250

10

5

Effective step size

Effective stepsize

2/L

Steps

Edge of Stability Phenomena. We may reinterpret NGD with stepsize η as simply GD with a timevarying 'effective stepsize' η eff ,t = η ∥∇ f ( x t ) ∥ . We plot this effective stepsize for an ℓ 2 -regularized linear regression problem in Figure 2 (b). Observe that the stepsize sharply increases, then decreases until it starts oscillating around 2 L . Recall that 2 L is the edge of stability for gradient descent: its iterates diverge when the stepsize crosses this threshold. Arora et al. (2022) observe this phenomenon for NGD, and give a detailed analysis of it under several technical assumptions and when the iterates are close to the manifold of local minimizers.

Theorem 2 offers an alternative, global, and less explicit explanation of this phenomenon: NGD matches the optimal gradient descent rate, and in order to do so it must drive the effective stepsize to be large. Specifically, suppose that we use the optimal stepsize η = D 0 √ T , and call the best iterate returned by NGD x τ . Then x τ satisfies f ( x τ ) -f ∗ ≤ D 2 0 L T and therefore by smoothness

This implies η eff ,τ = η ∥∇ f ( x t ) ∥ ≥ 1 L . Therefore the effective stepsize at convergence is forced to grow to Ω ( 1 L ) . But if the effective stepsize increases too much and crosses the threshold 2 L , the gradient norms start diverging, forcing the effective stepsize back down. Thus, NGD is selfstabilizing . We note that Edge of Stability phenomenon is not unique to NGD, and GD itself trains at the edge of stability for more complicated models where the smoothness also varies significantly over training (Cohen et al., 2021; Damian et al., 2023).

3.2 DoWG

We saw in the last section that NGD adapts to both the Lipschitz constant G and the smoothness L , but we have to choose η to vary with the distance to the optimum D 0 = ∥ x 0 -x ∗ ∥ . In this section, we develop a novel algorithm that adaptively estimates the distance to the optimum, and attains the optimal convergence rate of gradient descent for constrained convex and smooth optimization up to a logarithmic factor. Our algorithm builds upon the recently proposed Distance over Gradients (DoG)

algorithm developed by Ivgi et al. (2023). We call the new method DoWG (Distance over Weighted Gradients), and we describe it as Algorithm 1 below.

Algorithm 1: DoWG: Distance over Weighted Gradients

2

DoWG uses the same idea of estimating the distance from the optimum by using the distance from the initial point as a surrogate, but instead of using the square root of the running gradient sum G t = ∑ t k =0 ∥∇ f ( x k ) ∥ 2 as the normalization, DoWG uses the square root of the weighted gradient sum v t = ∑ t k =0 ¯ r 2 k ∥∇ f ( x k ) ∥ 2 . Observe that because the estimated distances ¯ r t are monotonically increasing, later gradients have a larger impact on v t than earlier ones compared to G t . Therefore, we may expect this to aid the method in adapting to the local properties of the problem once far away from the initialization x 0 . We note that using a weighted sum of gradients is not new: AcceleGrad (Levy et al., 2018) uses time-varying polynomial weights and Adam (Kingma and Ba, 2015) uses exponentially decreasing weights. The difference is that DoWG chooses the weights adaptively based on the running distance from the initial point. This use of distance-based weighted averaging is new, and we are not aware of any previous methods that estimate the running gradient sum in this manner.

Nonsmooth analysis. The next theorem shows that DoWG adapts to the Lipschitz constant G and the diameter D of the set X if the function f is nonsmooth but G -Lipschitz. We use the notation log + x = log x +1 following (Ivgi et al., 2023).

Theorem 3. (DoWG, Lipschitz f ). Suppose that the function f is convex, G -Lipschitz, and has a minimizer x ∗ ∈ X . Suppose that the domain X is a closed convex set of (unknown) diameter D > 0 . Let r ϵ < D . Then the output of Algorithm 1 satisfies for some t ∈ { 0 , 1 , . . . , T -1 }

where ¯ x t def = 1 ∑ t -1 k =0 r 2 k ∑ t -1 k =0 r 2 k x k is a weighted average of the iterates returned by the algorithm.

Discussion of convergence rate. DoWG matches the optimal O ( DG √ T ) rate of tuned GD and tuned NGD up to an extra logarithmic factor. We note that the recently proposed algorithms DoG (Ivgi et al., 2023) and D-Adaptation (Defazio and Mishchenko, 2023) achieve a similar rate in this setting.

Comparison with DoG. As we discussed before, DoWG uses an adaptively weighted sum of gradients for normalization compared to the simple sum used by DoG. In addition, DoG uses the stepsize ¯ r t √ ∑ t k =0 ∥∇ f ( x k ) ∥ 2 , whereas the DoWG stepsize is pointwise larger: since ¯ r 2 k is monotonically increasing in k we have

Of course, the pointwise comparison may not reflect the practical performance of the algorithms, since after the first iteration the sequence of iterates x 2 , x 3 , . . . generated by the two algorithms can be very different. We observe in practice that DoWG is in general more aggressive, and uses larger stepsizes than both DoG and D-Adaptation (see Section 4).

Smooth analysis. Our next theorem shows that DoWG adapts to the smoothness constant and the diameter D of the set X .

Theorem 4. (DoWG, Smooth f ). Suppose that the function f is L -smooth, convex, and has a minimizer x ∗ ∈ X . Suppose that the domain X is a closed convex set of diameter D > 0 . Let r ϵ < D . Then the output of Algorithm 1 satisfies for some t ∈ { 0 , 1 , . . . , T -1 }

where ¯ x t def = 1 ∑ t -1 k =0 r 2 k ∑ t -1 k =0 r 2 k x k is a weighted average of the iterates returned by the algorithm.

The proof of this theorem and all subsequent results is relegated to the supplementary material. We note that the proof of Theorem 4 uses the same trick used to show the adaptivity of NGD to smoothness: we use the fact that ∥∇ f ( x ) ∥ ≤ √ 2 L ( f ( x ) -f ∗ ) for all x ∈ X applied to a carefully-chosen weighted sum of gradients.

Comparison with GD/NGD. Both well-tuned gradient descent and normalized gradient descent achieve the convergence rate O ( LD 2 0 T ) where D 0 = ∥ x 0 -x ∗ ∥ ≤ D for the constrained convex minimization problem. Theorem 4 shows that DoWG essentially attains the same rate up to the difference between D 0 and D and an extra logarithmic factor. In the worst case, if we initialize far from the optimum, we have D 0 ≃ D and hence the difference is not significant. We note that DoG (Ivgi et al., 2023) suffers from a similar dependence on the diameter D of X , and can diverge in the unconstrained setting, where X is not compact. This can be alleviated by making the stepsize smaller by a polylogarithmic factor. A similar reduction of the stepsize also works for DoWG, and we provide the proof in Section 7 in the supplementary.

Comparison with DoG. After the initial version of this paper, Ivgi et al. (2023) reported a convergence guarantee for the unweighted average ˆ x T = 1 T ∑ T -1 k =0 x k returned by DoG. In particular, Proposition 3 in their work gives the rate

where D 0 = ∥ x 0 -x ∗ ∥ , and where in the second step we used the bound D 0 ≤ D and ¯ r T ≤ D . This rate is the same as that achieved by the weighted average of the DoWG iterates up to an extra logarithmic factor log + D ¯ r ϵ . We note that DoG also has a guarantee in the stochastic setting, provided the gradients are bounded locally with a known constant, while in this work we have focused exclusively on the deterministic setting.

0

50

100

150

200

250

Steps

10

2

10

1

10 0

f

(

x

)

f

*

Function suboptimality

0

50

100

150

200

250

10

10

10

8

10

6

Effective step size

Effective stepsize

2/L

Steps

Edge of Stability. Like NGD, DoWG also tends to increase the stepsize and train at the edge of stability. The intuition from NGD carries over: in order to preserve the convergence rate of GD, DoWG tends to drive the stepsize larger. However, once it overshoots, the gradients quickly diverge, forcing the stepsize back down. Figure 3 shows the performance of DoWG and its stepsize on the same regularized linear regression problem as in Figure 2. Comparing the two figures, we observe that DoWG is also non-monotonic and trains close to the edge of stability, but its stepsize oscillates less than NGD's effective stepsize.

Universality. Theorems 4 and 3 together show that DoWG is universal, i.e. it almost recovers the convergence of gradient descent with tuned stepsizes in both the smooth and nonsmooth settings. As the optimal stepsize for gradient descent can differ significantly between the two settings, we believe that achieving both rates simultaneously without any parameter-tuning or search procedures is a significant strength of DoWG.

4 Experimental results

0

50

100

150

200

Epoch

80

82

84

86

88

90

92

Test accuracy

Adam

DoG

L-DoG

DoWG

DoG-ave

L-DoG-ave

DoWG-ave

0

50

100

150

200

Epoch

80

85

90

95

100

Train accuracy

Adam

DoG

L-DoG

DoWG

DoG-ave

L-DoG-ave

DoWG-ave

0

20

40

60

80

100

Epoch

10

5

10

3

10

1

10

1

Step size

DoG

DoWG

D-Adapt-norm

0

50

100

150

200

Epoch

50

60

70

80

90

Test accuracy

Tuned AdaGrad-norm

Adam

DoG

L-DoG

DoWG

DoG-ave

L-DoG-ave

DoWG-ave

0

50

100

150

200

Epoch

50

60

70

80

90

100

Train accuracy

Tuned AdaGrad-norm

Adam

DoG

L-DoG

DoWG

DoG-ave

L-DoG-ave

DoWG-ave

0

20

40

60

80

100

Epoch

10

3

10

2

10

1

10

0

10

1

Step size

DoG

DoWG

We compare DoWG to DoG, L-DoG from Ivgi et al. (2023), for all of which we also report performance of the polynomially-averaged iterate with power 8 as recommended by Ivgi et al. (2023). We also add comparison against Adam (Kingma and Ba, 2015) with cosine annealing and the standard step size 10 -3 . All methods are used with batch size 256 with no weight decay on a single RTX3090 GPU. We plot the results in Figure 4 with the results averaged over 8 random seeds. We train the VGG11 (Simonyan and Zisserman, 2015) and ResNet-50 (He et al., 2016) neural network architectures on CIFAR10 (Krizhevsky, 2009) using PyTorch (Paszke et al., 2019), and implement 1 DoWG on top of the DoG code 2 . Unsurprisingly, DoWG's estimates of the step size are larger than that of DoG and D-Adapt-norm, which also makes it less stable on ResNet-50. While the last iterate of DoWG gives worse test accuracy than Adam, the average iterate of DoWG often performs better.

Finally, we note that while both neural networks tested are generally nonsmooth, recent work shows local smoothness can significantly influence and be influenced by a method's trajectory (Cohen et al., 2022; Pan and Li, 2022). We believe this adaptivity to smoothness might explain the empirical difference between DoWG and DoG, but leave a rigorous discussion of adaptivity to local smoothness to future work.

References

Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent on edge of stability in deep learning. arXiv preprint arXiv:2205.09745 , abs/2205.09745, 2022. URL https://arXiv.org/abs/2205.09745 . (Cited on page 6)

Léon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review , 60(2):223-311, 2018. doi: 10.1137/16M1080173. URL https://doi. org/10.1137/16M1080173 . (Cited on page 1)

Stephen Boyd and Lieven Vandenberghe. Convex Optimization . Cambridge University Press, 2004. doi: 10.1017/CBO9780511804441. (Cited on page 1)

Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning , 8(3-4):231-357, 2015. doi: 10.1561/2200000050. URL https://doi.org/ 10.1561/2200000050 . (Cited on page 4)

Yair Carmon and Oliver Hinder. Making SGD parameter-free. In Po-Ling Loh and Maxim Raginsky, editors, Conference on Learning Theory, 2-5 July 2022, London, UK , volume 178 of Proceedings of Machine Learning Research , pages 2360-2389. PMLR, 2022. URL https://proceedings. mlr.press/v178/carmon22a.html . (Cited on pages 1, 2, 3, 17, and 18)

Nicolo Cesa-Bianchi and Gabor Lugosi. Prediction, Learning, and Games . Cambridge University Press, USA, 2006. ISBN 0521841089. (Cited on page 3)

Nicolò Cesa-Bianchi, Yoav Freund, David Haussler, David P. Helmbold, Robert E. Schapire, and Manfred K. Warmuth. How to use expert advice. J. ACM , 44(3):427-485, may 1997. ISSN 0004-5411. doi: 10.1145/258128.258179. URL https://doi.org/10.1145/258128.258179 . (Cited on page 3)

Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM transactions on intelligent systems and technology (TIST) , 2(3):1-27, 2011. (Cited on pages 6 and 8)

Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In ICLR . OpenReview.net, 2021. (Cited on page 6)

Jeremy M. Cohen, Behrooz Ghorbani, Shankar Krishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, David Cardoze, Zachary Nado, George E. Dahl, and Justin Gilmer. Adaptive gradient methods at the edge of stability. arXiv preprint arXiv:2207.14484 , abs/2207.14484, 2022. URL https://arXiv.org/abs/2207.14484 . (Cited on page 9)

Ashok Cutkosky. Artificial constraints and hints for unbounded online learning. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 874-894. PMLR, 25-28 Jun 2019. URL https://proceedings.mlr.press/v99/cutkosky19a.html . (Cited on page 3)

Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=nhKHA59gXz . (Cited on page 6)

Aaron Defazio and Konstantin Mishchenko. Learning-rate-free learning by D-adaptation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 7449-7479. PMLR, 23-29 Jul 2023. (Cited on pages 2, 3, 7, and 16)

John C. Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. In Adam Tauman Kalai and Mehryar Mohri, editors, COLT 2010 - The 23rd Conference on Learning Theory, Haifa, Israel, June 27-29, 2010 , pages 257-269. Omnipress, 2010. URL http://colt2010.haifa.il.ibm.com/papers/COLT2010proceedings.pdf# page=265 . (Cited on pages 1 and 2)

Alina Ene, Huy L. Nguyen, and Adrian Vladu. Adaptive gradient methods for constrained convex optimization and variational inequalities. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 7314-7321, 2021. (Cited on pages 2 and 4)

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Practical Methodology , chapter 11. MIT Press, 2016. URL http://www.deeplearningbook.org . (Cited on page 1)

Benjamin Grimmer. Convergence rates for deterministic and stochastic subgradient methods without lipschitz continuity. SIAM Journal on Optimization , 29(2):1350-1365, 2019. doi: 10.1137/ 18M117306X. URL https://doi.org/10.1137/18M117306X . (Cited on page 5)

Benjamin Grimmer. On optimal universal first-order methods for minimizing heterogeneous sums. arXiv preprint arXiv:2208.08549 , abs/2208.08549, 2022. URL https://arXiv.org/abs/2208. 08549 . (Cited on pages 1 and 2)

Vineet Gupta, Tomer Koren, and Yoram Singer. A unified approach to adaptive regularization in online and stochastic optimization. arXiv preprint arXiv:1706.06569 , abs/1706.06569, 2017. URL https://arXiv.org/abs/1706.06569 . (Cited on page 17)

Elad Hazan and Sham Kakade. Revisiting the Polyak step size. arXiv preprint arXiv:1905.00313 , abs/1905.00313, 2019. URL https://arXiv.org/abs/1905.00313 . (Cited on pages 2 and 3)

Elad Hazan and Nimrod Megiddo. Online learning with prior knowledge. In Nader H. Bshouty and Claudio Gentile, editors, Learning Theory , pages 499-513, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg. ISBN 978-3-540-72927-3. (Cited on page 3)

Elad Hazan, Kfir Y. Levy, and Shai Shalev-Shwartz. Beyond convexity: Stochastic quasi-convex optimization. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 1594-1602, 2015. URL https://proceedings.neurips.cc/paper/2015/ hash/934815ad542a4a7c5e8a2dfa04fea9f5-Abstract.html . (Cited on page 5)

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016. doi: 10.1109/CVPR.2016.90. (Cited on page 9)

Maor Ivgi, Oliver Hinder, and Yair Carmon. DoG is SGD's best friend: A parameter-free dynamic step size schedule. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 14465-14499. PMLR, 23-29 Jul 2023. (Cited on pages 2, 3, 7, 8, 9, 15, 16, 17, and 22)

Ali Kavis, Kfir Y. Levy, Francis R. Bach, and Volkan Cevher. UniXGrad: A universal, adaptive algorithm with optimal guarantees for constrained optimization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 6257-6266, 2019. URL https://proceedings.neurips.cc/paper/2019/ hash/88855547570f7ff053fff7c54e5148cc-Abstract.html . (Cited on page 2)

Ahmed Khaled and Peter Richtárik. Better theory for SGD in the nonconvex world. arXiv preprint arXiv:2002.03329 , abs/2002.03329, 2020. URL https://arXiv.org/abs/2002.03329 . (Cited on page 15)

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arXiv.org/abs/1412.6980 . (Cited on pages 1, 2, 7, and 9)

Alex Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009. URL https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf . (Cited on page 9)

Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between SGD and Adam on transformers, but sign descent might be. In The Eleventh International Conference on Learning Representations , 2023. URL https: //openreview.net/forum?id=a65YK0cqH8g . (Cited on page 4)

Puya Latafat, Andreas Themelis, Lorenzo Stella, and Panagiotis Patrinos. Adaptive proximal algorithms for convex optimization under local Lipschitz continuity of the gradient. arXiv preprint arXiv:2301.04431 , 2023. (Cited on page 3)

Kfir Y. Levy. Online to offline conversions, universality and adaptive minibatch sizes. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 1613-1622, 2017. URL https://proceedings.neurips.cc/paper/2017/ hash/ce5140df15d046a66883807d18d0264b-Abstract.html . (Cited on pages 3, 5, and 15)

Kfir Yehuda Levy, Alp Yurtsever, and Volkan Cevher. Online adaptive methods, universality and acceleration. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 6501-6510, 2018. URL https://proceedings.neurips. cc/paper/2018/hash/b0169350cd35566c47ba83c6ec1d6f82-Abstract.html . (Cited on pages 1, 2, 4, 7, 15, and 18)

Haochuan Li, Ali Jadbabaie, and Alexander Rakhlin. Convergence of adam under relaxed assumptions. arXiv preprint arXiv:2304.13972 , abs/2304.13972, 2023. URL https://arXiv.org/abs/2304. 13972 . (Cited on page 2)

Xiaoyu Li and Francesco Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. In Kamalika Chaudhuri and Masashi Sugiyama, editors, The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan , volume 89 of Proceedings of Machine Learning Research , pages 983-992. PMLR, 2019. URL http://proceedings.mlr.press/v89/li19c.html . (Cited on page 4)

Zijian Liu, Ta Duy Nguyen, Alina Ene, and Huy L. Nguyen. On the convergence of AdaGrad(norm) on R d : Beyond convexity, non-asymptotic rate and acceleration. arXiv preprint arXiv:2209.14827 , abs/2209.14827, 2022. URL https://arXiv.org/abs/2209.14827 . (Cited on page 4)

Nicolas Loizou, Sharan Vaswani, Issam Hadj Laradji, and Simon Lacoste-Julien. Stochastic polyak step-size for SGD: an adaptive learning rate for fast convergence. In Arindam Banerjee and Kenji Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event , volume 130 of Proceedings of Machine Learning Research , pages 1306-1314. PMLR, 2021. URL http://proceedings.mlr.press/v130/ loizou21a.html . (Cited on page 3)

Yura Malitsky and Konstantin Mishchenko. Adaptive gradient descent without descent. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 6702-6712. PMLR, 2020. URL http://proceedings.mlr.press/v119/malitsky20a.html . (Cited on pages 3 and 4)

Brendan McMahan and Francesco Orabona. Unconstrained online linear learning in hilbert spaces: Minimax algorithms and normal approximations. In Maria-Florina Balcan, Vitaly Feldman, and Csaba Szepesvári, editors, Proceedings of The 27th Conference on Learning Theory, COLT 2014, Barcelona, Spain, June 13-15, 2014 , volume 35 of JMLR Workshop and Conference Proceedings , pages 1020-1039. JMLR.org, 2014. URL http://proceedings.mlr.press/v35/mcmahan14. html . (Cited on page 3)

Zakaria Mhammedi and Wouter M Koolen. Lipschitz and comparator-norm adaptivity in online learning. In Conference on Learning Theory , pages 2858-2887. PMLR, 2020. (Cited on page 3)

Zakaria Mhammedi, Wouter M Koolen, and Tim Van Erven. Lipschitz adaptivity with multiple learning rates in online learning. In Conference on Learning Theory , pages 2490-2511. PMLR, 2019. (Cited on page 3)

Ryan Murray, Brian Swenson, and Soummya Kar. Revisiting normalized gradient descent: Fast evasion of saddle points. IEEE Transactions on Automatic Control , 64(11):4818-4824, 2019. doi: 10.1109/TAC.2019.2914998. (Cited on page 5)

Yurii Nesterov. Universal gradient methods for convex optimization problems. Mathematical Programming , 152(1-2):381-404, 2014. doi: 10.1007/s10107-014-0790-0. URL https://doi. org/10.1007/s10107-014-0790-0 . (Cited on pages 1, 2, and 4)

Yurii Nesterov. Lectures on Convex Optimization . Springer Publishing Company, Incorporated, 2nd edition, 2018. ISBN 3319915770. (Cited on pages 4 and 5)

Francesco Orabona. Dimension-free exponentiated gradient. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper_ files/paper/2013/file/7634ea65a4e6d9041cfd3f7de18e334a-Paper.pdf . (Cited on page 3)

Francesco Orabona. Normalized gradients for all. arXiv preprint arXiv:2308.05621 , abs/2308.05621, 2023. URL https://arXiv.org/abs/2308.05621 . (Cited on pages 2 and 3)

Francesco Orabona and Ashok Cutkosky. ICML 2020 tutorial on parameter-free online optimization. ICML Tutorials , 2020. URL https://parameterfree.com/icml-tutorial/ . (Cited on pages 1, 2, 3, 15, and 18)

Francesco Orabona and Dávid Pál. Coin betting and parameter-free online learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems , NIPS'16, page 577-585, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819. (Cited on pages 2 and 3)

Francesco Orabona and Dávid Pál. Parameter-free stochastic optimization of variationally coherent functions. arXiv preprint arXiv:2102.00236 , abs/2102.00236, 2021. URL https://arXiv.org/ abs/2102.00236 . (Cited on page 3)

Francesco Orabona and Tatiana Tommasi. Training deep networks without learning rates through coin betting. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 2160-2170, 2017. URL https://proceedings.neurips. cc/paper/2017/hash/7c82fab8c8f89124e2ce92984e04fb40-Abstract.html . (Cited on page 3)

Antonio Orvieto, Simon Lacoste-Julien, and Nicolas Loizou. Dynamics of sgd with stochastic polyak stepsizes: Truly adaptive variants and convergence to exact solution. arXiv preprint arXiv:2205.04583 , abs/2205.04583, 2022. URL https://arXiv.org/abs/2205.04583 . (Cited on page 3)

Yan Pan and Yuanzhi Li. Toward understanding why Adam converges faster than SGD for transformers. OPT2023: 14th Annual Workshop on Optimization for Machine Learning , 2022. URL https://openreview.net/pdf?id=Sf1NlV2r6PO . (Cited on page 9)

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. pdf . (Cited on page 9)

David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350 , abs/2104.10350, 2021. URL https://arXiv.org/abs/2104. 10350 . (Cited on page 1)

Boris Polyak. Introduction to optimization . Optimization Software, 1987. (Cited on pages 2 and 3)

Boris T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics , 4(5):1-17, 1964. ISSN 0041-5553. doi: https://doi.org/10.1016/0041-5553(64)90137-5. URL https://www.sciencedirect.com/ science/article/pii/0041555364901375 . (Cited on page 1)

Or Sharir, Barak Peleg, and Yoav Shoham. The cost of training NLP models: a concise overview. arXiv preprint arXiv:2004.08900 , abs/2004.08900, 2020. URL https://arXiv.org/abs/2004. 08900 . (Cited on page 1)

Naum Zuselevich Shor. Minimization methods for non-differentiable functions , volume 3. Springer Science & Business Media, 2012. (Cited on page 5)

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1409.1556 . (Cited on page 9)

Matthew Streeter and H. Brendan McMahan. No-regret algorithms for unconstrained online convex optimization. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 2 , NIPS'12, page 2402-2410, Red Hook, NY, USA, 2012. Curran Associates Inc. (Cited on page 3)

Cheik Traoré and Edouard Pauwels. Sequential convergence of AdaGrad algorithm for smooth convex optimization. Operations Research Letters , 49(4):452-458, 2021. (Cited on page 4)

Rachel Ward, Xiaoxia Wu, and Léon Bottou. AdaGrad stepsizes: sharp convergence over nonconvex landscapes. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 6677-6686. PMLR, 2019. URL http://proceedings.mlr.press/v97/ward19a.html . (Cited on page 4)

Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 4148-4158, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ 81b3833e2504647f9d794f7d7b9bf341-Abstract.html . (Cited on page 1)

Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020a. URL https://openreview.net/forum?id=BJgnXpVYwS . (Cited on page 4)

Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank J. Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and HsuanTien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 612, 2020, virtual , 2020b. URL https://proceedings.neurips.cc/paper/2020/hash/ b05b57f6add810d3b7490866d74c0053-Abstract.html . (Cited on page 4)

Supplementary material

Contents

5 Algorithm-independent results

In this section we collect different results that are algorithm-independent, the first is a consequence of smoothness:

Fact 1. Suppose that f is smooth and lower bounded by f ∗ . Then for all x ∈ R d we have,

Proof. This is a common result in the literature, and finds applications in convex and non-convex optimization see e.g. (Levy, 2017; Levy et al., 2018; Orabona and Cutkosky, 2020; Khaled and Richtárik, 2020). We include the proof for completeness. Let x ∈ R d and define x + = x -1 L ∇ f ( x ) . Then by smoothness

Because f is lower bounded by f ∗ we thus have

Rearranging gives ∥∇ f ( x ) ∥ 2 ≤ 2 L ( f ( x ) -f )

The next two results are helpful algebraic identities that will be useful for the proof of DoWG.

Lemma 1. (Ivgi et al., 2023, Lemma 4). Let a 0 , .., a t be a nondecreasing sequence of nonnegative numbers. Then

Proof. This is (Ivgi et al., 2023, Lemma 4). We include the proof for completeness:

Lemma 2. ((Ivgi et al., 2023, Lemma 3), similar to (Defazio and Mishchenko, 2023, Lemma 11)). Let s 0 , s 1 , . . . , s T be a positive increasing sequence. Then

where log + x def = log x +1 .

Proof. This is (Ivgi et al., 2023, Lemma 3). We include the proof for completeness: Define K = ⌈ log s T s 0 ⌉ and n = ⌊ T K ⌋ . Then,

Rearranging and using K = ⌈ log s T s 0 ⌉ gives

Therefore,

Thus,

6 Proofs for DoWG

This section collects proofs for DoWG. First, we give the following lemma, which holds under convexity alone (regardless of whether f is smooth or Lipschitz).

Lemma 3. Suppose that f is convex and has minimizer x ∗ . For the iterations generated by Algorithm 1, we have

where ¯ d t = max k ≤ t d k .

Proof. This proof follows the proof of DoG (Ivgi et al., 2023, Lemma 1), itself a modification of the standard proof for adaptive cumulative gradient normalization methods (Gupta et al., 2017) incorporating insights from (Carmon and Hinder, 2022). We specifically modify the proof to handle the weighting scheme we use in DoWG. By the nonexpansivity of the projection we have

Rearranging and dividing by 2 η k we get

Multiplying both sides by ¯ r 2 k we get

Summing up as k varies from 0 to t -1 we get

We shall now bound each of the terms (A) and (B). We have

where eq. (8) holds by definition of the DoWG stepsize η k , eq. (9) holds by telescoping, eq. (10) holds because v k = v k -1 + ¯ r 2 k ∥∇ f ( x k ) ∥ 2 ≥ v k -1 and hence √ v k ≥ √ v k -1 , and d 2 k ≤ ¯ d 2 t by definition. Equation (11) just follows by telescoping. Finally observe that ¯ d 2 t -d 2 t = d 2 s -d 2 t for some s ∈ [ t ] , and d 2 s -d 2 t = ( d s -d t )( d s + d t ) . Then by the triangle inequality and that the sequence

¯ r k is monotonically nondecreasing we have

Therefore d 2 s -d 2 t ≤ (¯ r s + ¯ r t )( d s + d t ) ≤ 4¯ r t ¯ d t . This explains eq. (12). For the second term in eq. (7), we have

where eq. (13) is by Lemma 1. Plugging eqs. (12) and (14) in eq. (7) gives

6.1 Smooth case

We now prove the convergence of DoWG under smoothness. In particular, we shall use Fact 1 and the DoWG design to bound the weighted cumulative error S t = ∑ t -1 k =0 r 2 k [ f ( x k ) -f ∗ ] by its square root multiplied by a problem-dependent constant. We note that a similar trick is used in the analysis of AdaGrad-Norm (Levy et al., 2018), in reductions from online convex optimization to stochastic smooth optimization (Orabona and Cutkosky, 2020), and in the method of (Carmon and Hinder, 2022). However, in all the mentioned cases, the unweighted error M t = ∑ t -1 k =0 [ f ( x k ) -f ∗ ] is bounded by its square root. Here, DoWG's design allows us to bound the weighted errors S t instead.

Proof of Theorem 4. We start with Lemma 3. Let t ∈ [ T ] . By eq. (6) we have

Observe that by convexity we have

Using this to lower bound the left-hand side of eq. (15) gives

We have by smoothness that ∥∇ f ( x ) ∥ 2 ≤ 2 L ( f ( x ) -f ∗ ) for all x ∈ R d , therefore

Taking square roots we get

Squaring both sides gives

By convexity we have

Using eq. (17) in eq. (16) gives

If f ( x k ) -f ∗ = 0 for some k ∈ [ t -1] then the statement of the theorem is trivial. Otherwise, we can divide both sides by the latter square root to get

Dividing both sides by ∑ t -1 k =0 ¯ r 2 k we get

By Lemma 2 applied to the sequence s k = ¯ r 2 k we have that for some t ∈ [ T ]

Because X has diameter D we have ¯ r 2 T ≤ D 2 and therefore

We now have two cases:

If T ≥ 2 log + D 2 ¯ r 2 0 then T log D 2 ¯ r 2 0 -1 ≥ T 2 log D 2 ¯ r 2 0 and we use this in eqs. (18) and (19) to get

Observe that because X has diameter at most D we have ¯ d t + ¯ r t ≤ 2 D , therefore

· If T < 2 log + D 2 ¯ r 2 0 , then 1 < 2 log + D 2 ¯ r 2 0 T . Let t ∈ [ T ] . Using smoothness and this fact we have

Observe ¯ x t , x ∗ ∈ X and X has diameter D , hence ∥ ¯ x t -x ∗ ∥ 2 ≤ D 2 and we get

Thus in both cases we have that f (¯ x t ) -f ∗ = O ( LD 2 T log + D ¯ r 0 ) , this completes our proof.

6.2 Nonsmooth case

We now give the proof of DoWG's convergence when f is Lipschitz.

Proof of Theorem 3. We start with Lemma 3. Let t ∈ [ T ] . By eq. (6) we have

Observe that by convexity we have

Using this to lower bound the left-hand side of eq. (20) gives

We have by the fact that f is G -Lipschitz that ∥∇ f ( x ) ∥ 2 ≤ G 2 for all x ∈ X . Therefore,

Taking square roots and plugging into eq. (21) gives

Dividing both sides by ∑ t -1 k =0 ¯ r 2 k we get

By Lemma 2 applied to the sequence s k = ¯ r 2 k we have that for some t ∈ [ T ]

Because ¯ r T ≤ D we further have

We now have two cases:

If T ≥ 2 log + D 2 ¯ r 2 0 : then T log ¯ r 2 T ¯ r 2 0 -1 ≥ T 2 log ¯ r 2 T ¯ r 2 0 . We can use this in eq. (22) alongside eq. (23) and

the fact that log + x 2 = max(log x 2 , 1) = max(2 log x, 1) ≤ 2 log + x to get

Because the diameter of X is bounded by D we have ¯ r T ≤ D and ¯ d t ≤ D , using this and convexity we get

By convexity and Cauchy-Schwartz we have

Because f is G -Lipschitz then ∥∇ f (¯ x t ) ∥ ≤ G and because X has diameter D we have ∥ ¯ x t -x ∗ ∥ ≤ D . Using this and eq. (24) in eq. (25) gives

Now because T ≥ 1 we have √ T ≤ T and hence

In both cases, we have that f (¯ x t ) -f ∗ = O ( 4 DG log + D ¯ r 0 √ T ) , and this completes our proof.

If T < 2 log + D 2 ¯ r 2 0 : then

7 Unconstrained domain extension

In this section we consider the case where the domain set is unbounded, and we seek dependence only on d 0 = ∥ x 0 -x ∗ ∥ . We use the same technique for handling the unconstrained problem as (Ivgi et al., 2023) in this section and consider DoWG iterates with the reduced stepsizes

We prove that with this stepsize, the iterates do not venture far from the initialization. The proof follows (Ivgi et al., 2023).

Lemma 4. (Stability). For the iterates x t +1 = x t -η t ∇ f ( x t ) following the stepsize scheme given by (26) we have ¯ d 2 t ≤ 12 d 0 and ¯ r 2 t ≤ 32 d 2 0 provided that r 0 ≤ d 0 .

Proof. By expanding the square and convexity

Summing up as t varies from k = 1 to k = t and using (Ivgi et al., 2023, Lemma 6)

Therefore we have d 2 t ≤ d 2 1 + ¯ r 2 t 4 . Now suppose for the sake of induction that ¯ r 2 t ≤ 8 d 2 1 , then applying the last equation we get d 2 t ≤ 3 d 2 1 . Taking square roots gives d t ≤ √ 3 d 1 . By the triangle inequality we then get

Squaring both sides gives ∥ x t +1 -x 0 ∥ 2 ≤ (1 + √ 3) 2 d 2 1 ≤ 8 d 2 1 . This completes our induction and we have ¯ r 2 t ≤ 8 d 2 1 for all t . Finally, observe that

It follows that ¯ r 2 t ≤ 8 d 2 1 ≤ 32 d 2 0 for all t . Finally, we have d 2 t ≤ d 2 1 + ¯ r 2 t 4 ≤ 3 d 2 1 ≤ 12 d 2 0 . This completes our proof.

Therefore the iterates stay bounded. The rest of the proof then follows Theorems 3 and 4 and is omitted for simplicity.. In both cases it gives the same results with D 0 = ∥ x 0 -x ∗ ∥ instead of D , up to extra constants and polylogarithmic factors.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is concerned with parameter free - adaptive first order optimisation method, that is a method that achieves optimal rates of convergence in the class of function considered without having access to a priori quantities such as smoothness of the function or the minimum value of the function.\nThe authors discuss on normalized gradient descent and they show that this method is adaptive to the smoothness of the function when the step-size is proportional to the distance to the minimizer. \nThe main contribution of the paper is to introduce a new estimation of the distance to the minimizer that achieves optimal rate up to logarithm factor. The method is an improvement over the so-called distance over gradients (DoW) by introducing weights (DoWG) so that it gives more weight to the last gradients. \nNumerical experiments are provided, including comparison to Adam (which is not in the same category...).\n\nSTRENGTHS:\nI am not a specialist of the field so my contribution in this review was to check the proofs, also in the supplementary material.\n\nAll the proofs are correct. \n\nThe paper is globally nicely written, easy to read and accessible for an outsider of the field as I am. \n\nAlthough from the theoretical point of view, DoWG is incremental (correct me if I am wrong), numerical experiments tend to show that it is clearly more efficient practically.\n\nAlthough the resulting algorithm performs poorly compared to Adam, it seems a fair contribution on the understanding and the technics in the class of methods considered.\n\nWEAKNESSES:\nThis work appears as an improvement over DoG. It is a bit surprising that the result on Normalized Gradient Descent is new; I was not able to find it in the literature although, again, I am not an expert in the field. \nThe proposed method DoWG of using weights in front of the gradients is a natural idea. \nFrom the theoretical results, DoWG seems a rather incremental advance over DoG.\n\nThe resulting algorithm is probably not going to have any impact on the optimization method in deep learning.\n\nQUESTIONS:\nCould the rate obtained in the analysis of NGD be improved?\n\n- line 131: gives eta so that \n- line 554: Therefore.\n\nLIMITATIONS:\ndo no apply.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a modification to the recently proposed DoG algorithm to obtain a adaptive algorithm for the deterministic, convex and [Lipschitz or Smooth] setting that does not need any hyperparameter to achieve a convergence rate competitive with algorithms that know the problem-specific constants.\n\nThe submission discusses the benefits of normalization, linking its behavior to the edge of stability phenomenon and providing an analysis that normalized gradient descent obtains the convergence of GD on smooth function with step-size 1/L without needing to know the smoothness constant.\n\nSTRENGTHS:\nThe submission is well presented and I could understand the main objective of the paper at first read. The DoWG algorithm, and especially its proof, is simpler than the one found the prior work of Ivgi et al. (2023) on DoG, and would be an easier starting point for readers trying to understand algorithms that can adapt to the initial distance to the optimum.\n\nWEAKNESSES:\nBeyond the DoWG algorithm, the contributions are also already known in the literature. To the credit of the submission, this is clearly mentioned for the observation that normalized gradient descent on smooth functions leads to a similar behavior to the edge of stability phenomenon (Arora et al., 2022). However, the result that normalized gradient descent obtains the smooth rate can be found in the work of Levy (2017), which seems to have been missed. \n\nThe submission also suffers from a lack of formalism on some of its claims. Some concepts such as weak and strong adaptivity, “adapting to geometry” and what is meant by the edge of stability are left vague, and would benefit from a formal definition and nuanced discussion. \n\nIf those issues, detailed in the question section, are addressed in a revision, I will increase my score to 6.\n\nQUESTIONS:\n**Prior work**\n\nI found the discussion of the adaptivity to smoothness of normalized GD and the key inequalities (5—7) useful to understand how DoWG achieves a similar result. However, the paragraph prior to Theorem 4 should cite Levy (2017). Their analysis of what they call the AdaNGDk algorithm (an overly general form that recovers normalized gradient descent with a decreasing step-size of the order $1/\\sqrt{t}$\nfor k=1), in Theorem 2.1 is the equivalent of theorem 4 in the current submission, except with a decreasing step-size of $1/\\sqrt{t}$ rather than the constant but horizon-dependent step-size $1/\\sqrt{T}$ used here.\n\nKfir Levy, “Online to Offline Conversions, Universality and Adaptive Minibatch Sizes”, NeurIPS 2017 \n\n---\n\n**Formal definition for some concepts**\n\n**Strong vs. Weak Adaptivity**\n\nThe working definitions of “weak” and “strong” adaptivity are not clear from the current writing, or seems to lead to undesirable conclusions. I find the separation useful, and being able to formalize how sub-optimal the “adaptivity” of AdaGrad-Norm is in the smooth setting would be helpful. \n\nHowever, the definition of weakly adaptive requires that “[weak-adaptivity] just seeks non-divergence given stepsize misspecification. […] the algorithm’s objective is to ensure that the learning process does not result in divergence, even if the chosen stepsize is not optimal” (L145). This definition is problematic because an algorithm that does not move, or gradient descent with a step-size of 0, satisfies the above definition. \n\nSimilarly, for strongly adaptive, the text requires that “an algorithm [is strongly adaptive if] it preserves the convergence rate of optimally tuned gradient descent without any manual tuning.” The proposed DoWG algorithm does not fit this description as the rate worse, if only by a log-factor. \n\nA formalization of the above that could work would be to say that an algorithm is strongly adaptive in a problem setting if it achieves the same convergence rate, up to polylogarithmic terms, as an algorithm that knows problem specific constants. Weakly-adaptive could similarly be defined by allowing for multiplicative polynomial factors. \n\n**Adapting to geometry**\n\nThe term “adapting to geometry” is used in multiple places, but it isn’t clear what is meant by that statement. This term is often used in the literature as a way to convey the intuition of why a method is good, but with limited formalism. For example, Newton adapts to the geometry of the problem by using the Hessian, or AdaGrad adapts to the geometry of the problem by finding a preconditioner that is competitive with the optimal one in hindsight. On each usage of “adapting to geometry”, I do not understand what is meant,. I strongly suggest avoiding it and using a more direct description instead.\n\n- L32: “We say an algorithm is universal if it adapts to many different problem geometries or regularity conditions on the function f”\n(should be removed and only mention regularity condition)\n- L56: “In particular, such adaptive methods empirically show the ability to adapt to the local geometry of the problem (Cohen et al., 2022), especially compared to plain (stochastic) gradient descent (Pan and Li, 2022).”\nL107: “There are other justifications for why adaptive methods work outside of adapting to geometry.”\n- L246 “Therefore, we may may expect this to aid the method in adapting to the geometry of the problem once far away from the initialization x0.”\n\n---\n\n**A mention of the difficulties of obtaining similar results in online learning would be helpful to the reader.** \n\nWhile the online-learning algorithms of Streeter and McMahan (2012), Orabona et Pal (2016) and Orabona and Cutkosky (2020) aim for a similar goal of not having to know the diameter of the set and the Lipschitz constant, the rates presented for the current algorithm or those in the prior works of of Ivgi et al. (2023) and Carmon and Hinder (2022) are not achievable in the adversarial that is common in online learning, see Cutkosky and Boahen (2016, 2017).\n\nAshok Cutkosky, Kwabena Boahen, “Online Convex Optimization with Unconstrained Domains and Losses”, NeurIPS 2016\nAshok Cutkosky, Kwabena Boahen, “Online Learning Without Prior Information”, COLT 2017\n\n---\n\n**Question on Universality and the edge of stability**\n\nThe convergence of normalized GD or DoWG are not discussed under strong-convexity, which I think should be mentioned given the focus on universal algorithms. Especially as some alternative algorithms such as the Polyak step-size do benefit from strong-convexity, whether in the smooth or Lipschitz+bounded set case.\n\nGiven that the empirical results shown in Figure 2 are on a strongly convex problem and the effective step-size oscillates around a constant 2/L, I am interpreting it as the algorithm *not* achieving a linear rate (as the gradient norm should also go down as a linear rate, which does not seem to be the case if $\\eta/\\Vert\\nabla f(x_t)\\Vert$ is constant)?\n\n\n---\n\n**typo?**\n- L80: \"Orvieto et al. (2022) show that a variant of the Polyak stepsize with decreasing stepsizes can recover the convergence rate of gradient descent in the deterministic setting, provided the stepsize is initialized properly\" -- this should be stochastic?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of optimizing a convex function over a convex, closed, and possibly compact set $\\mathcal{X}$. In particular, they are interested in finding a first-order method which is (i) universal (i.e., the same algorithm can be used when the objective is Lipschitz or smooth), (ii) parameter-free (i.e., converges without any hyperparameter tuning w.r.t. problem parameters) (iii) has no search subroutines, and (iv) strongly-adaptive (i.e., preserves the convergence rate of an optimally-tuned GD without tuning). The authors propose an algorithm, Distance-over-Weighted-Gradients, which (essentially) achieves all of these desired properties. The two caveats are (i) the convergence rates have an extra log factor not present in optimally-tuned GD, and (ii) the convergence rates depend on the diameter of the constraint set $\\mathcal{X}$ instead of on $D_0 = || x_0 - x^* ||$. Towards establishing this result, the authors prove that normalized GD universal but not parameter-free.\n\nSTRENGTHS:\nThis paper provides an algorithm with the remarkable properties of being universal, parameter-free, (nearly) strongly-adaptive, and not requiring a search subroutine. Unless I have misunderstood, this seems to be the first such algorithm satisfying all of these properties simultaneously. Moreover, the analysis is well-written and easy to follow. I also found the discussion of the universal properties of normalized GD to be quite instructive and interesting. Overall, I think this is a very nice paper.\n\nWEAKNESSES:\nMuch of the analysis appears to rely on or extend arguments from Ivgi et al., (2023). Further, the results in this paper are restricted to the deterministic setting, while the results of Ivgi et al. hold also in the noisy setting (albeit, assuming uniformly bounded stochastic gradients, and thus the results are restricted to the Lipschitz setting). Despite this, I still think the results in this paper are quite nice.\n\nQUESTIONS:\nThe results of Ivgi et al. (2023) hold also in the stochastic setting (assuming stochastic gradients are uniformly-bounded). What are the main barriers to extending your results to the stochastic setting? Is there any similar result that you can obtain when, e.g., the noise of stochastic gradients is uniformly bounded? It might be beneficial to the paper to add a discussion on why extending to this setting is difficult.\n\nIn the introduction of your paper, you mention that the constraint set $\\mathcal{X}$ is “(potentially) compact“, However, your main results (Theorems 5 and 6) assume that the diameter $D$ of $\\mathcal{X}$ is finite (but unknown). Is it possible to extend these results to the unconstrainted case where $\\mathcal{X} = \\mathbb{R}^d$?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new algorithm for optimization problems. The paper introduces DoWG (Distance over Weighted Gradient) which is a simple extension of a previous work DOG (Distance over gradient). It is a parameter-free gradient optimization method where the step size is automatically adjusted to the function being optimized over the course of the optimization.\n\nThis works also provides justification of why DoWG produces close to the optimal convergence that can be achieved by Normalized Gradient Descent (NGD) for either Lipschitz or smooth functions. As part of this claim the authors also show a new derivation of the optimal convergence for NGD on smooth functions.\n\nFinally the authors show results on training Deep Nets for vision and these demonstrate the while DoWG performs better than other parameter-free methods it doesn't attain the level of accuracy achieved by momentum-based methods such as ADAM.\n\nSTRENGTHS:\n## Defining Universality\n\nThe authors have very systematically defined their concept of a Universal Gradient Descent-based optimization algorithm. In fact as part of this definition of Universality they have computed the optimal convergence rate of NGD for smooth functions which appears to be a new result although it is not a very surprising result.\n\n## Discussions of NGD\n\nThe paper provides some very strong intuition about how NGD is self-stabilizing. The discussion and example in the paper provides good insight about how parameter-free methods work in general.\n\nWEAKNESSES:\n## Comparison to DOG\n\nThe proposed algorithm itself is only slightly different than the existing DOG algorithm and the intuition of why it was proposed is not that clear. The paper mentions that DoWG gives higher weight to later gradients, but why is this important? They do also demonstrate on line 263 that DoWG step sizes could be larger than DOG's and this seems reasonable, but overall the motivation for the improvement is not clear.\n\nNow the paper does show that DoWG has optimal NGD convergence for Lipschitz functions, but it points out that the same holds for DOG. There is no discussion of whether optimal NGD convergence that is proven for smooth functions with DoWG also holds for DOG or not. If the authors could prove that DOG *doesn't* have optimal NGD convergence for smooth functions then their claim that DoWG is the first parameter-free Universal gradient descent algorithm without a search subroutine would be overall stronger.\n\n## Performance Relative to ADAM\n\nThe authors should certainly be lauded for including a superior baseline result. However, the fact that ADAM does so much better than DoWG makes their work more of a theoretical curiosity.\n\nIt appears that the authors hurried through the final evaluation and included very limited results. The previous work that introduced DOG had results showing both fine-tuning as well as training a model from scratch. I would encourage the authors of the current work to do something similar as well as introduce variants such as L-DOWG.\n\nQUESTIONS:\n- Does DOG have optimal NGD convergence for smooth functions? Have the authors found a reason why this is not the case or why this would be unlikely and could provide a justification?\n\n- Have the authors considered L-DOWG (layer-wise DOWG similar to L-DOG)?\n\n- The DOG paper (https://arxiv.org/pdf/2302.12022.pdf) in Table 2 page 14 show results for DOG that are superior than ADAM for training a model from scratch. Why could the authors not reproduce those results as a baseline?\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new algorithm, DoWG, which modifies DoG (Ivgi et al., 2023) by utilizing the weighted sum instead of the usual sum for the squared gradients.\nConvergence is established for convex and convex & smooth deterministic settings, under the assumption of a (possibly unknown) bounded domain.\nThe method is adaptive to the Lipschitz/smoothness constant and is parameter-free in the sense that the bound of the domain may be unknown.\nExperiments comparing DoWG to several adaptive methods are presented.\nAdditionally, a complementary result for convex & smooth NGD establish that NGD is weakly adaptive to the diameter of the problem.\n\nSTRENGTHS:\n1. The modification from DoG to DoWG is appealing as it results in larger stepsizes, potentially leading to faster convergence.\n2. A (limited) set of experiments is provided, demonstrating positive empirical evidence for the effectiveness of the DoWG method.\n\nWEAKNESSES:\n3. It is important to highlight that the convergence of DoWG is established exclusively in the deterministic setting, making it weaker than DoG's convergence, which extends to both deterministic and stochastic settings (note that AdaGrad-Norm results also apply to the stochastic setting). The paper should explicitly emphasize this difference and consider exploring DoWG's performance in the stochastic setting.\n4. The theoretical results for DoWG hold significance when dealing with unknown domain bounds. When the domain bound is known, AdaGrad-Norm with stepsizes $\\eta_t=D/\\sqrt{\\sum_t \\lVert \\nabla f(w_t) \\rVert^2}$ has tighter bounds for both convex and convex & smooth settings.\n5. The term \"parameter-free method\" typically used for rates that depends on the distance to the comparator or the minimizer. In this context the convergence of DoWG is not truly parameter-free as it is relevant only for bounded domains and does not improve with a good initialization.\n\nQUESTIONS:\n6. Can DoWG achieve similar convergence results to DoG in the stochastic setting? If it is yet established, what is the additional difficulty with respect to DoG?\n7. In the deterministic setting, would DoG achieve similar convergence results to DoGW, or was the modification to DoWG necessary to address specific issues in that setting?\n8. What is the motivation for the problem of deterministic optimization with unknown bounded domain? Corollary 1 of Ivgi et al. (2023) mentions two-stage stochastic programming as an application, but is the deterministic counterpart of interest as well?\n\nOverall, while DoWG shows promise and may yield novel convergence properties, I find the current established guarantees to be partly insufficient compared to previous work on adaptive and parameter-free methods.\n\nEdit: Per the authors response regarding the unbounded case with adaptivity to $D_0$ and the improved log factor with respect to the smooth result of DoG (unknown at the time of submission), i raise my score.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}

comments: SUMMARY:
The paper proposes a new optimization algorithm, DOWG, that does not rely on additional hyperparameter tuning or a line search subroutine. Interestingly, the paper also contains a proof and analysis regarding the behavior of NGD and shows that (1) NGD adapts to the smoothness of continuous loss surfaces and (2) NGD operates at the edge of the maximum stable learning rate for continuous loss functions. Finally, the paper presents two empirical studies on CIFAR10 with two different neural networks.

STRENGTHS:
I am not an expert in optimization, but I thoroughly enjoyed reading this paper. The writing is clearly organized and accessible to readers with only some rudimentary knowledge of optimization. 

Regarding related work, the paper carefully assigns credit to existing work and precisely describes how the current work stands out at each step along the way, which demonstrates a high level of expertise in the field. 

The theoretical contribution seems solid, and the new algorithm seems to be well motivated and work well, when compared with other parameter-free algorithms. I strongly recommend this paper be accepted.

WEAKNESSES:
DOWG does not work as well as Adam with cosine annealing on both neural network problems. However, this is not very surprising because cosine annealing is extremely strong. I do not consider this to be a fatal issue.

QUESTIONS:
Nil

LIMITATIONS:
Nil
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-23