SlowGuess/NeurIPS_2023_OCR huggingface dataset:
DoWG uses the same idea of estimating the distance from the optimum by using the distance from the initial point as a surrogate, but instead of using the square root of the running gradient sum  $G_{t} = \sum_{k=0}^{t} \|\nabla f(x_{k})\|^{2}$  as the normalization, DoWG uses the square root of the weighted gradient sum  $v_{t} = \sum_{k=0}^{t} \bar{r}_{k}^{2} \|\nabla f(x_{k})\|^{2}$ . Observe that because the estimated distances  $\bar{r}_{t}$  are monotonically increasing, later gradients have a larger impact on  $v_{t}$  than earlier ones compared to  $G_{t}$ . Therefore, we may expect this to aid the method in adapting to the local properties of the problem once far away from the initialization  $x_{0}$ . We note that using a weighted sum of gradients is not new: AcceleGrad (Levy et al., 2018) uses time-varying polynomial weights and Adam (Kingma and Ba, 2015) uses exponentially decreasing weights. The difference is that DoWG chooses the weights adaptively based on the running distance from the initial point. This use of distance-based weighted averaging is new, and we are not aware of any previous methods that estimate the running gradient sum in this manner.
Nonsmooth analysis. The next theorem shows that DoWG adapts to the Lipschitz constant  $G$  and the diameter  $D$  of the set  $\mathcal{X}$  if the function  $f$  is nonsmooth but  $G$ -Lipschitz. We use the notation  $\log_+ x = \log x + 1$  following (Ivgi et al., 2023).
Theorem 3. (DoWG, Lipschitz  $f$ ). Suppose that the function  $f$  is convex,  $G$ -Lipschitz, and has a minimizer  $x_{*} \in \mathcal{X}$ . Suppose that the domain  $\mathcal{X}$  is a closed convex set of (unknown) diameter  $D > 0$ . Let  $r_{\epsilon} < D$ . Then the output of Algorithm 1 satisfies for some  $t \in \{0, 1, \dots, T - 1\}$
$$
f (\bar {x} _ {t}) - f _ {*} = \mathcal {O} \left[ \frac {G D}{\sqrt {T}} \log_ {+} \frac {D}{r _ {\epsilon}} \right],
$$
where  $\bar{x}_t \stackrel{\text{def}}{=} \frac{1}{\sum_{k=0}^{t-1} \overline{r}_k^2} \sum_{k=0}^{t-1} \overline{r}_k^2 x_k$  is a weighted average of the iterates returned by the algorithm.
Discussion of convergence rate. DoWG matches the optimal  $\mathcal{O}\left(\frac{DG}{\sqrt{T}}\right)$  rate of tuned GD and tuned NGD up to an extra logarithmic factor. We note that the recently proposed algorithms DoG (Ivgi et al., 2023) and D-Adaptation (Defazio and Mishchenko, 2023) achieve a similar rate in this setting.
Comparison with DoG. As we discussed before, DoWG uses an adaptively weighted sum of gradients for normalization compared to the simple sum used by DoG. In addition, DoG uses the stepsize  $\frac{\bar{r}_t}{\sqrt{\sum_{k=0}^t \|\nabla f(x_k)\|^2}$ , whereas the DoWG stepsize is pointwise larger: since  $\bar{r}_k^2$  is monotonically increasing in  $k$  we have
$$
\eta_ {\mathrm {D o W G}, t} = \frac {\bar {r} _ {t} ^ {2}}{\sqrt {\sum_ {k = 0} ^ {t} \bar {r} _ {k} ^ {2} \| \nabla f (x _ {k}) \| ^ {2}}} \geq \frac {\bar {r} _ {t} ^ {2}}{\bar {r} _ {t} \sqrt {\sum_ {k = 0} ^ {t} \| \nabla f (x _ {k}) \| ^ {2}}} = \frac {\bar {r} _ {t}}{\sqrt {\sum_ {k = 0} ^ {t} \| \nabla f (x _ {k}) \| ^ {2}}} = \eta_ {\mathrm {D o G}, t}.
$$
Of course, the pointwise comparison may not reflect the practical performance of the algorithms, since after the first iteration the sequence of iterates  $x_{2}, x_{3}, \ldots$  generated by the two algorithms can be very different. We observe in practice that DoWG is in general more aggressive, and uses larger stepsizes than both DoG and D-Adaptation (see Section 4).
Smooth analysis. Our next theorem shows that DoWG adapts to the smoothness constant and the diameter  $D$  of the set  $\mathcal{X}$ .
Theorem 4. (DoWG, Smooth  $f$ ). Suppose that the function  $f$  is  $L$ -smooth, convex, and has a minimizer  $x_{*} \in \mathcal{X}$ . Suppose that the domain  $\mathcal{X}$  is a closed convex set of diameter  $D > 0$ . Let  $r_{\epsilon} < D$ . Then the output of Algorithm 1 satisfies for some  $t \in \{0, 1, \dots, T - 1\}$
$$
f (\bar {x} _ {t}) - f _ {*} = \mathcal {O} \left[ \frac {L D ^ {2}}{T} \log_ {+} \frac {D}{r _ {\epsilon}} \right],
$$
The proof of this theorem and all subsequent results is relegated to the supplementary material. We note that the proof of Theorem 4 uses the same trick used to show the adaptivity of NGD to smoothness: we use the fact that  $\| \nabla f(x)\| \leq \sqrt{2L(f(x) - f_{*})}$  for all  $x\in \mathcal{X}$  applied to a carefully-chosen weighted sum of gradients.
Comparison with GD/NGD. Both well-tuned gradient descent and normalized gradient descent achieve the convergence rate  $\mathcal{O}\left(\frac{LD_0^2}{T}\right)$  where  $D_0 = \| x_0 - x_*\| \leq D$  for the constrained convex minimization problem. Theorem 4 shows that DoWG essentially attains the same rate up to the difference between  $D_0$  and  $D$  and an extra logarithmic factor. In the worst case, if we initialize far from the optimum, we have  $D_0\simeq D$  and hence the difference is not significant. We note that DoG (Ivgi et al., 2023) suffers from a similar dependence on the diameter  $D$  of  $\mathcal{X}$ , and can diverge in the unconstrained setting, where  $\mathcal{X}$  is not compact. This can be alleviated by making the stepsize smaller by a polylogarithmic factor. A similar reduction of the stepsize also works for DoWG, and we provide the proof in Section 7 in the supplementary.
Comparison with DoG. After the initial version of this paper, Ivgi et al. (2023) reported a convergence guarantee for the unweighted average  $\hat{x}_T = \frac{1}{T}\sum_{k=0}^{T-1}x_k$  returned by DoG. In particular, Proposition 3 in their work gives the rate
$$
f (\hat {x} _ {T}) - f _ {*} = \mathcal {O} \left(\frac {L (D _ {0} \log_ {+} \frac {\bar {r} _ {T}}{r _ {\epsilon}} + \bar {r} _ {T}) ^ {2}}{T}\right) = \mathcal {O} \left(\frac {L D ^ {2}}{T} \log_ {+} ^ {2} \frac {D}{\bar {r} _ {\epsilon}}\right).
$$
where  $D_0 = \| x_0 - x_*\|$ , and where in the second step we used the bound  $D_0 \leq D$  and  $\bar{r}_T \leq D$ . This rate is the same as that achieved by the weighted average of the DoWG iterates up to an extra logarithmic factor  $\log_{+}\frac{D}{\bar{r}_{\epsilon}}$ . We note that DoG also has a guarantee in the stochastic setting, provided the gradients are bounded locally with a known constant, while in this work we have focused exclusively on the deterministic setting.
Figure 3: DoWG iterations on  $\ell_2$ -regularized linear regression on the mushrooms dataset from LibSVM (Chang and Lin, 2011) with  $r_{\epsilon} = 10^{-6}$ . Top (a) shows the function suboptimality over time. Observe that as the number of iterations grow, the method becomes non-monotonic. Bottom (b) shows the DoWG stepsize over time.
Edge of Stability. Like NGD, DoWG also tends to increase the stepsize and train at the edge of stability. The intuition from NGD carries over: in order to preserve the convergence rate of GD, DoWG tends to drive the stepsize larger. However, once it overshoots, the gradients quickly diverge, forcing the stepsize back down. Figure 3 shows the performance of DoWG and its stepsize on the same regularized linear regression problem as in Figure 2. Comparing the two figures, we observe that DoWG is also non-monotonic and trains close to the edge of stability, but its stepsize oscillates less than NGD's effective stepsize.
Universality. Theorems 4 and 3 together show that DoWG is universal, i.e. it almost recovers the convergence of gradient descent with tuned stepsizes in both the smooth and nonsmooth settings. As the optimal stepsize for gradient descent can differ significantly between the two settings, we believe that achieving both rates simultaneously without any parameter-tuning or search procedures is a significant strength of DoWG.
Figure 4: VGG11 (top) and ResNet-50 (bottom) training on CIFAR10. Left: test accuracy, middle: train loss, right: step sizes.
We compare DoWG to DoG, L-DoG from Ivgi et al. (2023), for all of which we also report performance of the polynomially-averaged iterate with power 8 as recommended by Ivgi et al. (2023). We also add comparison against Adam (Kingma and Ba, 2015) with cosine annealing and the standard step size  $10^{-3}$ . All methods are used with batch size 256 with no weight decay on a single RTX3090 GPU. We plot the results in Figure 4 with the results averaged over 8 random seeds. We train the VGG11 (Simonyan and Zisserman, 2015) and ResNet-50 (He et al., 2016) neural network architectures on CIFAR10 (Krizhevsky, 2009) using PyTorch (Paszke et al., 2019), and implement<sup>1</sup> DoWG on top of the DoG code<sup>2</sup>. Unsurprisingly, DoWG's estimates of the step size are larger than that of DoG and D-Adapt-norm, which also makes it less stable on ResNet-50. While the last iterate of DoWG gives worse test accuracy than Adam, the average iterate of DoWG often performs better.
Finally, we note that while both neural networks tested are generally nonsmooth, recent work shows local smoothness can significantly influence and be influenced by a method's trajectory (Cohen et al., 2022; Pan and Li, 2022). We believe this adaptivity to smoothness might explain the empirical difference between DoWG and DoG, but leave a rigorous discussion of adaptivity to local smoothness to future work.
1https://github.com/rka97/dowg
$^{2}$ https://github.com/formll dog
1 Introduction 1
2 Related Work 2
3 Algorithms and theory 4
3.1Baselines: gradient descent and normalized gradient descent 4
3.2 DoWG 6
4 Experimental results 9
5 Algorithm-independent results 15
6 Proofs for DoWG 16
6.1 Smooth case 18
6.2 Nonsmooth case 20
7 Unconstrained domain extension 22
5 Algorithm-independent results
In this section we collect different results that are algorithm-independent, the first is a consequence of smoothness:
Fact 1. Suppose that  $f$  is smooth and lower bounded by  $f_{*}$ . Then for all  $x \in \mathbb{R}^d$  we have,
$$
\left\| \nabla f (x) \right\| ^ {2} \leq 2 L \left(f (x) - f _ {*}\right).
$$
Proof. This is a common result in the literature, and finds applications in convex and non-convex optimization see e.g. (Levy, 2017; Levy et al., 2018; Orabona and Cutkosky, 2020; Khaled and Richtárik, 2020). We include the proof for completeness. Let  $x \in \mathbb{R}^d$  and define  $x_+ = x - \frac{1}{L}\nabla f(x)$ . Then by smoothness
$$
\begin{array}{l} f (x _ {+}) \leq f (x) + \langle \nabla f (x), x _ {+} - x \rangle + \frac {L}{2} \| x _ {+} - x \| ^ {2} \\ = f (x) - \frac {1}{L} \| \nabla f (x) \| ^ {2} + \frac {1}{2 L} \| \nabla f (x) \| ^ {2} \\ = f (x) - \frac {1}{2 L} \| \nabla f (x) \| ^ {2}. \\ \end{array}
$$
Because  $f$  is lower bounded by  $f_{*}$  we thus have
$$
f _ {*} \leq f (x _ {+}) \leq f (x) - \frac {1}{2 L} \| \nabla f (x) \| ^ {2}.
$$
Rearranging gives  $\| \nabla f(x)\| ^2\leq 2L(f(x) - f_*)$
The next two results are helpful algebraic identities that will be useful for the proof of DoWG.
Lemma 1. (Ivgi et al., 2023, Lemma 4). Let  $a_0, \dots, a_t$  be a nondecreasing sequence of nonnegative numbers. Then
$$
\sum_ {k = 1} ^ {t} \frac {a _ {k} - a _ {k - 1}}{\sqrt {a _ {k}}} \leq 2 \left(\sqrt {a _ {t}} - \sqrt {a _ {0}}\right).
$$
Proof. This is (Ivgi et al., 2023, Lemma 4). We include the proof for completeness:
$$
\begin{array}{l} \sum_ {k = 1} ^ {t} \frac {a _ {k} - a _ {k - 1}}{\sqrt {a _ {k}}} = \sum_ {k = 1} ^ {t} \frac {\left(\sqrt {a _ {k}} - \sqrt {a _ {k - 1}}\right) \left(\sqrt {a _ {k}} + \sqrt {a _ {k - 1}}\right)}{\sqrt {a _ {k}}} \\ \leq 2 \sum_ {k = 1} ^ {t} \left(\sqrt {a _ {k}} - \sqrt {a _ {k - 1}}\right) \\ = 2 \left(\sqrt {a _ {t}} - \sqrt {a _ {0}}\right). \\ \end{array}
$$
Lemma 2. ((Ivgi et al., 2023, Lemma 3), similar to (Defazio and Mishchenko, 2023, Lemma 11)). Let  $s_0, s_1, \ldots, s_T$  be a positive increasing sequence. Then
$$
\max _ {t \leq T} \sum_ {i <   t} \frac {s _ {i}}{s _ {t}} \geq \frac {1}{e} \left(\frac {T}{\log_ {+} (s _ {T} / s _ {0})} - 1\right),
$$
where  $\log_+x\stackrel {def}{=}\log x + 1$
Proof. This is (Ivgi et al., 2023, Lemma 3). We include the proof for completeness: Define  $K = \left\lceil \log \frac{s_T}{s_0} \right\rceil$  and  $n = \left\lfloor \frac{T}{K} \right\rfloor$ . Then,
$$
\log \left(\frac {s _ {T}}{s _ {0}}\right) \geq \sum_ {k = 0} ^ {K - 1} \log \left(\frac {s _ {n (k + 1)}}{s _ {n k}}\right) \geq K \min  _ {k <   K} \log \frac {s _ {n (k + 1)}}{s _ {n k}}.
$$
Rearranging and using  $K = \lceil \log \frac{s_T}{s_0} \rceil$  gives
$$
\min  _ {k <   K} \log \frac {s _ {n (k + 1)}}{s _ {n k}} \leq \frac {\log \frac {s _ {T}}{s _ {0}}}{K} \leq 1.
$$
$$
\min  _ {k <   K} \frac {s _ {n (k + 1)}}{s _ {n k}} \leq e.
$$
$$
\begin{array}{l} \max  _ {t \leq T} \sum_ {i \leq t} \frac {s _ {i}}{s _ {t}} \geq \max  _ {t \leq T} n \frac {s _ {t - n}}{s _ {t}} \\ \geq \max  _ {k \leq K} n \frac {\mathbf {s} _ {n (k - 1)}}{s _ {n k}} \\ \geq e ^ {- 1} n \\ = e ^ {- 1} \left\lfloor \frac {T}{K} \right\rfloor \\ \geq e ^ {- 1} \left(\frac {T}{K} - 1\right) \\ \geq e ^ {- 1} \left(\frac {T}{\log \left(\frac {s _ {T}}{s _ {0}}\right) + 1} - 1\right). \\ \end{array}
$$
6 Proofs for DoWG
This section collects proofs for DoWG. First, we give the following lemma, which holds under convexity alone (regardless of whether  $f$  is smooth or Lipschitz).
Lemma 3. Suppose that  $f$  is convex and has minimizer  $x_{*}$ . For the iterations generated by Algorithm 1, we have
$$
\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} \left\langle \nabla f (x _ {k}), x _ {k} - x _ {*} \right\rangle \leq 2 \bar {r} _ {t} \left[ \bar {d} _ {t} + \bar {r} _ {t} \right] \sqrt {v _ {t - 1}}, \tag {6}
$$
where  $\bar{d}_t = \max_{k\leq t}d_k$
Proof. This proof follows the proof of DoG (Ivgi et al., 2023, Lemma 1), itself a modification of the standard proof for adaptive cumulative gradient normalization methods (Gupta et al., 2017) incorporating insights from (Carmon and Hinder, 2022). We specifically modify the proof to handle the weighting scheme we use in DoWG. By the nonexpansivity of the projection we have
$$
\begin{array}{l} d _ {k + 1} ^ {2} = \left\| x _ {k + 1} - x _ {*} \right\| ^ {2} \\ \leq \left\| x _ {k} - \eta_ {k} \nabla f (x _ {k}) - x _ {*} \right\| ^ {2} \\ = \left\| x _ {k} - x _ {*} \right\| ^ {2} - 2 \eta_ {k} \left\langle \nabla f (x _ {k}), x _ {k} - x _ {*} \right\rangle + \eta_ {k} ^ {2} \left\| \nabla f (x _ {k}) \right\| ^ {2} \\ = d _ {k} ^ {2} - 2 \eta_ {k} \left\langle \nabla f (x _ {k}), x _ {k} - x _ {*} \right\rangle + \eta_ {k} ^ {2} \| \nabla f (x _ {k}) \| ^ {2}. \\ \end{array}
$$
Rearranging and dividing by  $2\eta_{k}$  we get
$$
\langle \nabla f (x _ {k}), x _ {k} - x _ {*} \rangle \leq \frac {d _ {k} ^ {2} - d _ {k + 1} ^ {2}}{2 \eta_ {k}} + \frac {\eta_ {k}}{2} \| \nabla f (x _ {k}) \| ^ {2}.
$$
Multiplying both sides by  $\bar{r}_k^2$  we get
$$
\bar {r} _ {k} ^ {2} \left<   \nabla f (x _ {k}), x _ {k} - x _ {*} \right> \leq \frac {1}{2} \frac {\bar {r} _ {k} ^ {2}}{\eta_ {k}} \left[ d _ {k} ^ {2} - d _ {k + 1} ^ {2} \right] + \frac {1}{2} \bar {r} _ {k} ^ {2} \eta_ {k} \| \nabla f (x _ {k}) \| ^ {2}.
$$
Summing up as  $k$  varies from 0 to  $t - 1$  we get
$$
\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} \left\langle \nabla f (x _ {k}), x _ {k} - x _ {*} \right\rangle \leq \frac {1}{2} \underbrace {\left[ \sum_ {k = 0} ^ {t - 1} \frac {\bar {r} _ {k} ^ {2}}{\eta_ {k}} \left(d _ {k} ^ {2} - d _ {k + 1} ^ {2}\right) \right]} _ {\text {(A)}} + \frac {1}{2} \underbrace {\left[ \sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} \eta_ {k} \| \nabla f (x _ {k}) \| ^ {2} \right]} _ {\text {(B)}}. \tag {7}
$$
We shall now bound each of the terms (A) and (B). We have
$$
\begin{array}{l} (A) = \sum_ {k = 0} ^ {t - 1} \frac {\bar {r} _ {k} ^ {2}}{\eta_ {k}} \left(d _ {k} ^ {2} - d _ {k + 1} ^ {2}\right) \\ = \sum_ {k = 0} ^ {t - 1} \sqrt {v _ {k}} \left(d _ {k} ^ {2} - d _ {k + 1} ^ {2}\right) (8) \\ = d _ {0} ^ {2} \sqrt {v _ {0}} - d _ {t} ^ {2} \sqrt {v _ {t - 1}} + \sum_ {k = 1} ^ {t - 1} d _ {k} ^ {2} \left(\sqrt {v _ {k}} - \sqrt {v _ {k - 1}}\right) (9) \\ \leq \bar {d} _ {t} ^ {2} \sqrt {v _ {0}} - d _ {t} ^ {2} \sqrt {v _ {t - 1}} + \bar {d} _ {t} ^ {2} \sum_ {k = 1} ^ {t - 1} \left(\sqrt {v _ {k}} - \sqrt {v _ {k - 1}}\right) (10) \\ = \sqrt {v _ {t - 1}} \left[ \bar {d} _ {t} ^ {2} - d _ {t} ^ {2} \right] (11) \\ \leq 4 \bar {r} _ {t} \bar {d} _ {t} \sqrt {v _ {t - 1}}, (12) \\ \end{array}
$$
where eq. (8) holds by definition of the DoWG stepsize  $\eta_{k}$ , eq. (9) holds by telescoping, eq. (10) holds because  $v_{k} = v_{k - 1} + \bar{r}_{k}^{2}\| \nabla f(x_{k})\|^{2}\geq v_{k - 1}$  and hence  $\sqrt{v_k}\geq \sqrt{v_{k - 1}}$  and  $d_k^2\leq \bar{d}_t^2$  by definition. Equation (11) just follows by telescoping. Finally observe that  $\bar{d}_t^2 -d_t^2 = d_s^2 -d_t^2$  for some  $s\in [t]$ , and  $d_s^2 -d_t^2 = (d_s - d_t)(d_s + d_t)$ . Then by the triangle inequality and that the sequence
$\bar{r}_k$  is monotonically nondecreasing we have
$$
\begin{array}{l} d _ {s} - d _ {t} = \left\| x _ {s} - x _ {*} \right\| - \left\| x _ {t} - x _ {*} \right\| \\ \leq \left\| x _ {s} - x _ {t} \right\| \\ \leq \left\| x _ {s} - x _ {0} \right\| + \left\| x _ {t} - x _ {0} \right\| \\ = r _ {s} + r _ {t} \\ \leq \bar {r} _ {s} + \bar {r} _ {t} \\ \leq 2 \bar {r} _ {t}. \\ \end{array}
$$
Therefore  $d_s^2 - d_t^2 \leq (\bar{r}_s + \bar{r}_t)(d_s + d_t) \leq 4\bar{r}_t\bar{d}_t$ . This explains eq. (12).
For the second term in eq. (7), we have
$$
\begin{array}{l} (\mathbf {B}) = \sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} \eta_ {k} \| \nabla f (x _ {k}) \| ^ {2} \\ = \sum_ {k = 1} ^ {t - 1} \frac {\bar {r} _ {k} ^ {4}}{\sqrt {v _ {k}}} \| \nabla f (x _ {k}) \| ^ {2} \\ = r _ {0} ^ {2} \sqrt {v _ {0}} + \sum_ {k = 1} ^ {t - 1} \frac {\bar {r} _ {k} ^ {4}}{\sqrt {v _ {k}}} \| \nabla f (x _ {k}) \| ^ {2} \\ \leq \bar {r} _ {t} ^ {2} \sqrt {v _ {0}} + \bar {r} _ {t} ^ {2} \sum_ {k = 1} ^ {t - 1} \frac {\bar {r} _ {k} ^ {2} \| \nabla f (x _ {k}) \| ^ {2}}{\sqrt {v _ {k}}} \\ = \bar {r} _ {t} ^ {2} \sqrt {v _ {0}} + \bar {r} _ {t} ^ {2} \sum_ {k = 1} ^ {t - 1} \frac {v _ {k} - v _ {k - 1}}{\sqrt {v _ {k}}} \\ = \bar {r} _ {t} ^ {2} \sqrt {v _ {0}} + \bar {r} _ {t} ^ {2} \sum_ {k = 1} ^ {t - 1} \frac {v _ {k} - v _ {k - 1}}{\sqrt {v _ {k}}} \\ \leq \bar {r} _ {t} ^ {2} \sqrt {v _ {0}} + 2 \bar {r} _ {t} ^ {2} \left[ \sqrt {v _ {t - 1}} - \sqrt {v _ {0}} \right] (13) \\ = 2 \bar {r} _ {t} ^ {2} \sqrt {v _ {t - 1}}. (14) \\ \end{array}
$$
where eq. (13) is by Lemma 1. Plugging eqs. (12) and (14) in eq. (7) gives
$$
\begin{array}{l} \sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} \left\langle \nabla f (x _ {k}), x _ {k} - x _ {*} \right\rangle \leq 2 \bar {r} _ {t} \bar {d} _ {t} \sqrt {v _ {t - 1}} + \bar {r} _ {t} ^ {2} \sqrt {v _ {t - 1}} \\ \leq 2 \bar {r} _ {t} \left[ \bar {d} _ {t} + \bar {r} _ {t} \right] \sqrt {v _ {t - 1}}. \\ \end{array}
$$
6.1 Smooth case
We now prove the convergence of DoWG under smoothness. In particular, we shall use Fact 1 and the DoWG design to bound the weighted cumulative error  $S_{t} = \sum_{k=0}^{t-1} \overline{r}_{k}^{2} [f(x_{k}) - f_{*}]$  by its square root multiplied by a problem-dependent constant. We note that a similar trick is used in the analysis of AdaGrad-Norm (Levy et al., 2018), in reductions from online convex optimization to stochastic smooth optimization (Orabona and Cutkosky, 2020), and in the method of (Carmon and Hinder, 2022). However, in all the mentioned cases, the unweighted error  $M_{t} = \sum_{k=0}^{t-1} [f(x_{k}) - f_{*}]$  is bounded by its square root. Here, DoWG's design allows us to bound the weighted errors  $S_{t}$  instead.
Proof of Theorem 4. We start with Lemma 3. Let  $t \in [T]$ . By eq. (6) we have
$$
\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} \left\langle \nabla f (x _ {k}), x _ {k} - x _ {*} \right\rangle \leq 2 \bar {r} _ {t} \left[ \bar {d} _ {t} + \bar {r} _ {t} \right] \sqrt {v _ {t - 1}}. \tag {15}
$$
Observe that by convexity we have
$$
\langle \nabla f (x _ {k}), x _ {k} - x _ {*} \rangle \geq f (x _ {k}) - f _ {*}.
$$
Using this to lower bound the left-hand side of eq. (15) gives
$$
\begin{array}{l} \sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} [ f (x _ {k}) - f _ {*} ] \leq \sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} \langle \nabla f (x _ {k}), x _ {k} - x _ {*} \rangle \\ \leq 2 \bar {r} _ {t} \left[ \bar {d} _ {t} + \bar {r} _ {t} \right] \sqrt {v _ {t - 1}}. \tag {16} \\ \end{array}
$$
We have by smoothness that  $\| \nabla f(x)\| ^2\leq 2L(f(x) - f_*)$  for all  $x\in \mathbb{R}^d$ , therefore
$$
v _ {t - 1} = \sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} \| \nabla f (x _ {k}) \| ^ {2} \leq 2 L \sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} [ f (x _ {k}) - f _ {*} ].
$$
Taking square roots we get
$$
\sqrt {v _ {t - 1}} \leq \sqrt {2 L} \sqrt {\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} \left[ f \left(x _ {k}\right) - f _ {*} \right]}. \tag {17}
$$
Using eq. (17) in eq. (16) gives
$$
\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} [ f (x _ {k}) - f _ {*} ] \leq 2 \sqrt {2 L} \bar {r} _ {t} (\bar {d} _ {t} + \bar {r} _ {t}) \sqrt {\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} [ f (x _ {k}) - f _ {*} ]}.
$$
If  $f(x_{k}) - f_{*} = 0$  for some  $k \in [t - 1]$  then the statement of the theorem is trivial. Otherwise, we can divide both sides by the latter square root to get
$$
\sqrt {\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} [ f (x _ {k}) - f _ {*} ]} \leq 2 \sqrt {2 L} \bar {r} _ {t} (\bar {d} _ {t} + \bar {r} _ {t}).
$$
Squaring both sides gives
$$
\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} \left[ f (x _ {k}) - f _ {*} \right] \leq 8 L \bar {r} _ {t} ^ {2} \left(\bar {d} _ {t} + \bar {r} _ {t}\right) ^ {2}.
$$
Dividing both sides by  $\sum_{k=0}^{t-1} \bar{r}_k^2$  we get
$$
\begin{array}{l} \frac {1}{\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2}} \sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} [ f (x _ {k}) - f _ {*} ] \leq \frac {8 L \bar {r} _ {t} ^ {2} (\bar {d} _ {t} + \bar {r} _ {t}) ^ {2}}{\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2}} \\ = \frac {8 L (\bar {d} _ {t} + \bar {r} _ {t}) ^ {2}}{\sum_ {k = 0} ^ {t - 1} \frac {\bar {r} _ {k} ^ {2}}{\bar {r} _ {t} ^ {2}}}. \\ \end{array}
$$
By convexity we have
$$
\begin{array}{l} f (\bar {x} _ {t}) - f _ {*} \leq \frac {1}{\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2}} \sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} [ f (x _ {k}) - f _ {*} ] \\ \leq \frac {8 L \left(\bar {d} _ {t} + \bar {r} _ {t}\right) ^ {2}}{\sum_ {k = 0} ^ {t - 1} \frac {\bar {r} _ {k} ^ {2}}{\bar {r} _ {t} ^ {2}}}. \tag {18} \\ \end{array}
$$
By Lemma 2 applied to the sequence  $s_k = \bar{r}_k^2$  we have that for some  $t \in [T]$
$$
\sum_ {k = 0} ^ {t - 1} \frac {\bar {r} _ {k} ^ {2}}{\bar {r} _ {t} ^ {2}} \geq \frac {1}{e} \left(\frac {T}{\log_ {+} \frac {\bar {r} _ {T} ^ {2}}{\bar {r} _ {0} ^ {2}}} - 1\right).
$$
Because  $\mathcal{X}$  has diameter  $D$  we have  $\bar{r}_T^2\leq D^2$  and therefore
$$
\sum_ {k = 0} ^ {t - 1} \frac {\bar {r} _ {k} ^ {2}}{\bar {r} _ {t} ^ {2}} \geq \frac {1}{e} \left(\frac {T}{\log_ {+} \frac {D ^ {2}}{\bar {r} _ {0} ^ {2}}} - 1\right). \tag {19}
$$
We now have two cases:
- If  $T \geq 2\log_{+}\frac{D^{2}}{\bar{r}_{0}^{2}}$  then  $\frac{T}{\log\frac{D^2}{r_0^2}} - 1 \geq \frac{T}{2\log\frac{D^2}{r_0^2}}$  and we use this in eqs. (18) and (19) to get
$$
f (\bar {x} _ {t}) - f _ {*} \leq \frac {1 6 e L (\bar {d} _ {t} + \bar {r} _ {t}) ^ {2}}{T} \log \frac {\bar {r} _ {T} ^ {2}}{\bar {r} _ {0} ^ {2}}.
$$
Observe that because  $\mathcal{X}$  has diameter at most  $D$  we have  $\bar{d}_t + \bar{r}_t\leq 2D$  , therefore
$$
f (\bar {x} _ {t}) - f _ {*} \leq \frac {6 4 e L D ^ {2}}{T} \log_ {+} \frac {\bar {r} _ {T} ^ {2}}{\bar {r} _ {0} ^ {2}} = \mathcal {O} \left[ \frac {L D ^ {2}}{T} \log_ {+} \frac {D}{\bar {r} _ {0}} \right].
$$
- If  $T < 2\log_{+}\frac{D^{2}}{\bar{r}_{0}^{2}}$ , then  $1 < \frac{2\log_{+}\frac{D^{2}}{\bar{r}_{0}^{2}}}{T}$ . Let  $t \in [T]$ . Using smoothness and this fact we have
$$
f (\bar {x} _ {t}) - f _ {*} \leq \frac {L}{2} \| \bar {x} _ {t} - x _ {*} \| ^ {2} \leq \frac {L \| \bar {x} _ {t} - x _ {*} \| ^ {2}}{T} \log_ {+} \frac {D ^ {2}}{\bar {r} _ {0} ^ {2}}.
$$
Observe  $\bar{x}_t, x_* \in \mathcal{X}$  and  $\mathcal{X}$  has diameter  $D$ , hence  $\| \bar{x}_t - x_* \|^2 \leq D^2$  and we get