SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
f (\bar {x} _ {t}) - f _ {*} \leq \frac {L D ^ {2}}{T} \log_ {+} \frac {D ^ {2}}{\bar {r} _ {0} ^ {2}} = \mathcal {O} \left(\frac {L D ^ {2}}{T} \log_ {+} \frac {D}{\bar {r} _ {0}}\right).
$$
Thus in both cases we have that  $f(\bar{x}_t) - f_* = \mathcal{O}\left(\frac{LD^2}{T}\log_+ \frac{D}{\bar{r}_0}\right)$ , this completes our proof.
6.2 Nonsmooth case
We now give the proof of DoWG's convergence when  $f$  is Lipschitz.
Proof of Theorem 3. We start with Lemma 3. Let  $t \in [T]$ . By eq. (6) we have
$$
\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} \left\langle \nabla f (x _ {k}), x _ {k} - x _ {*} \right\rangle \leq 2 \bar {r} _ {t} \left[ \bar {d} _ {t} + \bar {r} _ {t} \right] \sqrt {v _ {t - 1}}. \tag {20}
$$
Using this to lower bound the left-hand side of eq. (20) gives
$$
\begin{array}{l} \sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} [ f (x _ {k}) - f _ {*} ] \leq \sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} \langle \nabla f (x _ {k}), x _ {k} - x _ {*} \rangle \\ \leq 2 \bar {r} _ {t} \left[ \bar {d} _ {t} + \bar {r} _ {t} \right] \sqrt {v _ {t - 1}}. \tag {21} \\ \end{array}
$$
We have by the fact that  $f$  is  $G$ -Lipschitz that  $\| \nabla f(x) \|^{2} \leq G^{2}$  for all  $x \in \mathcal{X}$ . Therefore,
$$
\begin{array}{l} v _ {t - 1} = \sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} \| \nabla f (x _ {k}) \| ^ {2} \\ \leq \bar {r} _ {t} ^ {2} \sum_ {k = 0} ^ {t - 1} \| \nabla f (x _ {k}) \| ^ {2} \\ \leq \bar {r} _ {t} ^ {2} G ^ {2} T. \\ \end{array}
$$
Taking square roots and plugging into eq. (21) gives
$$
\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} [ f (x _ {k}) - f _ {*} ] \leq 2 \bar {r} _ {t} ^ {2} [ \bar {d} _ {t} + \bar {r} _ {t} ] G \sqrt {T}.
$$
$$
\frac {1}{\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2}} \sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} [ f (x _ {k}) - f _ {*} ] \leq \frac {2 [ \bar {d} _ {t} + \bar {r} _ {t} ] G \sqrt {T}}{\sum_ {k = 0} ^ {t - 1} \frac {\bar {r} _ {k} ^ {2}}{\bar {r} _ {t} ^ {2}}}. \tag {22}
$$
Because  $\bar{r}_T\leq D$  we further have
$$
\sum_ {k = 0} ^ {t - 1} \frac {\bar {r} _ {k} ^ {2}}{\bar {r} _ {t} ^ {2}} \geq \frac {1}{e} \left(\frac {T}{\log_ {+} \frac {D ^ {2}}{\bar {r} _ {0} ^ {2}}} - 1\right). \tag {23}
$$
- If  $T \geq 2\log_{+}\frac{D^{2}}{\bar{r}_{0}^{2}}$ : then  $\frac{T}{\log\frac{\bar{r}_{T}^{2}}{\bar{r}_{0}^{2}}} - 1 \geq \frac{T}{2\log\frac{\bar{r}_{T}^{2}}{\bar{r}_{0}^{2}}}$ . We can use this in eq. (22) alongside eq. (23) and the fact that  $\log_{+}x^{2} = \max(\log x^{2}, 1) = \max(2\log x, 1) \leq 2\log_{+}x$  to get
$$
\frac {1}{\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2}} \sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} [ f (x _ {k}) - f _ {*} ] \leq \frac {8 [ \bar {d} _ {t} + \bar {r} _ {t} ] G}{\sqrt {T}} \log_ {+} \frac {D}{\bar {r} _ {0}}.
$$
Because the diameter of  $\mathcal{X}$  is bounded by  $D$  we have  $\bar{r}_T\leq D$  and  $\bar{d}_t\leq D$ , using this and convexity we get
$$
\begin{array}{l} f (\bar {x} _ {t}) - f _ {*} \leq \frac {1}{\sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2}} \sum_ {k = 0} ^ {t - 1} \bar {r} _ {k} ^ {2} [ f (x _ {k}) - f _ {*} ] \\ \leq \frac {8 [ \bar {d} _ {t} + \bar {r} _ {t} ] G}{\sqrt {T}} \log \frac {\bar {r} _ {T}}{\bar {r} _ {0}} \\ \leq \frac {1 6 D G}{\sqrt {T}} \log \frac {D}{r _ {0}}. \\ \end{array}
$$
If  $T <   2\log_{+}\frac{D^{2}}{\vec{r}_{0}^{2}}$  : then
$$
1 <   \frac {2 \log_ {+} \frac {D ^ {2}}{\bar {r} _ {0} ^ {2}}}{T} \leq \frac {4 \log_ {+} \frac {D}{\bar {r} _ {0}}}{T}. \tag {24}
$$
By convexity and Cauchy-Schwartz we have
$$
\begin{array}{l} f (\bar {x} _ {t}) - f _ {*} \leq \langle \nabla f (\bar {x} _ {t}), \bar {x} _ {t} - x _ {*} \rangle \\ \leq \| \nabla f (\bar {x} _ {t}) \| \| \bar {x} _ {t} - x _ {*} \|. \tag {25} \\ \end{array}
$$
Because  $f$  is  $G$ -Lipschitz then  $\| \nabla f(\bar{x}_t) \| \leq G$  and because  $\mathcal{X}$  has diameter  $D$  we have  $\| \bar{x}_t - x_* \| \leq D$ . Using this and eq. (24) in eq. (25) gives
$$
\begin{array}{l} f (\bar {x} _ {t}) - f _ {*} \leq D G \\ <   \frac {4 D G \log_ {+} \frac {D}{\bar {r} _ {0}}}{T}. \\ \end{array}
$$
Now because  $T \geq 1$  we have  $\sqrt{T} \leq T$  and hence
$$
f (\bar {x} _ {t}) - f _ {*} \leq \frac {4 D G \log_ {+} \frac {D}{\bar {r} _ {0}}}{\sqrt {T}}.
$$
In both cases, we have that  $f(\bar{x}_t) - f_* = \mathcal{O}\left(\frac{4DG\log_+ \frac{D}{r_0}}{\sqrt{T}}\right)$ , and this completes our proof.
7 Unconstrained domain extension
In this section we consider the case where the domain set is unbounded, and we seek dependence only on  $d_0 = \| x_0 - x_*\|$ . We use the same technique for handling the unconstrained problem as (Ivgi et al., 2023) in this section and consider DoWG iterates with the reduced stepsizes
$$
\eta_ {t} = \frac {\bar {r} _ {t} ^ {2}}{\sqrt {v _ {t}} \log \frac {2 v _ {t}}{v _ {0}}} \quad v _ {t} = v _ {t - 1} + \bar {r} _ {t} ^ {2} \| \nabla f (x _ {t}) \| ^ {2}. \tag {26}
$$
We prove that with this stepsize, the iterates do not venture far from the initialization. The proof follows (Ivgi et al., 2023).
Lemma 4. (Stability). For the iterates  $x_{t + 1} = x_t - \eta_t\nabla f(x_t)$  following the stepsize scheme given by (26) we have  $\bar{d}_t^2\leq 12d_0$  and  $\bar{r}_t^2\leq 32d_0^2$  provided that  $r_0\leq d_0$ .
Proof. By expanding the square and convexity
$$
d _ {k + 1} ^ {2} - d _ {k} ^ {2} \leq \eta_ {t} ^ {2} \| g _ {t} \| ^ {2}.
$$
Summing up as  $t$  varies from  $k = 1$  to  $k = t$  and using (Ivgi et al., 2023, Lemma 6)
$$
d _ {t} ^ {2} - d _ {1} ^ {2} \leq \sum_ {k = 1} ^ {t} \frac {\bar {r} _ {k} ^ {4}}{v _ {k}} \frac {\| \nabla f (x _ {k}) \| ^ {2}}{4 \log^ {2} \frac {2 v _ {k}}{v _ {0}}} \leq \frac {\bar {r} _ {t} ^ {2}}{4} \sum_ {k = 1} ^ {t} \frac {v _ {k} - v _ {k - 1}}{v _ {k} \log_ {+} ^ {2} \frac {v _ {k}}{v _ {0}}} \leq \frac {\bar {r} _ {t} ^ {2}}{4}.
$$
Therefore we have  $d_t^2 \leq d_1^2 + \frac{\bar{r}_t^2}{4}$ . Now suppose for the sake of induction that  $\bar{r}_t^2 \leq 8d_1^2$ , then applying the last equation we get  $d_t^2 \leq 3d_1^2$ . Taking square roots gives  $d_t \leq \sqrt{3} d_1$ . By the triangle inequality we then get
$$
\left\| x _ {t + 1} - x _ {0} \right\| \leq \left\| x _ {t + 1} - x _ {*} \right\| + \left\| x _ {*} - x _ {0} \right\| \leq (1 + \sqrt {3}) d _ {1}.
$$
Squaring both sides gives  $\| x_{t + 1} - x_0\|^2\leq (1 + \sqrt{3})^2 d_1^2\leq 8d_1^2$ . This completes our induction and we have  $\bar{r}_t^2\leq 8d_1^2$  for all  $t$ . Finally, observe that
$$
d _ {1} = \left\| x _ {1} - x _ {*} \right\| \leq \left\| x _ {1} - x _ {0} \right\| + \left\| x _ {0} - x _ {*} \right\| = r _ {\epsilon} + d _ {0} \leq 2 d _ {0}.
$$
It follows that  $\bar{r}_t^2\leq 8d_1^2\leq 32d_0^2$  for all  $t$ . Finally, we have  $d_t^2\leq d_1^2 +\frac{\bar{r}_t^2}{4}\leq 3d_1^2\leq 12d_0^2$ . This completes our proof.
Therefore the iterates stay bounded. The rest of the proof then follows Theorems 3 and 4 and is omitted for simplicity.. In both cases it gives the same results with  $D_0 = \| x_0 - x_*\|$  instead of  $D$  , up to extra constants and polylogarithmic factors.