SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {t = 1} ^ {T} \frac {a _ {t}}{b t} \leq \frac {a _ {\max}}{b} (1 + \ln A) + \frac {1}{b ^ {2}}.
$$
Proof of Lemma 6. The term of  $\| \mathbf{x} - \mathbf{y}\|^2$  can be decomposed as follows:
$$
\begin{array}{l} \left\| \mathbf {x} - \mathbf {y} \right\| ^ {2} = \left\| \sum_ {i = 1} ^ {N} p _ {i} \mathbf {x} _ {i} - \sum_ {i = 1} ^ {N} q _ {i} \mathbf {y} _ {i} \right\| ^ {2} = \left\| \sum_ {i = 1} ^ {N} p _ {i} \mathbf {x} _ {i} - \sum_ {i = 1} ^ {N} p _ {i} \mathbf {y} _ {i} + \sum_ {i = 1} ^ {N} p _ {i} \mathbf {y} _ {i} - \sum_ {i = 1} ^ {N} q _ {i} \mathbf {y} _ {i} \right\| ^ {2} \\ \leq 2 \left\| \sum_ {i = 1} ^ {N} p _ {i} (\mathbf {x} _ {i} - \mathbf {y} _ {i}) \right\| ^ {2} + 2 \left\| \sum_ {i = 1} ^ {N} (p _ {i} - q _ {i}) \mathbf {y} _ {i} \right\| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \leq 2 \left(\sum_ {i = 1} ^ {N} p _ {i} \| \mathbf {x} _ {i} - \mathbf {y} _ {i} \|\right) ^ {2} + 2 \left(\sum_ {i = 1} ^ {N} | p _ {i} - q _ {i} | \| \mathbf {y} _ {i} \|\right) ^ {2} \\ \leq 2 \sum_ {i = 1} ^ {N} p _ {i} \| \mathbf {x} _ {i} - \mathbf {y} _ {i} \| ^ {2} + 2 D ^ {2} \| \boldsymbol {p} - \boldsymbol {q} \| _ {1} ^ {2}, \\ \end{array}
$$
where the first inequality is due to  $(a + b)^2 \leq 2a^2 + 2b^2$  for any  $a, b \in \mathbb{R}$ , and the last step is due to Cauchy-Schwarz inequality, Assumption 1 and the definition of  $\ell_1$ -norm, finishing the proof.
Proof of Lemma 7. Denoting the optimal step size by  $\eta_{\star} \triangleq \sqrt{\ln(4XY) / X}$ , if the optimal step size satisfies  $\eta \leq \eta_{\star} \leq 2\eta$ , where  $\eta \leq \eta_{\star}$  can be guaranteed if  $C_0 \geq \frac{\sqrt{X}}{2T}$ , then it holds that
$$
\frac {1}{\eta} \ln \frac {Y}{\eta^ {2}} + \eta X \leq \frac {2}{\eta_ {\star}} \ln \frac {4 Y}{\eta_ {\star} ^ {2}} + \eta_ {\star} X \leq 3 \sqrt {X \ln (4 X Y)}.
$$
Otherwise, if the optimal step size is greater than the maximum step size in the parameter pool, i.e.,  $\eta_{\star} \geq (\eta = \eta_1 = \frac{1}{2C_0})$ , then we have
$$
\frac {1}{\eta} \ln \frac {Y}{\eta^ {2}} + \eta X \leq \frac {1}{\eta} \ln \frac {Y}{\eta^ {2}} + \eta_ {\star} X \leq 2 C _ {0} \ln \left(4 Y C _ {0} ^ {2}\right) + \sqrt {X \ln (4 X Y)}.
$$
Overall, it holds that
$$
\frac {1}{\eta} \ln \frac {Y}{\eta^ {2}} + \eta X \leq 2 C _ {0} \ln \left(4 Y C _ {0} ^ {2}\right) + 4 \sqrt {X \ln (4 X Y)},
$$
Proof of Lemma 8. The proof follows the same flow as Lemma 7.
Proof of Lemma 9. This result is inspired by Lemma 5 of Chen et al. [2023b], and we generalize it to arbitrary variables for our purpose. Specifically, we consider two cases:  $A < T$  and  $A \geq T$ . For the first case, if  $A < T$ , it holds that
$$
\sum_ {t = 1} ^ {T} \frac {a _ {t}}{b t} = \sum_ {t = 1} ^ {A} \frac {a _ {t}}{b t} + \sum_ {A + 1} ^ {T} \frac {a _ {t}}{b t} \leq \frac {a _ {\max}}{b} \sum_ {t = 1} ^ {A} \frac {1}{t} + \frac {1}{b (A + 1)} \sum_ {A + 1} ^ {T} a _ {t} \leq \frac {a _ {\max}}{b} (1 + \ln A) + \frac {1}{b ^ {2}},
$$
where the last step is due to  $\sum_{A + 1}^{T}a_{t}\leq \sum_{t = 1}^{T}a_{t}\leq A / b$ . The case of  $A < T$  can be proved similarly, which finishes the proof.
E.2 Stability Analysis of Base Algorithms
In this part, we analyze the negative stability terms in optimism OMD analysis, for convex, exp-concave and strongly convex functions. For simplicity, we define the empirical gradient variation:
$$
\bar {V} _ {T} \triangleq \sum_ {t = 2} ^ {T} \| \mathbf {g} _ {t} - \mathbf {g} _ {t - 1} \| ^ {2}, \text {w h e r e} \mathbf {g} _ {t} \triangleq \nabla f _ {t} (\mathbf {x} _ {t}). \tag {E.1}
$$
In the following, we provide two kinds of decompositions that are used in the analysis of gradient-variation and small-loss guarantees, respectively. First, using smoothness (Assumption 2), we have
$$
\bar {V} _ {T} = \sum_ {t = 2} ^ {T} \| \mathbf {g} _ {t} - \nabla f _ {t - 1} (\mathbf {x} _ {t}) + \nabla f _ {t - 1} (\mathbf {x} _ {t}) - \mathbf {g} _ {t - 1} \| ^ {2} \leq 2 V _ {T} + 2 L ^ {2} \sum_ {t = 2} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \| ^ {2}, \tag {E.2}
$$
Second, for an  $L$ -smooth and non-negative function  $f:\mathcal{X}\mapsto \mathbb{R}_+$ ,  $\| \nabla f(\mathbf{x})\| _2^2\leq 4Lf(\mathbf{x})$  holds for any  $\mathbf{x}\in \mathcal{X}$  [Srebro et al., 2010, Lemma 3.1]. It holds that
$$
\bar {V} _ {T} = \sum_ {t = 2} ^ {T} \| \mathbf {g} _ {t} - \mathbf {g} _ {t - 1} \| ^ {2} \leq 2 \sum_ {t = 2} ^ {T} \| \mathbf {g} _ {t} \| ^ {2} + 2 \sum_ {t = 2} ^ {T} \| \mathbf {g} _ {t - 1} \| ^ {2} \leq 4 \sum_ {t = 1} ^ {T} \| \mathbf {g} _ {t} \| ^ {2} \leq 1 6 L \sum_ {t = 1} ^ {T} f _ {t} (\mathbf {x} _ {t}). \tag {E.3}
$$
Next we provide the regret analysis in terms of the empirical gradient-variation  $\vec{V}_T$ , for convex (Lemma 10), exp-concave (Lemma 11), and strongly convex (Lemma 12) functions.
Lemma 10. Under Assumptions 1 and 2, if the loss functions are convex, optimistic OGD with the following update rule:
$$
\mathbf {x} _ {t} = \Pi_ {\mathcal {X}} \left[ \widehat {\mathbf {x}} _ {t} - \eta_ {t} \mathbf {m} _ {t} \right], \quad \widehat {\mathbf {x}} _ {t + 1} = \Pi_ {\mathcal {X}} \left[ \widehat {\mathbf {x}} _ {t} - \eta_ {t} \nabla f _ {t} (\mathbf {x} _ {t}) \right], \tag {E.4}
$$
where  $\Pi_{\mathcal{X}}[\mathbf{x}]\triangleq \arg \min_{\mathbf{y}\in \mathcal{X}}\| \mathbf{x} - \mathbf{y}\|$ ,  $\mathbf{m}_t = \nabla f_{t - 1}(\mathbf{x}_{t - 1})$  and  $\eta_t = \min \{D / \sqrt{1 + \bar{V}_{t - 1}},1 / \gamma \}$  enjoys the following empirical gradient-variation bound:
$$
\mathrm {R E G} _ {T} \leq 5 D \sqrt {1 + \bar {V} _ {T}} + \gamma D ^ {2} - \frac {\gamma}{4} \sum_ {t = 2} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \| ^ {2} + \mathcal {O} (1).
$$
Lemma 11. Under Assumptions 1 and 2, if the loss functions are  $\alpha$ -exp-concave, optimistic OMD with the following update rule:
$$
\mathbf {x} _ {t} = \underset {\mathbf {x} \in \mathcal {X}} {\arg \min } \left\{\langle \mathbf {m} _ {t}, \mathbf {x} \rangle + \mathcal {D} _ {\psi_ {t}} (\mathbf {x}, \widehat {\mathbf {x}} _ {t}) \right\}, \quad \widehat {\mathbf {x}} _ {t + 1} = \underset {\mathbf {x} \in \mathcal {X}} {\arg \min } \left\{\langle \nabla f _ {t} (\mathbf {x} _ {t}), \mathbf {x} \rangle + \mathcal {D} _ {\psi_ {t}} (\mathbf {x}, \widehat {\mathbf {x}} _ {t}) \right\},
$$
where  $\psi_t(\cdot) = \frac{1}{2} \| \cdot \|_{U_t}^2$ ,  $U_t = \gamma I + \frac{\alpha G^2}{2} I + \frac{\alpha}{2} \sum_{s=1}^{t-1} \nabla f_s(\mathbf{x}_s) \nabla f_s(\mathbf{x}_s)^\top$  and  $\mathbf{m}_t = \nabla f_{t-1}(\mathbf{x}_{t-1})$ , enjoys the following empirical gradient-variation bound:
$$
\operatorname {R E G} _ {T} \leq \frac {1 6 d}{\alpha} \ln \left(1 + \frac {\alpha}{8 \gamma d} \bar {V} _ {T}\right) + \frac {1}{2} \gamma D ^ {2} - \frac {\gamma}{4} \sum_ {t = 2} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \| ^ {2} + \mathcal {O} (1).
$$
Lemma 12. Under Assumptions 1 and 2, if the loss functions are  $\lambda$ -strongly convex, optimistic OGD (E.4) with  $\eta_t = 2 / (\gamma + \lambda t)$  and  $\mathbf{m}_t = \nabla f_{t-1}(\mathbf{x}_{t-1})$ , enjoys the following empirical gradient-variation bound:
$$
\operatorname {R E G} _ {T} \leq \frac {1 6 G ^ {2}}{\lambda} \ln \left(1 + \lambda \bar {V} _ {T}\right) + \frac {1}{4} \gamma D ^ {2} - \frac {\gamma}{8} \sum_ {t = 2} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \| ^ {2} + \mathcal {O} (1).
$$
Proof of Lemma 10. The proof mainly follows Theorem 11 of Chiang et al. [2012]. Following the standard analysis of optimistic OMD, e.g., Theorem 1 of Zhao et al. [2021], it holds that
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} f _ {t} (\mathbf {x} _ {t}) - \sum_ {t = 1} ^ {T} f _ {t} (\mathbf {x} ^ {\star}) \leq \underbrace {\sum_ {t = 1} ^ {T} \eta_ {t} \| \nabla f _ {t} (\mathbf {x} _ {t}) - \mathbf {m} _ {t} \| ^ {2}} _ {\text {A D A P T I V I T Y}} + \underbrace {\sum_ {t = 1} ^ {T} \frac {1}{\eta_ {t}} \left(\mathcal {D} _ {\psi} \left(\mathbf {x} ^ {\star} , \widehat {\mathbf {x}} _ {t}\right) - \mathcal {D} _ {\psi} \left(\mathbf {x} ^ {\star} , \widehat {\mathbf {x}} _ {t + 1}\right)\right)} _ {\text {O P T - G A P}} \\ - \underbrace {\sum_ {t = 1} ^ {T} \frac {1}{\eta_ {t}} \left(\mathcal {D} _ {\psi} \left(\widehat {\mathbf {x}} _ {t + 1} , \mathbf {x} _ {t}\right) + \mathcal {D} _ {\psi_ {t}} \left(\mathbf {x} _ {t} , \widehat {\mathbf {x}} _ {t}\right)\right)} _ {\text {S T A B I L i t y}}, \tag {E.5} \\ \end{array}
$$
where  $\mathbf{x}^{\star}\in \arg \min_{\mathbf{x}\in \mathcal{X}}\sum_{t = 1}^{T}f_{t}(\mathbf{x})$  and  $\psi (\cdot)\triangleq \frac{1}{2}\| \cdot \| ^2$ . The adaptivity term satisfies
$$
\begin{array}{l} \text {A D A P T I V I T Y} = \sum_ {t = 1} ^ {T} \eta_ {t} \| \nabla f _ {t} (\mathbf {x} _ {t}) - \mathbf {m} _ {t} \| ^ {2} \leq D \sum_ {t = 1} ^ {T} \frac {\| \nabla f _ {t} (\mathbf {x} _ {t}) - \nabla f _ {t - 1} (\mathbf {x} _ {t - 1}) \| ^ {2}}{\sqrt {1 + \sum_ {s = 1} ^ {t - 1} \| \nabla f _ {s} (\mathbf {x} _ {s}) - \nabla f _ {s - 1} (\mathbf {x} _ {s - 1}) \| ^ {2}}} \\ \leq 4 D \sqrt {1 + \sum_ {t = 1} ^ {T} \left\| \nabla f _ {t} (\mathbf {x} _ {t}) - \nabla f _ {t - 1} (\mathbf {x} _ {t - 1}) \right\| ^ {2}} + 4 D G ^ {2}, \\ \end{array}
$$
where the last step uses  $\sum_{t=1}^{T} a_t / \sqrt{1 + \sum_{s=1}^{t-1} a_s} \leq 4\sqrt{1 + \sum_{t=1}^{T} a_t} + \max_{t \in [T]} a_t$  [Pogodin and Lattimore, 2019, Lemma 4.8]. Next, we move on to the optimality gap,
$$
\mathrm {O P T - G A P} = \sum_ {t = 1} ^ {T} \frac {1}{\eta_ {t}} \left(\mathcal {D} _ {\psi} \left(\mathbf {x} ^ {\star}, \widehat {\mathbf {x}} _ {t}\right) - \mathcal {D} _ {\psi} \left(\mathbf {x} ^ {\star}, \widehat {\mathbf {x}} _ {t + 1}\right)\right) = \sum_ {t = 1} ^ {T} \frac {1}{2 \eta_ {t}} \left(\left\| \mathbf {x} ^ {\star} - \widehat {\mathbf {x}} _ {t} \right\| ^ {2} - \left\| \mathbf {x} ^ {\star} - \widehat {\mathbf {x}} _ {t + 1} \right\| ^ {2}\right)
$$
$\| \mathbf{x}\|_{U_t}\triangleq \sqrt{\mathbf{x}^\top U_t\mathbf{x}}$  refers to the matrix norm.
$$
\begin{array}{l} \leq \frac {\left\| \mathbf {x} ^ {\star} - \widehat {\mathbf {x}} _ {1} \right\| ^ {2}}{2 \eta_ {1}} + \sum_ {t = 2} ^ {T} \left(\frac {1}{2 \eta_ {t}} - \frac {1}{2 \eta_ {t - 1}}\right) \left\| \mathbf {x} ^ {\star} - \widehat {\mathbf {x}} _ {t} \right\| ^ {2} \\ \leq \frac {D}{2} (1 + \gamma D) + D ^ {2} \sum_ {t = 2} ^ {T} \left(\frac {1}{2 \eta_ {t}} - \frac {1}{2 \eta_ {t - 1}}\right) \leq \frac {D}{2} (1 + \gamma D) + \frac {D ^ {2}}{2 \eta_ {T}} \\ = \gamma D ^ {2} + \frac {D}{2} \sqrt {1 + \bar {V} _ {T}} + \mathcal {O} (1). \\ \end{array}
$$
Finally, we analyze the stability term,
$$
\begin{array}{l} \text {S T A B I L I T Y} = \sum_ {t = 1} ^ {T} \frac {1}{2 \eta_ {t}} \left(\| \widehat {\mathbf {x}} _ {t + 1} - \mathbf {x} _ {t} \| ^ {2} + \| \mathbf {x} _ {t} - \widehat {\mathbf {x}} _ {t} \| ^ {2}\right) \geq \sum_ {t = 2} ^ {T} \frac {1}{2 \eta_ {t}} \left(\| \widehat {\mathbf {x}} _ {t} - \mathbf {x} _ {t - 1} \| ^ {2} + \| \mathbf {x} _ {t} - \widehat {\mathbf {x}} _ {t} \| ^ {2}\right) \\ \geq \sum_ {t = 2} ^ {T} \frac {1}{4 \eta_ {t}} \| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \| ^ {2} \geq \frac {\gamma}{4} \sum_ {t = 2} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \| ^ {2}. \\ \end{array}
$$
Combining the above inequalities completes the proof.
Proof of Lemma 11. The proof mainly follows Theorem 15 of Chiang et al. [2012]. Denoting by  $\mathbf{x}^{\star}\in \arg \min_{\mathbf{x}\in \mathcal{X}}\sum_{t = 1}^{T}f_{t}(\mathbf{x})$ , it holds that
$$
\begin{array}{l} \sum_{t = 1}^{T}f_{t}(\mathbf{x}_{t}) - \sum_{t = 1}^{T}f_{t}(\mathbf{x}^{\star})\leq \underbrace{\sum_{t = 1}^{T}\|\nabla f_{t}(\mathbf{x}_{t}) - \mathbf{m}_{t}\|_{U_{t}^{-1}}^{2}}_{\text{ADAPTIVITY}} + \underbrace{\sum_{t = 1}^{T}(\mathcal{D}_{\psi_{t}}(\mathbf{x}^{\star},\widehat{\mathbf{x}}_{t}) - \mathcal{D}_{\psi_{t}}(\mathbf{x}^{\star},\widehat{\mathbf{x}}_{t + 1}))}_{\text{OPT-GAP}} \\ - \underbrace {\sum_ {t = 1} ^ {T} (\mathcal {D} _ {\psi_ {t}} (\widehat {\mathbf {x}} _ {t + 1} , \mathbf {x} _ {t}) + \mathcal {D} _ {\psi_ {t}} (\mathbf {x} _ {t} , \widehat {\mathbf {x}} _ {t}))} _ {\text {S T A B I L I T Y}} - \underbrace {\frac {\alpha}{2} \sum_ {t = 1} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \| _ {\nabla f _ {t} (\mathbf {x} _ {t}) \nabla f _ {t} (\mathbf {x} _ {t}) ^ {\top}} ^ {2}} _ {\text {N E G A T I V I T Y}}, \\ \end{array}
$$
where the last term is imported by the definition of exp-concavity. First, the optimality gap satisfies
$$
\begin{array}{l} \mathrm {O P T - G A P} = \frac {1}{2} \sum_ {t = 1} ^ {T} \| \mathbf {x} ^ {\star} - \widehat {\mathbf {x}} _ {t} \| _ {U _ {t}} ^ {2} - \frac {1}{2} \sum_ {t = 1} ^ {T} \| \mathbf {x} ^ {\star} - \widehat {\mathbf {x}} _ {t + 1} \| _ {U _ {t}} ^ {2} \\ \leq \frac {1}{2} \| \mathbf {x} ^ {\star} - \widehat {\mathbf {x}} _ {1} \| _ {V _ {1}} ^ {2} + \frac {1}{2} \sum_ {t = 1} ^ {T} \left(\| \mathbf {x} ^ {\star} - \widehat {\mathbf {x}} _ {t + 1} \| _ {U _ {t + 1}} ^ {2} - \| \mathbf {x} ^ {\star} - \widehat {\mathbf {x}} _ {t + 1} \| _ {U _ {t}} ^ {2}\right) \\ \leq \frac {1}{2} \gamma D ^ {2} + \frac {\alpha G ^ {2} D ^ {2}}{4} + \frac {\alpha}{4} \sum_ {t = 1} ^ {T} \| \mathbf {x} ^ {\star} - \widehat {\mathbf {x}} _ {t + 1} \| _ {\nabla f _ {t} (\mathbf {x} _ {t}) \nabla f _ {t} (\mathbf {x} _ {t}) ^ {\top}} ^ {2}. \\ \end{array}
$$
We handle the last term by leveraging the negative term imported by exp-concavity:
OPT-GAP - NEGATIVITY
$$
\begin{array}{l} \leq \frac {1}{2} \gamma D ^ {2} + \frac {\alpha}{4} \sum_ {t = 1} ^ {T} \| \mathbf {x} ^ {\star} - \widehat {\mathbf {x}} _ {t + 1} \| _ {\nabla f _ {t} (\mathbf {x} _ {t}) \nabla f _ {t} (\mathbf {x} _ {t}) ^ {\top}} ^ {2} - \frac {\alpha}{2} \sum_ {t = 1} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \| _ {\nabla f _ {t} (\mathbf {x} _ {t}) \nabla f _ {t} (\mathbf {x} _ {t}) ^ {\top}} ^ {2} + \mathcal {O} (1) \\ \leq \frac {1}{2} \gamma D ^ {2} + \frac {\alpha}{2} \sum_ {t = 1} ^ {T} \| \mathbf {x} _ {t} - \widehat {\mathbf {x}} _ {t + 1} \| _ {\nabla f _ {t} (\mathbf {x} _ {t}) \nabla f _ {t} (\mathbf {x} _ {t}) ^ {\top}} ^ {2} + \mathcal {O} (1), \\ \end{array}
$$
where the local norm of the second term above can be transformed into  $U_{t}$ :
$$
\frac {\alpha}{2} \sum_ {t = 1} ^ {T} \| \mathbf {x} _ {t} - \widehat {\mathbf {x}} _ {t + 1} \| _ {\nabla f _ {t} (\mathbf {x} _ {t}) \nabla f _ {t} (\mathbf {x} _ {t}) ^ {\top}} ^ {2} \leq \frac {\alpha G ^ {2}}{2} \sum_ {t = 1} ^ {T} \| \mathbf {x} _ {t} - \widehat {\mathbf {x}} _ {t + 1} \| ^ {2} \leq \sum_ {t = 1} ^ {T} \| \mathbf {x} _ {t} - \widehat {\mathbf {x}} _ {t + 1} \| _ {U _ {t}} ^ {2}.
$$
Using the stability of optimistic OMD [Chiang et al., 2012, Proposition 7], the above term can be further bounded by
$$
\sum_ {t = 1} ^ {T} \left\| \mathbf {x} _ {t} - \widehat {\mathbf {x}} _ {t + 1} \right\| _ {U _ {t}} ^ {2} \leq \sum_ {t = 1} ^ {T} \left\| \nabla f _ {t} (\mathbf {x} _ {t}) - \mathbf {m} _ {t} \right\| _ {U _ {t} ^ {- 1}} ^ {2}.
$$
By choosing the optimism as  $\mathbf{m}_t = \nabla f_{t - 1}(\mathbf{x}_{t - 1})$ , the above term can be consequently bounded due to Lemma 19 of Chiang et al. [2012]:
$$
\sum_ {t = 1} ^ {T} \| \nabla f _ {t} (\mathbf {x} _ {t}) - \nabla f _ {t - 1} (\mathbf {x} _ {t - 1}) \| _ {U _ {t} ^ {- 1}} ^ {2} \leq \frac {8 d}{\alpha} \ln \left(1 + \frac {\alpha}{8 \gamma d} \bar {V} _ {T}\right).
$$
The last step is to analyze the negative stability term:
$$
\begin{array}{l} \text {S T A B I L I T Y} = \sum_ {t = 1} ^ {T} \left(\mathcal {D} _ {\psi_ {t}} \left(\widehat {\mathbf {x}} _ {t + 1}, \mathbf {x} _ {t}\right) + \mathcal {D} _ {\psi_ {t}} \left(\mathbf {x} _ {t}, \widehat {\mathbf {x}} _ {t}\right)\right) = \frac {1}{2} \sum_ {t = 1} ^ {T} \| \widehat {\mathbf {x}} _ {t + 1} - \mathbf {x} _ {t} \| _ {U _ {t}} ^ {2} + \frac {1}{2} \sum_ {t = 1} ^ {T} \| \mathbf {x} _ {t} - \widehat {\mathbf {x}} _ {t} \| _ {U _ {t}} ^ {2} \\ \geq \frac {\gamma}{2} \sum_ {t = 1} ^ {T} \| \widehat {\mathbf {x}} _ {t + 1} - \mathbf {x} _ {t} \| ^ {2} + \frac {\gamma}{2} \sum_ {t = 1} ^ {T} \| \mathbf {x} _ {t} - \widehat {\mathbf {x}} _ {t} \| ^ {2} \geq \frac {\gamma}{4} \sum_ {t = 2} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \| ^ {2}. \\ \end{array}
$$
Combining existing results, we have
$$
\mathrm {R E G} _ {T} \leq \frac {1 6 d}{\alpha} \ln \left(1 + \frac {\alpha}{8 \gamma d} \bar {V} _ {T}\right) + \frac {1}{2} \gamma D ^ {2} - \frac {\gamma}{4} \sum_ {t = 2} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \| ^ {2} + \mathcal {O} (1),
$$
Proof of Lemma 12. The proof mainly follows Theorem 3 of Chen et al. [2023b]. Following the almost the same regret decomposition in Lemma 10, it holds that
$$
\sum_ {t = 1} ^ {T} f _ {t} (\mathbf {x} _ {t}) - \sum_ {t = 1} ^ {T} f _ {t} (\mathbf {x} ^ {\star}) \leq (\mathrm {E}. 5) - \underbrace {\frac {\lambda}{2} \sum_ {t = 1} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \| ^ {2}} _ {\text {N E G A T I V I T Y}}.
$$
First, we analyze the optimality gap,
$$
\begin{array}{l} \mathrm {O P T - G A P} \leq \frac {1}{\eta_ {1}} \mathcal {D} _ {\psi} (\mathbf {x} ^ {\star}, \widehat {\mathbf {x}} _ {1}) + \sum_ {t = 1} ^ {T} \left(\frac {1}{\eta_ {t + 1}} - \frac {1}{\eta_ {t}}\right) \mathcal {D} _ {\psi} (\mathbf {x} ^ {\star}, \widehat {\mathbf {x}} _ {t + 1}) \\ \leq \frac {1}{4} (\gamma + \lambda) D ^ {2} + \frac {\lambda}{4} \sum_ {t = 1} ^ {T} \| \mathbf {x} ^ {\star} - \widehat {\mathbf {x}} _ {t + 1} \| ^ {2}. \\ \end{array}
$$
We handle the last term by leveraging the negative term imported by strong convexity:
$$
\begin{array}{l} \text {O P T - G A P} - \text {N E G A T I V I T Y} \leq \frac {1}{4} (\gamma + \lambda) D ^ {2} + \frac {\lambda}{4} \sum_ {t = 1} ^ {T} \| \mathbf {x} ^ {\star} - \widehat {\mathbf {x}} _ {t + 1} \| ^ {2} - \frac {\lambda}{2} \sum_ {t = 1} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \| ^ {2} \\ \leq \frac {1}{4} \gamma D ^ {2} + \frac {\lambda}{2} \sum_ {t = 1} ^ {T} \| \mathbf {x} _ {t} - \widehat {\mathbf {x}} _ {t + 1} \| ^ {2} + \mathcal {O} (1). \\ \end{array}
$$
The second term above can be bounded by the stability of optimistic OMD:
$$
\frac {\lambda}{2} \sum_ {t = 1} ^ {T} \| \mathbf {x} _ {t} - \widehat {\mathbf {x}} _ {t + 1} \| ^ {2} \leq \frac {\lambda}{2} \sum_ {t = 1} ^ {T} \eta_ {t} ^ {2} \| \nabla f _ {t} (\mathbf {x} _ {t}) - \mathbf {m} _ {t} \| ^ {2} \leq \sum_ {t = 1} ^ {T} \eta_ {t} \| \nabla f _ {t} (\mathbf {x} _ {t}) - \mathbf {m} _ {t} \| ^ {2}.
$$
Finally, we lower-bound the stability term as
$$
\begin{array}{l} \text {S T A B I L I T Y} = \sum_ {t = 1} ^ {T} \frac {\gamma + \lambda t}{4} \left(\left\| \widehat {\mathbf {x}} _ {t + 1} - \mathbf {x} _ {t} \right\| ^ {2} + \left\| \mathbf {x} _ {t} - \widehat {\mathbf {x}} _ {t} \right\| ^ {2}\right) \quad \left(\text {b y} \eta_ {t} = \frac {2}{\gamma + \lambda t}\right) \\ \geq \frac {\gamma}{4} \sum_ {t = 2} ^ {T} (\| \widehat {\mathbf {x}} _ {t} - \mathbf {x} _ {t - 1} \| ^ {2} + \| \mathbf {x} _ {t} - \widehat {\mathbf {x}} _ {t} \| ^ {2}) \geq \frac {\gamma}{8} \sum_ {t = 2} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \| ^ {2}. \\ \end{array}
$$
Choosing the optimism as  $\mathbf{m}_t = \nabla f_{t - 1}(\mathbf{x}_{t - 1})$ , we have
$$
\mathrm {R E G} _ {T} \leq 2 \sum_ {t = 1} ^ {T} \eta_ {t} \| \nabla f _ {t} (\mathbf {x} _ {t}) - \nabla f _ {t - 1} (\mathbf {x} _ {t - 1}) \| ^ {2} + \frac {1}{4} \gamma D ^ {2} - \frac {\gamma}{8} \sum_ {t = 2} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \| ^ {2} + \mathcal {O} (1).
$$
To analyze the first term above, we follow the similar argument of Chen et al. [2023b]. By Lemma 9 with  $a_{t} = \| \nabla f_{t}(\mathbf{x}_{t}) - \nabla f_{t - 1}(\mathbf{x}_{t - 1})\|^{2}$ ,  $a_{\max} = 4G^2$ ,  $A = \lceil \lambda \bar{V}_T\rceil$ , and  $b = \lambda$ , it holds that
$$
\sum_ {t = 1} ^ {T} \frac {1}{\lambda t} \| \mathbf {g} _ {t} - \mathbf {g} _ {t - 1} \| ^ {2} \leq \frac {4 G ^ {2}}{\lambda} \ln (1 + \lambda \bar {V} _ {T}) + \frac {4 G ^ {2}}{\lambda} + \frac {1}{\lambda^ {2}}.
$$
Since  $\eta_t = 2 / (\gamma + \lambda t) \leq 2 / (\lambda t)$ , combining existing results, we have
$$
\operatorname {R E G} _ {T} \leq \frac {1 6 G ^ {2}}{\lambda} \ln (1 + \lambda \bar {V} _ {T}) + \frac {1}{4} \gamma D ^ {2} - \frac {\gamma}{8} \sum_ {t = 2} ^ {T} \| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \| ^ {2} + \mathcal {O} (1),
$$