SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\| \mathbb{E}\nabla_t - \nabla \widetilde{C}_{t,\delta}(M^t)\| \leq \varepsilon$ ,  $\|\widehat{\nabla}_t\| \leq \widehat{G}$ ,  $D_{\mathcal{M}} = \max_{M \in \mathcal{M}} \|M\|$ . Additionally, if  $\max_{M \in \mathcal{M}}\|\nabla \widetilde{C}_{t,\delta}(M)\| \leq G_{\mathcal{M}}$ , then we have for any  $\eta > 0$  that
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \widetilde {C} _ {t, \delta} (M ^ {t}) - \sum_ {t = 1} ^ {T} \widetilde {C} _ {t, \delta} (M ^ {*}) \right] \leq 2 \varepsilon D _ {\mathcal {M}} T \sqrt {h d _ {m i n}} + 2 \eta h ^ {2} d _ {\min} \widehat {G} ^ {2} T + \frac {2 h d _ {m i n} D _ {\mathcal {M}} ^ {2}}{\eta}
$$
Now, we invoke the regret upper bound from Lemma 16 to arrive at
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} c _ {t} (\mathbf {x} _ {t}, \mathbf {u} _ {t}) \right] - \inf  _ {M \in \mathcal {M}} \sum_ {t = 1} ^ {T} c _ {t} (\mathbf {x} _ {t} ^ {M}, \mathbf {u} _ {t} ^ {M}) \\ \leq 2 \varepsilon D _ {\mathcal {M}} T \sqrt {h d _ {m i n}} + 2 \eta h ^ {2} d _ {\mathrm {m i n}} \widehat {G} ^ {2} T + \frac {2 h d _ {m i n} D _ {\mathcal {M}} ^ {2}}{\eta} \\ + 2 G D _ {x, u} ^ {2} + \eta G D _ {x, u} W \kappa^ {3} h ^ {2} \widehat {G} T + 2 \delta h G D _ {x, u} \kappa^ {3} T \\ \end{array}
$$
Finally, we plug the value of  $\widehat{G}$  from Lemma 17, and  $\varepsilon$  from the first part of Lemma 14.
Lemma 17. The stochastic gradients produced by Algorithm 3 satisfy the following bound
$$
\| \widehat {\nabla_ {t}} \| \leq \widehat {G} := \frac {d _ {u} h ^ {2} W G D _ {x , u} ^ {2}}{\delta}
$$
As evident from the definition of  $\mathcal{M}$ ,  $D_{\mathcal{M}} = 2\sqrt{h}\kappa^4$ . Setting  $\eta = \sqrt{d_{min} / d_u} T^{-3 / 4}, \delta = \sqrt{d_u d_{min}} T^{-1 / 4}$  yields the result of  $O(T^{3 / 4})$  regret for (possibly) non-smooth costs.
For the second part of the claim, we show an improved analogue of the second part of Lemma 14.
Lemma 18. As long as  $c_t$  is  $L$ -smooth, for all  $M \in \mathcal{M}$ ,  $|\widetilde{C}_{t,\delta}(M) - \widetilde{C}_t(M)| \leq 25L\kappa^8 W^2 h^2\delta^2 /\alpha$ .
Using this, in a manner similar to the derivation for non-smooth costs, we arrive at
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} c _ {t} (\mathbf {x} _ {t}, \mathbf {u} _ {t}) \right] - \inf  _ {M \in \mathcal {M}} \sum_ {t = 1} ^ {T} c _ {t} (\mathbf {x} _ {t} ^ {M}, \mathbf {u} _ {t} ^ {M}) \\ \leq 2 \varepsilon D _ {\mathcal {M}} T \sqrt {h d _ {\min }} + 2 \eta h ^ {2} d _ {\min } \widehat {G} ^ {2} T + \frac {2 h d _ {\min } D _ {\mathcal {M}} ^ {2}}{\eta} \\ + 2 G D _ {x, u} ^ {2} + \eta G D _ {x, u} W \kappa^ {3} h ^ {2} \widehat {G} T + 5 0 L \kappa^ {8} W ^ {2} h ^ {2} \delta^ {2} T / \alpha \\ \end{array}
$$
In this case, we set  $\delta = (d_{u}d_{min})^{1 / 3}T^{-1 / 6}$ ,  $\eta = d_{min}^{1 / 3} / (d_u^{2 / 3}T^{2 / 3})$  to arrive at the final bound as stated in the claim.
E.4 Proof of Supporting Claims
Proof of Lemma 13. The properties follow from Lemma 6 in [Cassel and Koren, 2020], while using the fact that  $\delta = o(1)$ .
Proof of Lemma 14. Using the chain rule, we note that
$$
\nabla \widetilde {C} _ {t} (M) = \sum_ {i = 1} ^ {h} \left(\left. \nabla_ {\mathbf {u} _ {i}} F _ {t} (\mathbf {u} _ {1: h}) \right| _ {\mathbf {u} _ {k} = \sum_ {j = 0} ^ {h - 1} M _ {j} \mathbf {w} _ {t - h + k - j - 1} \forall k}\right) \otimes \mathbf {w} _ {t - h + i - 1: t - 2 h + i}
$$
Now, we note that the smoothed function  $F_{t,\delta}$  will satisfy
$$
\left| F _ {t, \delta} \left(\mathbf {u} _ {1: h}\right) - F _ {t} \left(\mathbf {u} _ {1: h}\right) \right| \leq \delta h G _ {F}, \quad \left| \widetilde {C} _ {t, \delta} (M) - \widetilde {C} _ {t} (M) \right| \leq \delta h G _ {F}
$$
where  $G_{F}$  is the Lipschitz constant of  $F_{t}$  with respect to a single  $u$ . This follows, by a hybrid-like argument smoothing one argument at a time using standard smoothing results (see e.g. [Gradu et al., 2020] Fact 3.2). We note that  $G_{F}$  can be bound by  $GD_{x,u}\kappa^3$ . Furthermore, this smoothing preserves convexity of  $F_{t,\delta}$  and composition of a linear and convex function is convex, so  $\widetilde{C}_{t,\delta}$  also remains convex.
The gradients of the smoothed function then has the following form due to Lemma 6.7 from Hazan et al. [2016].
$$
\begin{array}{l} \nabla_ {\mathbf {u} _ {i}} F _ {t, \delta} (\mathbf {u} _ {1: h}) = \mathbb {E} _ {\mathbf {n} _ {1: h} \sim \mathbb {S} _ {d _ {u}}} [ \frac {d _ {u}}{\delta} F _ {t} (\mathbf {u} _ {1: h} + \delta \mathbf {n} _ {1: h}) \mathbf {n} _ {i} ] \\ \nabla \widetilde {C} _ {t, \delta} (M) = \mathbb {E} _ {\mathbf {n} _ {1: h} \sim \mathbb {S} _ {d _ {u}}} \left[ \frac {d _ {u}}{\delta} \sum_ {i = 1} ^ {h} \left(\underbrace {F _ {t} (\mathbf {u} _ {1 : h} + \delta \mathbf {n} _ {1 : h}) | _ {\mathbf {u} _ {k} = \sum_ {j = 0} ^ {h - 1} M _ {j} \mathbf {w} _ {t - h + k - j - 1} \forall k}} _ {C _ {t} (M | \mathbf {n} _ {1: h})}\right) \mathbf {n} _ {i} \otimes \mathbf {w} _ {t - h + i - 1: t - 2 h + i} \right]. \\ \end{array}
$$
Rearranging, we have
$$
\nabla \widetilde {C} _ {t, \delta} (M ^ {t}) = \mathbb {E} _ {\mathbf {n} _ {t - h + 1: t} \sim \mathbb {S} _ {d _ {u}}} \left[ \frac {d _ {u} C _ {t} (M ^ {t} | \mathbf {n} _ {t - h + 1 : t})}{\delta} \sum_ {j = 0} ^ {h - 1} \mathbf {n} _ {t - i} \otimes \mathbf {w} _ {t - i - 1: t - h - i} \right]
$$
Now, to relate this to  $\mathbb{E}_{\mathbf{n}_{1:h} \sim \mathbb{S}_{d_u}}[\widehat{\nabla}_t]$ , we note in expression for  $\nabla \widetilde{C}_{t,\delta}(M^t)$ , we bound  $c_t(\mathbf{x}_t, \mathbf{u}_t | \mathbf{n}_{t-h+1:t}) - C_t(M^t | \mathbf{n}_{t-h+1:t})$  via Lemma 15. Using bounds on  $\mathbf{w}$ ,  $\mathbf{n}$  along with this bound, we have
$$
\| \nabla \widetilde {C} _ {t, \delta} (M ^ {t}) - \mathbb {E} [ \widehat {\nabla} _ {t} ] \| \leq \frac {d _ {u} h ^ {2} W}{\delta} \left(\frac {G D _ {x , u} ^ {2}}{T} + \eta G D _ {x, u} W \kappa^ {3} h ^ {2} \widehat {G}\right) \leq \frac {2 \eta d _ {u} h ^ {4} W ^ {2} \kappa^ {3} \widehat {G} G D _ {x , u}}{\delta}
$$
Proof of Lemma 15. We start with triangle inequality
$$
| c _ {t} (\mathbf {x} _ {t}, \mathbf {u} _ {t}) - \widetilde {C} _ {t} (M ^ {t}) | \leq | c _ {t} (\mathbf {x} _ {t}, \mathbf {u} _ {t}) - \widetilde {C} _ {t} (M ^ {t - h: t} | \mathbf {n} _ {t - h + 1: t}) | + | \widetilde {C} _ {t} (M ^ {t - h: t} | \mathbf {n} _ {t - h + 1: t}) - \widetilde {C} _ {t} (M ^ {t} | \mathbf {n} _ {t - h + 1: t}) |
$$
The first term is handled via Lemma 13, so we only need to bound the second term.
$$
\begin{array}{l} | \widetilde {C} _ {t} (M ^ {t - h: t}) - \widetilde {C} _ {t} (M ^ {t}) | = | c _ {t} (y _ {t} (M ^ {t - h: t}), \sum_ {i = 1} ^ {h} M _ {i} ^ {t} \mathbf {w} _ {t - i}) - c _ {t} (\tilde {y} _ {t} (M ^ {t}), \sum_ {i = 1} ^ {h} M _ {i} ^ {t} \mathbf {w} _ {t - i}) | \\ \leq G D _ {x, u} \left\| y _ {t} \left(M ^ {t - h: t}\right) - \tilde {y} _ {t} \left(M ^ {t}\right) \right\| \\ = G D _ {x, u} \| \sum_ {i = 1} ^ {2 h} \Psi_ {i} (M ^ {t - h: t}) \mathbf {w} _ {t - i} - \sum_ {i = 1} ^ {2 h} \Psi_ {i} (M ^ {t} \dots M ^ {t}) \mathbf {w} _ {t - i} \| \\ = G D _ {x, u} \| \sum_ {i = 1} ^ {2 h} \Psi_ {i} \left(M ^ {t - h: t} - \left(M ^ {t} \dots M ^ {t}\right)\right) \mathbf {w} _ {t - i} \| \\ \leq G D _ {x, u} \| \sum_ {i = 1} ^ {2 h} \Psi_ {i} \left(M ^ {t - h: t} - \left(M ^ {t} \dots M ^ {t}\right)\right) \mathbf {w} _ {t - i} \| \\ \end{array}
$$
Now we note that each matrix  $M_{i}^{s}$ , only occurs in one term of the form  $A^{k}BM_{i}^{s}\mathbf{w}_{l}$ , so we can refine the bound above to
$$
\begin{array}{l} \left| \widetilde {C} _ {t} \left(M ^ {t - h: t}\right) - \widetilde {C} _ {t} \left(M ^ {t}\right) \right| \leq G D _ {x, u} W \kappa^ {3} (1 - \alpha) \sum_ {i = 1} ^ {h} \| M ^ {t - i} - M ^ {t} \| \\ \leq G D _ {x, u} W \kappa^ {3} (1 - \alpha) \sum_ {i = 1} ^ {h} \sum_ {s = t - i} ^ {t} \| \eta \widehat {\nabla} _ {s - h} \| \\ \leq \eta G D _ {x, u} W \kappa^ {3} h ^ {2} \widehat {G}. \\ \end{array}
$$
Combining, we have
$$
\left| c _ {t} \left(\mathbf {x} _ {t}, \mathbf {u} _ {t}\right) - \widetilde {C} _ {t} \left(M ^ {t}\right) \right| \leq \frac {G D _ {x , u} ^ {2}}{T} + \eta G D _ {x, u} W \kappa^ {3} h ^ {2} \widehat {G}
$$
Proof of Lemma 16. Since  $c_t$  is convex, so is  $\widetilde{C}_{t,\delta}$ . Using this fact and the observation that  $M^t$  is independent of  $\mathbf{n}_{t:t - h}$  used to construct  $\widehat{\nabla}_t$  due to the delayed update of gradients, we have
$$
\begin{array}{l} \widetilde {C} _ {t, \delta} (M ^ {t}) - \widetilde {C} _ {t, \delta} (M ^ {*}) \\ \leq \langle \nabla \widetilde {C} _ {t, \delta} (M ^ {t}), M ^ {t} - M ^ {*} \rangle \\ \leq \mathbb {E} \langle \widehat {\nabla} _ {t}, M ^ {t} - M ^ {*} \rangle + 2 \varepsilon D _ {\mathcal {M}} \sqrt {h d _ {m i n}} \\ \leq \mathbb {E} \langle \widehat {\nabla} _ {t}, M ^ {t + h} - M ^ {*} \rangle + \| \widehat {\nabla} _ {t} \| _ {F} \| M ^ {t + h} - M ^ {t} \| _ {F} + 2 \varepsilon D _ {\mathcal {M}} \sqrt {h d _ {m i n}} \\ \leq \mathbb {E} \langle \widehat {\nabla} _ {t}, M ^ {t + h} - M ^ {*} \rangle + \eta h ^ {2} \widehat {G} ^ {2} d _ {m i n} + 2 \varepsilon D _ {\mathcal {M}} \sqrt {h d _ {m i n}} \\ \end{array}
$$
The gradient update can be rewritten as
$$
\begin{array}{l} \langle \widehat {\nabla} _ {t}, M ^ {t + h} - M ^ {*} \rangle \leq \frac {\| M ^ {t + h} - M ^ {*} \| _ {F} ^ {2} - \| M ^ {t + h} - \eta \widehat {\nabla} _ {t} - M ^ {*} \| _ {F} ^ {2}}{2 \eta} + \frac {\eta \widehat {G} ^ {2} h d _ {m i n}}{2} \\ \leq \frac {\| M ^ {t + h} - M ^ {*} \| _ {F} ^ {2} - \| M ^ {t + h + 1} - M ^ {*} \| _ {F} ^ {2}}{2 \eta} + \frac {\eta \widehat {G} ^ {2} h d _ {m i n}}{2}, \\ \end{array}
$$
where we use the fact that the projection operator is non-expansive, hence  $M^{t + h + 1}$  is closer in Euclidean distance to  $M^*$  than  $M^{t + h} - \eta \widehat{\nabla}_t$ . Telescoping this, we have for any  $M^*$  that
Proof of Lemma 17. Plugging in line 6 from Algorithm 3 and using our bounds on the cost, we have
$$
\| \widehat {\nabla} _ {t} \| \leq \frac {d _ {u} G D _ {x , u} ^ {2}}{\delta} \sum_ {j = 0} ^ {h - 1} \| \mathbf {n} _ {t - i} \| \| \mathbf {w} _ {t - i - 1: t - h - i} \| \leq \frac {d _ {u} h ^ {2} W G D _ {x , u} ^ {2}}{\delta}.
$$
Proof of Lemma 18. We first make note of the following characterization of idealized costs under smoothness due to Cassel and Koren [2020] (Lemma 7.2, therein).
Lemma 19 (Cassel and Koren [2020]). If  $c_t$  is  $L$ -smooth, then the smoothed and non-smoothed variants of the idealized costs  $\widetilde{C}_t, F_t, \widetilde{C}_{t,\delta}, F_{t,\delta}$  are  $L'$ -smooth, where  $L' = 25L\kappa^8 W^2 h / \alpha$ .
Note that  $\widetilde{C}_t, \widetilde{C}_{t,\delta}$  only differ in that the latter is a noise-smoothed version of the former. Let  $\mathbf{u}_{1:h} = \left[\sum_{i=0}^{h-1} M_i \mathbf{w}_{t-i-h}, \sum_{i=0}^{h-1} M_i \mathbf{w}_{t-i-h+1} \cdots \sum_{i=0}^{h-1} M_i \mathbf{w}_{t-i-1}\right]$ . Using the fact the noise  $\mathbf{n}_{1:h}$  is zero-mean and independent of  $\mathbf{u}_{1:h}$ , we create a second-order expansion using Taylor's theorem to conclude
$$
\begin{array}{l} \left| \widetilde {C} _ {t, \delta} (M) - \widetilde {C} _ {t} (M) \right| \\ = \left| \mathbb {E} _ {\mathbf {u} _ {1: h} \sim \mathbb {S} _ {d _ {u}}} F _ {t} \left(\mathbf {u} _ {1: h} + \delta \mathbf {n} _ {1: h}\right) - F _ {t} \left(\mathbf {u} _ {1: h}\right) \right| \\ \leq | \underbrace {\mathbb {E} _ {\mathbf {n} _ {1 : h} \sim \mathbb {S} _ {d _ {u}}} \left\langle \nabla F _ {t} (\mathbf {u} _ {1 : h}) , \delta \mathbf {n} _ {1 : h} \right\rangle} _ {= 0} | + \frac {L ^ {\prime}}{2} \| \delta \mathbf {n} _ {1: h} \| _ {F} ^ {2} \\ \leq L ^ {\prime} \delta^ {2} h. \\ \end{array}
$$