SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbf {B o n u s} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}}, \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \right\rangle \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}}, \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \right\rangle \right] \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}}, \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \right\rangle \mid \mathcal {E} _ {t - 1} \right] \mathbb {I} \{\mathcal {E} _ {t - 1} \} \right] + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}}, \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 2} \right\rangle \mid \overline {{\mathcal {E} _ {t - 1}}} \right] \mathbb {I} \{\overline {{\mathcal {E} _ {t - 1}}} \} \right] \\ \leq 2 (d + 2) \sum_ {t = 1} ^ {T} \alpha_ {t} - \frac {(1 - \delta)}{4} \sum_ {t = 1} ^ {T} \alpha_ {t} \| u - x _ {t} \| _ {\hat {\Sigma} _ {t} ^ {- 1}} ^ {2} + \mathcal {O} \left(\sum_ {t = 1} ^ {T} \frac {d ^ {3} \alpha_ {t} \log (T / \delta)}{t} + \delta T \sum_ {t = 1} ^ {T} \alpha_ {t}\right) \\ \leq 2 (d + 2) \sum_ {t = 1} ^ {T} \alpha_ {t} - \frac {1}{4} \sum_ {t = 1} ^ {T} \alpha_ {t} \| u - x _ {t} \| _ {\hat {\Sigma} _ {t} ^ {- 1}} ^ {2} + \mathcal {O} \left(\sum_ {t = 1} ^ {T} \frac {d ^ {3} \alpha_ {t} \log (T / \delta)}{t} + \delta T \sum_ {t = 1} ^ {T} \alpha_ {t}\right) \\ \end{array}
$$
D.3 Bounding the Penalty term
Lemma 24.  $\overline{U}^{A_0}$ , we have
$$
\frac {F \left(\overline {{\boldsymbol {U}}} ^ {\mathcal {A} _ {0}}\right) - \min  _ {\boldsymbol {H} \in \mathcal {H} ^ {\mathcal {A} _ {0}}} F (\boldsymbol {H})}{\eta_ {T}} \leq \frac {2 d \log (T)}{\eta_ {T}}
$$
Proof. Since  $\overline{\boldsymbol{U}}^{\mathcal{A}_0} = \left(1 - \frac{1}{T^2}\right)\boldsymbol{U}^{\mathcal{A}_0} + \frac{1}{T^2}\boldsymbol{H}_*^{\mathcal{A}_0}$ , we have  $\overline{\boldsymbol{U}}^{\mathcal{A}_0} \succeq \frac{1}{T^2}\boldsymbol{H}_*^{\mathcal{A}_0}$ . Then
$$
\frac {F \left(\overline {{\boldsymbol {U}}} ^ {\mathcal {A} _ {0}}\right) - \min  _ {\boldsymbol {H} \in \mathcal {H} ^ {\mathcal {A} _ {0}}} F (\boldsymbol {H})}{\eta_ {T}} = \frac {1}{\eta_ {T}} \log \frac {\det  \left(\boldsymbol {H} _ {*} ^ {\mathcal {A} _ {0}}\right)}{\det  \left(\overline {{\boldsymbol {U}}} ^ {\mathcal {A} _ {0}}\right)} \leq \frac {2 d \log (T)}{\eta_ {T}}.
$$
D.4 Bounding the Stability-1 term
Zimmert and Lattimore [2022] gave a useful identity to bound the Bregman divergence. We restate it in Lemma 25 for completeness.
Lemma 25. Let  $\pmb{G} = \begin{bmatrix} G + gg^{\top} & g \\ g^{\top} & 1 \end{bmatrix}$  and  $\pmb{H} = \begin{bmatrix} H + hh^{\top} & h \\ h^{\top} & 1 \end{bmatrix}$ , we have
$$
D (\boldsymbol {G}, \boldsymbol {H}) = D (G, H) + \| g - h \| _ {H ^ {- 1}} ^ {2} \geq \| g - h \| _ {H ^ {- 1}} ^ {2}
$$
$$
\begin{array}{l} D (\boldsymbol {G}, \boldsymbol {H}) = F (\boldsymbol {G}) - F (\boldsymbol {H}) - \langle \nabla F (\boldsymbol {H}), \boldsymbol {G} - \boldsymbol {H} \rangle \\ = \log \left(\frac {\det  (\boldsymbol {H})}{\det  (\boldsymbol {G})}\right) + \operatorname {T r} \left(\boldsymbol {H} ^ {- 1} (\boldsymbol {G} - \boldsymbol {H})\right) \\ = \log \left(\frac {\det  (\boldsymbol {H})}{\det  (\boldsymbol {G})}\right) + \operatorname {T r} (\boldsymbol {H} ^ {- 1} \boldsymbol {G}) - d - 1 \\ = \log \left(\frac {\operatorname {d e t} (\boldsymbol {H})}{\operatorname {d e t} (\boldsymbol {G})}\right) + \operatorname {T r} (\boldsymbol {H} ^ {- 1} \boldsymbol {G}) - d - 1 \\ = \log \left(\frac {\det (H)}{\det (G)}\right) + \operatorname {T r} \left(H ^ {- 1} G\right) + \| g - h \| _ {H ^ {- 1}} ^ {2} - d \tag {Lemma 21} \\ = D (G, H) + \| g - h \| _ {H ^ {- 1}} ^ {2} \\ \geq \| g - h \| _ {H ^ {- 1}} ^ {2} \\ \end{array}
$$
Lemma 26. For any  $\pmb{H} \in \mathcal{H}^{\mathcal{A}_0}$ , we have
$$
\mathbf {S t a b i l i t y - 1} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {H}, \hat {\gamma} _ {t} \right\rangle - \frac {D \left(\boldsymbol {H} , \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}}\right)}{2 \eta_ {t}} \right] \leq 2 d \sum_ {t = 1} ^ {T} \eta_ {t} + \mathcal {O} (\delta T ^ {2})
$$
Proof. Recall that  $\mathbf{H}_t^{A_0} = \widehat{\mathrm{Cov}}(p_t^{A_0})$  and  $\widehat{\mathrm{Cov}}(p) = \begin{bmatrix} \mathrm{Cov}(p) + \mu(p)\mu(p)^\top & \mu(p) \\ \mu(p)^\top & 1 \end{bmatrix}$ , we have
$$
\begin{array}{l} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {H}, \hat {\gamma} _ {t} \right\rangle - \frac {D \left(\boldsymbol {H} , \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}}\right)}{2 \eta_ {t}} \leq \left\langle x _ {t} ^ {\mathcal {A} _ {0}} - \mu (p), \hat {y} _ {t} \right\rangle - \frac {\| \mu (p) - x _ {t} ^ {\mathcal {A} _ {0}} \| _ {\operatorname {C o v} \left(p _ {t} ^ {\mathcal {A} _ {0}}\right) ^ {- 1}} ^ {2}}{2 \eta_ {t}} \tag {Lemma 25} \\ \leq \left\| x _ {t} ^ {\mathcal {A} _ {0}} - \mu (p) \right\| _ {\operatorname {C o v} \left(p _ {t} ^ {\mathcal {A} _ {0}}\right) ^ {- 1}} \left\| \hat {y} _ {t} \right\| _ {\operatorname {C o v} \left(p _ {t} ^ {\mathcal {A} _ {0}}\right)} - \frac {\left\| \mu (p) - x _ {t} ^ {\mathcal {A} _ {0}} \right\| _ {\operatorname {C o v} \left(p _ {t} ^ {\mathcal {A} _ {0}}\right) ^ {- 1}} ^ {2}}{2 \eta_ {t}} \\ \leq \frac {\eta_ {t}}{2} \| \hat {y} _ {t} \| _ {\operatorname {C o v} \left(p _ {t} ^ {A _ {0}}\right)} ^ {2} \quad (\text {A M - G M i n e q u a l i t y}) \\ = \frac {\eta_ {t}}{2} \| \hat {\Sigma} _ {t} ^ {- 1} (a _ {t} - \hat {x} _ {t}) \ell_ {t} \| _ {\operatorname {C o v} \left(p _ {t} ^ {A _ {0}}\right)} ^ {2} \\ \leq \frac {\eta_ {t}}{2} \left(a _ {t} - \hat {x} _ {t}\right) ^ {\top} \hat {\Sigma} _ {t} ^ {- 1} \operatorname {C o v} \left(p _ {t} ^ {\mathcal {A} _ {0}}\right) \hat {\Sigma} _ {t} ^ {- 1} \left(a _ {t} - \hat {x} _ {t}\right) \quad \left(\left| \ell_ {t} \right| \leq 1\right) \\ = \frac {\eta_ {t}}{2} \operatorname {T r} \left(\left(a _ {t} - \hat {x} _ {t}\right) \left(a _ {t} - \hat {x} _ {t}\right) ^ {\top} \hat {\Sigma} _ {t} ^ {- 1} \operatorname {C o v} \left(p _ {t} ^ {\mathcal {A} _ {0}}\right) \hat {\Sigma} _ {t} ^ {- 1}\right) \\ \end{array}
$$
Since  $\mathbb{E}_{\mathcal{A}\sim \mathcal{D}}\mathbb{E}_{a\sim p^{\mathcal{A}}}\left[(a - \hat{x}_t)(a - \hat{x}_t)^\top \right] = H_t$ , taking expectations over  $\mathcal{A}_t$ ,  $a_{t}$  and  $\mathcal{A}_0$  conditioned on  $\mathcal{E}_{t - 1}$ , we have
$$
\begin{array}{l} \mathbb {E} _ {t} ^ {\mathcal {E}} \left[ \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {H}, \hat {\gamma} _ {t} \right\rangle - \frac {D (\boldsymbol {H} , \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})}{2 \eta_ {t}} \right] \leq \mathbb {E} _ {t} ^ {\mathcal {E}} \left[ \frac {\eta_ {t}}{2} \operatorname {T r} \left((a _ {t} - \hat {x} _ {t}) (a _ {t} - \hat {x} _ {t}) ^ {\top} \hat {\Sigma} _ {t} ^ {- 1} \operatorname {C o v} (p _ {t} ^ {\mathcal {A} _ {0}}) \hat {\Sigma} _ {t} ^ {- 1}\right) \right] \\ = \mathbb {E} _ {t} ^ {\mathcal {E}} \left[ \frac {\eta_ {t}}{2} \operatorname {T r} \left(H _ {t} \hat {\Sigma} _ {t} ^ {- 1} \mathbb {E} _ {\mathcal {A} _ {0} \sim D} \left[ \operatorname {C o v} \left(p _ {t} ^ {\mathcal {A} _ {0}}\right) \right] \hat {\Sigma} _ {t} ^ {- 1}\right) \right]. \\ \end{array}
$$
Notice that given  $\mathcal{E}_{t - 1}$
$$
\hat {\Sigma} _ {t} \succeq \frac {1}{2} H _ {t} = \frac {1}{2} \mathbb {E} _ {\mathcal {A} \sim D} [ \operatorname {C o v} (p _ {t} ^ {\mathcal {A}}) ] + \frac {1}{2} (\hat {x} _ {t} - x _ {t}) (\hat {x} _ {t} - x _ {t}) ^ {\top} \succeq \frac {1}{2} \mathbb {E} _ {\mathcal {A} \sim D} [ \operatorname {C o v} (p _ {t} ^ {\mathcal {A}}) ]
$$
Hence we continue to upper bound the last expression by
$$
\left. \right. \mathbb {E} _ {t} ^ {\mathcal {E}} \left[ \eta_ {t} \operatorname {T r} \left(H _ {t} \hat {\Sigma} _ {t} ^ {- 1} \hat {\Sigma} _ {t} \hat {\Sigma} _ {t} ^ {- 1}\right)\right] \leq \mathbb {E} _ {t} ^ {\mathcal {E}} \left[ \eta_ {t} \operatorname {T r} \left(H _ {t} \hat {\Sigma} _ {t} ^ {- 1}\right)\right] \leq 2 \eta_ {t} d.
$$
On the other hand, since  $\hat{\Sigma}_t\succeq \frac{1}{t} I\succeq \frac{1}{T} I$ , we have trivial bound
$$
\mathbb {E} _ {t} \left[ \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {H}, \hat {\gamma} _ {t} \right\rangle - \frac {D (\boldsymbol {H} , \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})}{2 \eta_ {t}} \mid \overline {{\mathcal {E} _ {t - 1}}} \right] \leq \mathcal {O} (T)
$$
Combining everything, we get
$$
\begin{array}{l} \mathbf {S t a b i l i t y - 1} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {H}, \hat {\gamma} _ {t} \right\rangle - \frac {D (\boldsymbol {H} , \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})}{2 \eta_ {t}} \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {H}, \hat {\gamma} _ {t} \right\rangle - \frac {D (\boldsymbol {H} , \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})}{2 \eta_ {t}} \right] \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {H}, \hat {\gamma} _ {t} \right\rangle - \frac {D \left(\boldsymbol {H} , \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}}\right)}{2 \eta_ {t}} \mid \mathcal {E} _ {t - 1} \right] \mathbb {I} \{\mathcal {E} _ {t - 1} \} \right] \\ + \mathbb {E} \left[ \right. \sum_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ \right.\left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {H}, \hat {\gamma} _ {t} \right\rangle - \frac {D (\boldsymbol {H} , \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})}{2 \eta_ {t}} \left. \right| \overline {{\mathcal {E} _ {t - 1}}} \left. \right] \mathbb {I} \{\overline {{\mathcal {E} _ {t - 1}}} \} \left. \right] \\ \leq 2 d \sum_ {t = 1} ^ {T} \eta_ {t} + \mathcal {O} (\delta T ^ {2}). \\ \end{array}
$$
D.5 Bounding the Stability-2 term
Note that Lemma 8 does not require matrix  $A, B$  to be positive semi-definite. We will use it to prove the following lemma based on Lemma 34 in Dann et al. [2023b].
Lemma 27. If  $\eta_t\alpha_t \leq \frac{1}{64t}$ , then
$$
\mathbf {S t a b i l i t y - 2} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \max _ {\boldsymbol {H} \in \mathcal {H} ^ {\mathcal {A} _ {0}}} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {H}, - \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \right\rangle - \frac {D (\boldsymbol {H} , \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})}{2 \eta_ {t}} \right] \leq d \sum_ {t = 1} ^ {T} \alpha_ {t} + \mathcal {O} \left(\delta T ^ {2}\right)
$$
Proof. We first show that  $\max_{\pmb{H} \in \mathcal{H}^{\mathcal{A}_0}} \left\langle \pmb{H}_t^{\mathcal{A}_0} - \pmb{H}, -\alpha_t \hat{\pmb{\Sigma}}_t^{-1} \right\rangle - \frac{D(\pmb{H}, \pmb{H}_t^{\mathcal{A}_0})}{2\eta_t} \leq \frac{\alpha_t}{2} \| \hat{\pmb{\Sigma}}_t^{-1} \|_{\nabla^{-2}F(\pmb{H}_t^{\mathcal{A}_0})}$ .
$$
G (\pmb {H}) = \left\langle \pmb {H} _ {t} ^ {\mathcal {A} _ {0}} - \pmb {H}, - \alpha_ {t} \hat {\pmb {\Sigma}} _ {t} ^ {- 1} \right\rangle - \frac {D (\pmb {H} , \pmb {H} _ {t} ^ {\mathcal {A} _ {0}})}{2 \eta_ {t}}
$$
and  $\lambda = \| \alpha_{t}\hat{\Sigma}_{t}^{-1}\|_{\nabla^{-2}F(\pmb{H}_{t}^{\mathcal{A}_{0}})}$ . Since  $\hat{\Sigma}_t\succeq \frac{1}{t} I,\pmb {H}_t^{\mathcal{A}_0}\preceq 2I,\eta_t\alpha_t\leq \frac{1}{64t}$ , we have
$$
\eta_ {t} \lambda = \eta_ {t} \| \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \| _ {\nabla^ {- 2} F (\boldsymbol {H} _ {t} ^ {A _ {0}})} = \eta_ {t} \alpha_ {t} \sqrt {\operatorname {T r} (\boldsymbol {H} _ {t} ^ {A _ {0}} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \boldsymbol {H} _ {t} ^ {A _ {0}} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1})} \leq 2 \eta_ {t} \alpha_ {t} t \leq \frac {1}{3 2}.
$$
Let  $\pmb{H}^{\prime}$  be the maximizer of  $G$ . Since  $G(\pmb{H}_t^{\mathcal{A}_0}) = 0$ , we have  $G(\pmb{H}^{\prime}) \geq 0$ . It suffices to show  $\| \pmb{H}^{\prime} - \pmb{H}_t^{\mathcal{A}_0} \|_{\nabla^2 F(\pmb{H}_t^{\mathcal{A}_0})} \leq 16 \eta_t \lambda$  because from Lemma 8, it leads to
$$
G (\boldsymbol {H} ^ {\prime}) \leq \| \boldsymbol {H} _ {t} ^ {A _ {0}} - \boldsymbol {H} ^ {\prime} \| _ {\nabla^ {2} F (\boldsymbol {H} _ {t} ^ {A _ {0}})} \| \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \| _ {\nabla^ {- 2} F (\boldsymbol {H} _ {t} ^ {A _ {0}})} \leq 1 6 \eta_ {t} \lambda \alpha_ {t} \| \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \| _ {\nabla^ {- 2} F (\boldsymbol {H} _ {t} ^ {A _ {0}})} = \frac {\alpha_ {t}}{2} \| \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \| _ {\nabla^ {- 2} F (\boldsymbol {H} _ {t} ^ {A _ {0}})}
$$
To show  $\| H' - H_{t}^{A_{0}} \|_{\nabla^{2}F(H_{t}^{A_{0}})} \leq 16\eta_{t}\lambda$ , it suffices to show that for all  $U$  such that  $\| U - H_{t}^{A_{0}} \|_{\nabla^{2}F(H_{t}^{A_{0}})} = 16\eta_{t}\lambda$ ,  $G(U) \leq 0$ . This is because given this condition, if  $\| H' - H_{t}^{A_{0}} \|_{\nabla^{2}F(H_{t}^{A_{0}})} > 16\eta_{t}\lambda$ , then there is a  $U$  in the line segment between  $H_{t}^{A_{0}}$  and  $H'$  such that  $\| U - H_{t}^{A_{0}} \|_{\nabla^{2}F(H_{t}^{A_{0}})} = 16\eta_{t}\lambda$ . From the condition,  $G(U) \leq 0 \leq \min \{G(H_{t}^{A_{0}}), G(H')\}$  which contradicts to the strictly concave of  $G$ .
Now consider any  $U$  such that  $\| U - H_t^{\mathcal{A}_0}\|_{\nabla^2 F(H_t^{\mathcal{A}_0})} = 16\eta_t\lambda$ . By Taylor expansion, there exists  $U'$  in the line segment between  $U$  and  $H_t^{\mathcal{A}_0}$  such that
$$
G (\boldsymbol {U}) \leq \| \boldsymbol {U} - \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} \| _ {\nabla^ {2} F (\boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})} \| \alpha_ {t} \hat {\Sigma} _ {t} ^ {- 1} \| _ {\nabla^ {- 2} F (\boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})} - \frac {1}{4 \eta_ {t}} \| \boldsymbol {U} - \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} \| _ {\nabla^ {2} F (\boldsymbol {U} ^ {\prime})} ^ {2}
$$
We have  $\| \pmb{U}^{\prime} - \pmb{H}_{t}^{A_{0}}\|_{\nabla^{2}F(\pmb{H}_{t}^{A_{0}})} \leq \| \pmb{U} - \pmb{H}_{t}^{A_{0}}\|_{\nabla^{2}F(\pmb{H}_{t}^{A_{0}})} = 16\eta_{t}\lambda \leq \frac{1}{2}$ . From the Equation 2.2 in page 23 of Nemirovski [2004] (also appear in Eq.(5) of Abernethy et al. [2009]) and log det is a self-concordant function, we have  $\| \pmb{U} - \pmb{H}_{t}^{A_{0}}\|_{\nabla^{2}F(\pmb{U}^{\prime})}^{2} \geq \frac{1}{4}\| \pmb{U} - \pmb{H}_{t}^{A_{0}}\|_{\nabla^{2}F(\pmb{H}_{t}^{A_{0}})}^{2}$ . Thus, we have
$$
G (\boldsymbol {U}) \leq \| \boldsymbol {U} - \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} \| _ {\nabla^ {2} F (\boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})} \| \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \| _ {\nabla^ {- 2} F (\boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})} - \frac {1}{1 6 \eta_ {t}} \| \boldsymbol {U} - \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} \| _ {(\boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}}) ^ {- 1}} ^ {2} = 1 6 \eta_ {t} \lambda^ {2} - \frac {(1 6 \eta_ {t} \lambda) ^ {2}}{1 6 \eta_ {t}} = 0
$$
We have  $\| \hat{\Sigma}_t^{-1} \|_{\nabla^{-2} F(\mathbf{H}_t^{\mathcal{A}_0})} = \sqrt{\operatorname{Tr}(\mathbf{H}_t^{\mathcal{A}_0} \hat{\Sigma}_t^{-1} \mathbf{H}_t^{\mathcal{A}_0} \hat{\Sigma}_t^{-1})} = \sqrt{\operatorname{Tr}((\mathbf{H}_t^{\mathcal{A}_0} \hat{\Sigma}_t^{-1})^2)}$ . Observe the following two facts: 1) all eigenvalues of  $\mathbf{H}_t^{\mathcal{A}_0} \hat{\Sigma}_t^{-1}$  are non-negative since  $\mathbf{H}_t^{\mathcal{A}_0}$  and  $\hat{\Sigma}_t^{-1}$  are both positive semi-definite, 2) for a square matrix  $A$  with all non-negative eigenvalues,  $\operatorname{Tr}(A^2) \leq \operatorname{Tr}(A)^2$  because  $\operatorname{Tr}(A^2) = \sum_i \lambda_i(A^2) = \sum_i \lambda_i(A)^2 \leq (\sum_i \lambda_i(A))^2$ . We have
$$
\sqrt {\operatorname {T r} \left(\left(\boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1}\right) ^ {2}\right)} \leq \operatorname {T r} \left(\boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1}\right).
$$
This allows us to conclude
$$
\mathbb {E} _ {t} ^ {\mathcal {E}} \left[ \frac {\alpha_ {t}}{2} \| \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \| _ {\nabla^ {- 2} F (\boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})} \right] \leq \frac {\alpha_ {t}}{2} \mathbb {E} _ {t} ^ {\mathcal {E}} \left[ \operatorname {T r} \left(\boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1}\right) \right] \leq \alpha_ {t} d
$$
where we use that  $\hat{\Sigma}_t\succeq \frac{1}{2}\mathbb{E}_{\mathcal{A}_0\sim D}[H_t^{A_0}]$  given  $\mathcal{E}_{t - 1}$
On the other hand, since  $\hat{\Sigma}_t\succeq \frac{1}{t}\pmb {I}\succeq \frac{1}{T}\pmb {I},$  for any  $t = 1,\dots ,T$  , we have trivial bound
$$
\mathbb {E} _ {t} \left[ \right. \max  _ {\boldsymbol {H} \in \mathcal {H} ^ {\mathcal {A} _ {0}}} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {H}, - \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \right\rangle - \frac {D (\boldsymbol {H} , \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})}{2 \eta_ {t}} \left| \right. \overline {{\mathcal {E} _ {t - 1}}} \left. \right] \leq \mathcal {O} (T)
$$
Overall,
$$
\begin{array}{l} \mathbf {S t a b i l i t y - 2} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \max  _ {\boldsymbol {H} \in \mathcal {H} ^ {\mathcal {A} _ {0}}} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {H}, - \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \right\rangle - \frac {D (\boldsymbol {H} , \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})}{2 \eta_ {t}} \right] \\ \leq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ \max  _ {\boldsymbol {H} \in \mathcal {H} ^ {\mathcal {A} _ {0}}} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {H}, - \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \right\rangle - \frac {D (\boldsymbol {H} , \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})}{2 \eta_ {t}} \right] \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ \max  _ {\boldsymbol {H} \in \mathcal {H} ^ {\mathcal {A} _ {0}}} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {H}, \hat {\gamma} _ {t} \right\rangle - \frac {D (\boldsymbol {H} , \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})}{2 \eta_ {t}} \mid \mathcal {E} _ {t - 1} \right] \mathbb {I} \{\mathcal {E} _ {t - 1} \} \right] \\ + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ \max _ {\boldsymbol {H} \in \mathcal {H} ^ {\mathcal {A} _ {0}}} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {H}, \hat {\gamma} _ {t} \right\rangle - \frac {D (\boldsymbol {H} , \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}})}{2 \eta_ {t}} \middle | \overline {{\mathcal {E} _ {t - 1}}} \right] \mathbb {I} \{\overline {{\mathcal {E} _ {t - 1}}} \} \right] \\ \leq d \sum_ {t = 1} ^ {T} \alpha_ {t} + \mathcal {O} (\delta T ^ {2}). \\ \end{array}
$$
D.6 Bounding the Error term
Lemma 28.
$$
\mathbf {E r r o r} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \overline {{\boldsymbol {U}}} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}}, \hat {\gamma} _ {t} - \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \right\rangle \right] \leq \mathcal {O} (1).
$$
Proof. Since  $\overline{\boldsymbol{U}}^{\mathcal{A}_0} = \left(1 - \frac{1}{T^2}\right)\boldsymbol{U}^{\mathcal{A}_0} + \frac{1}{T^2}\boldsymbol{H}_*\mathcal{A}_0$ , and  $\hat{\Sigma}_t\succeq \frac{1}{T} I,\hat{\Sigma}_t\succeq \frac{1}{T} I$  we have
$$
\begin{array}{l} \mathbf {E r r o r} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \bar {\boldsymbol {U}} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}}, \hat {\gamma} _ {t} - \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \right\rangle \right] \\ = \mathbb {E} \left[ \frac {1}{T ^ {2}} \sum_ {t = 1} ^ {T} \left\langle - \boldsymbol {U} ^ {\mathcal {A} _ {0}} + \boldsymbol {H} _ {*} ^ {\mathcal {A} _ {0}}, \hat {\gamma} _ {t} - \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \right\rangle \right] \\ \leq \mathcal {O} (1). \\ \end{array}
$$
D.7 Finishing up
Recall the regret decomposition at the beginning of Appendix D. From Lemma 24, Lemma 26, Lemma 27, and Lemma 28, we have
$$
\begin{array}{l} \mathrm {F T R L - R e g} = \text {P e n a l t y} + \text {S t a b i l i t y - 1} + \text {S t a b i l i t y - 2} + \text {E r r o r} \\ \leq \mathcal {O} \left(\frac {d \log (T)}{\eta_ {T}} + d \sum_ {t = 1} ^ {T} \eta_ {t} + d \sum_ {t = 1} ^ {T} \alpha_ {t} + \delta T ^ {2}\right) \\ \end{array}
$$
From Lemma 20 and Lemma 23, we can cancel out the additional regret induced by bias through the well-designed bonus term. Namely,
$$
\begin{array}{l} \mathbf {B i a s} + \mathbf {B o u n s} = \frac {1}{4} \sum_ {t = 1} ^ {T} \alpha_ {t} \| x _ {t} - u \| _ {\hat {\Sigma} _ {t} ^ {- 1}} ^ {2} + \mathcal {O} \left(\sum_ {t = 1} ^ {T} \frac {d ^ {3} \log (T / \delta)}{\alpha_ {t} t} + \delta T ^ {2}\right) \\ + 2 (d + 2) \sum_ {t = 1} ^ {T} \alpha_ {t} - \frac {1}{4} \sum_ {t = 1} ^ {T} \alpha_ {t} \| u - x _ {t} \| _ {\hat {\Sigma} _ {t} ^ {- 1}} ^ {2} + \mathcal {O} \left(\sum_ {t = 1} ^ {T} \frac {d ^ {3} \alpha_ {t} \log \frac {T}{\delta}}{t} + \delta \sum_ {t = 1} ^ {T} \alpha_ {t} T\right) \\ = \mathcal {O} \left(d \sum_ {t = 1} ^ {T} \alpha_ {t} + \sum_ {t = 1} ^ {T} \frac {d ^ {3} \log (T / \delta)}{\alpha_ {t} t} + \sum_ {t = 1} ^ {T} \frac {d ^ {3} \alpha_ {t} \log (T / \delta)}{t} + \delta T ^ {2}\right) \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {R e g} = \text {B i a s} + \text {B o u n s} + \text {F T R L - R e g} \\ = \mathcal {O} \left(\frac {d \log (T)}{\eta_ {T}} + d \sum_ {t = 1} ^ {T} \eta_ {t} + d \sum_ {t = 1} ^ {T} \alpha_ {t} + \sum_ {t = 1} ^ {T} \frac {d ^ {3} \log (T / \delta)}{\alpha_ {t} t} + \sum_ {t = 1} ^ {T} \frac {d ^ {3} \alpha_ {t} \log (T / \delta)}{t} + \delta T ^ {2}\right) \\ \end{array}
$$
Recall that we have an additional condition in Lemma 27 such that for any  $t$ ,  $\eta_t \alpha_t \leq \frac{1}{64t}$ . Picking  $\alpha_t = \frac{d}{\sqrt{t}}$ ,  $\eta_t = \frac{1}{64d\sqrt{t}}$  and  $\delta = \frac{1}{T^2}$ , we get
$$
\operatorname {R e g} = \mathcal {O} \left(d ^ {2} \sqrt {T} \log (T) + d ^ {4} \log (T)\right) = \mathcal {O} \left(d ^ {2} \sqrt {T} \log (T)\right)
$$
where we assume  $d^2 \leq \sqrt{T}$  without loss of generality (otherwise the bound is vacuous).
E Handling Misspecification
In this section, we discuss how to handle misspecification as defined in Section 3.6. In Appendix E.1, we study the case where the amount of misspecification  $\varepsilon$  is known by the learner. In Appendix E.2, we use a blackbox approach to turn it into an algorithm that achieves almost the same regret bound (up to  $\log T$  factors) without knowing  $\varepsilon$ .
E.1 Known misspecification
As discussed in Section 3.6, when the amount of misspecification  $\varepsilon$  is known, we still use Algorithm 1, but with different  $\alpha_{t}$  and  $\eta_{t}$ . Throughout this subsection, we let  $\alpha_{t} = \frac{d}{\sqrt{t}} + \frac{\varepsilon}{\sqrt{d}}$  and  $\eta_{t} = \frac{1}{64\left(d\sqrt{t} + \frac{\varepsilon}{\sqrt{d}} t\right)}$ , and point out the modifications of the analysis from Appendix D.
We start with the regret decomposition similar to that in Appendix D, but here we define
$$
\begin{array}{l} y_{t} = \operatorname *{argmin}_{y\in \mathbb{B}_{2}^{d}}\max_{\mathcal{A}\in \operatorname {supp}(D)}\max_{a\in \mathcal{A}}|f_{t}(a) - \langle a,y\rangle |, \\ \varepsilon_{t} = \max_{\mathcal{A}\in \operatorname {supp}(D)}\max_{a\in \mathcal{A}}|f_{t}(a) - \langle a,y_{t}\rangle |, \\ c _ {t} (a) = f _ {t} (a) - \langle a, y _ {t} \rangle . \\ \end{array}
$$
The regret decomposition goes as follows: