SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $G_{t} = 8\sqrt{2\widehat{V}_{t - 1}}\ln T + 144\ln^{2}T + 176\ln T$ . We show that with probability at least  $1 - \delta$  the following holds for all  $t\in [T]$ :
$$
V _ {t} \leq \widehat {V} _ {t} + 8 \sqrt {\widehat {V} _ {t} \ln (2 T / \delta) + 7 2 \ln (2 T / \delta) ^ {2}} + 8 8 \ln (T / \delta) \tag {39}
$$
Let  $Y_{s} = \mathbb{E}_{s}\left[(\ell_{t}(A_{t},X_{t}) - X_{t}^{\top}m_{t,A_{t}})^{2}\right] - (\ell_{t}(A_{t},X_{t}) - X_{t}^{\top}m_{t,A_{t}})^{2}$ . Then,  $Y_{s} \leq 4$  almost surely, since  $\ell_t(A_t,X_t) - X_t^\top m_{t,A_t} \leq 2$ . Similarly we bound the second moment of  $Y_{s}$ , using Jensen's inequality:
$$
\begin{array}{l} \mathbb {E} _ {s} \left[ Y _ {s} ^ {2} \right] = \mathbb {E} _ {s} \left[ \left(\mathbb {E} _ {s} \left[ \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\intercal} m _ {t, A _ {t}}\right) ^ {2} \right] - \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\intercal} m _ {t, A _ {t}}\right) ^ {2}\right) ^ {2} \right] \\ \leq 2 \mathbb {E} _ {s} \left[ \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\top} m _ {t, A _ {t}}\right) ^ {2} \right] ^ {2} + 2 \mathbb {E} _ {s} \left[ \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\top} m _ {t, A _ {t}}\right) ^ {4} \right] \\ \leq 1 6 \mathbb {E} _ {s} \left[ \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\top} m _ {t, A _ {t}}\right) ^ {2} \right]. \\ \end{array}
$$
By Lemma B.2, the following holds for some  $\delta' \in (0,1)$ :
$$
V _ {t} \leq \widehat {V} _ {t} + 8 \max  \left\{2 \sqrt {V _ {t}}, \sqrt {\ln \left(1 / \delta^ {\prime}\right)} \right\} \sqrt {\ln \left(1 / \delta^ {\prime}\right)} \tag {40}
$$
Note that this inequality i can be rearranged as
$$
V _ {t} \leq \widehat {V} _ {t} + 8 \sqrt {\widehat {V} _ {t} \ln (1 / \delta^ {\prime}) + 7 2 \ln (1 / \delta^ {\prime}) ^ {2}} + 8 8 \ln (1 / \delta^ {\prime}).
$$
Then, taking a union bound over  $t \in [T]$  and taking  $\delta = \delta' / T$ , we get that (39) holds for all  $t \in [T]$ . Let  $\mathcal{E}_T$  be an event that for all  $t \in [1,T]$ , (39) holds with  $\delta = 1 / T$ . From (37), (38), and the choice of  $\eta_t$ , we get:
$$
\begin{array}{l} R _ {T} \leq \mathbb {E} \left[ 2 d K \gamma^ {2} \sum_ {t = 1} ^ {T} \eta_ {t} \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\top} m _ {t, A _ {t}}\right) ^ {2} + 2 + \frac {K \log T}{\eta_ {T}} \right] \tag {41} \\ = 2 d K \gamma^ {2} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \eta_ {t} \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\top} m _ {t, A _ {t}}\right) ^ {2} \mathbb {1} \left\{\mathcal {E} _ {T} \right\} \right] \\ + 2 d K \gamma^ {2} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \eta_ {t} \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\top} m _ {t, A _ {t}}\right) ^ {2} \mathbb {1} \left\{\overline {{\mathcal {E}}} _ {T} \right\} \right] + 2 + \mathbb {E} \left[ \frac {K \log T}{\eta_ {T}} \right] \\ \leq 2 \sqrt {d} K \gamma^ {2} \sum_ {t = 1} ^ {T} \frac {V _ {t} - V _ {t - 1}}{\sqrt {V _ {t}}} + \frac {1}{T} 2 \sqrt {d} K \gamma^ {2} T + 2 + \frac {K \log T}{\eta_ {T} ^ {\prime}} \\ = 2 \sqrt {d} K \gamma^ {2} \sum_ {t = 1} ^ {T} \frac {(\sqrt {V _ {t}} - \sqrt {V _ {t - 1}}) (\sqrt {V _ {t}} + \sqrt {V _ {t - 1}})}{\sqrt {V _ {t}}} + 2 \sqrt {d} K \gamma^ {2} + 2 + \frac {K \log T}{\eta_ {T} ^ {\prime}} \\ \end{array}
$$
$$
\begin{array}{l} \leq 4 \sqrt {d} K \gamma^ {2} \sum_ {t = 1} ^ {T} \left(\sqrt {V _ {t}} - \sqrt {V _ {t - 1}}\right) + 2 \sqrt {d} K \gamma^ {2} + 2 + \frac {K \log T}{\eta_ {T} ^ {\prime}} \\ \leq 4 \sqrt {d} K \gamma^ {2} \sqrt {V _ {T}} + 2 \sqrt {d} K \gamma^ {2} + 2 + \frac {K \log T}{\eta_ {T} ^ {\prime}}. \\ \end{array}
$$
which implies the result of Theorem 3.1. In the equation above,  $\eta_T' = (100dK\gamma^2 + d(V_{t-1} + 1 + G_t'))^{-1/2}$  and  $G_t' = 8\sqrt{2V_{t-1}\ln T + 144\ln^2T} + 176\ln T$ . In line 4 we used that  $\mathbb{E}[1/\eta_T] \leq \mathbb{E}[1/\eta_T']$  by Jensen's inequality to show that
$$
\mathbb {E} \left[ \frac {1}{\eta_ {T}} \right] = \mathbb {E} \left[ (1 0 0 d K \gamma^ {2} + d K (\widehat {V} _ {t - 1} + 1 + G _ {t})) ^ {1 / 2} \right] \leq (1 0 0 d K \gamma^ {2} + d (V _ {t - 1} + 1 + G _ {t} ^ {\prime})) ^ {1 / 2} = \frac {1}{\eta_ {T} ^ {\prime}}.
$$
Proof of Theorem 3.2 As it was done in the proof of Theorem 3.1 we control the deviation of the learning rate
$$
\eta_ {t} = \left(1 0 0 d K \gamma^ {2} + d \left(\widehat {L} _ {t - 1} + 1 + H _ {t}\right)\right) ^ {- 1 / 2},
$$
where  $H_{t}$  is as defined in (14). Using Lemma B.2, we show that with probability at least  $1 - \delta$  the following holds for all  $t\in [T]$ :
$$
L _ {t} \leq \widehat {L} _ {t} + 8 \sqrt {\widehat {L} _ {t} \ln (1 / \delta) + 2 0 \ln (2 T / \delta) ^ {2}} + 3 6 \ln (2 T / \delta) \tag {42}
$$
$D_{s} = \mathbb{E}_{s}\left[\langle X_{s},\theta_{s,A_{s}}\rangle \right] - \langle X_{s},\theta_{s,A_{s}}\rangle$  . Then,  $D_{s}\leq 2$  almost surely and by Jensen's inequality
$$
\mathbb {E} _ {s} \left[ D _ {s} ^ {2} \right] = \mathbb {E} _ {s} \left[ \left(\mathbb {E} _ {s} \left[ \langle X _ {s}, \theta_ {s, A _ {s}} \rangle \right] - \langle X _ {s}, \theta_ {s, A _ {s}} \rangle\right) ^ {2} \right] \leq 2 \mathbb {E} _ {s} \left[ \langle X _ {s}, \theta_ {s, A _ {s}} \rangle \right] ^ {2} + 2 \mathbb {E} _ {s} \left[ \left(\langle X _ {s}, \theta_ {s, A _ {s}} \rangle\right) ^ {2} \right] \leq 4 \mathbb {E} _ {t} \left[ \ell_ {t} (X _ {t}, A _ {t}) \right].
$$
$$
L _ {t} \leq \widehat {L} _ {t} + 4 \max  \left\{2 \sqrt {L _ {t}}, \sqrt {\ln \left(1 / \delta^ {\prime}\right)} \right\} \sqrt {\ln \left(1 / \delta^ {\prime}\right)} \tag {43}
$$
$$
L _ {t} \leq \widehat {L} _ {t} + 8 \sqrt {\widehat {L} _ {t} \ln (1 / \delta^ {\prime}) + 2 0 \ln (1 / \delta^ {\prime}) ^ {2}} + 3 6 \ln (1 / \delta^ {\prime}).
$$
Then, taking a union bound over  $t \in [T]$  and taking  $\delta = \delta' / T$ , we get that (42) holds for all  $t \in [T]$ . Let  $\mathcal{E}_T$  be an event that for all  $t \in [1, T]$ , (42) holds with  $\delta = 1 / T$ . From (37), (38), the choice of  $\eta_t$ ,  $m_t = \bar{0}$  and since  $0 \leq \ell_t \leq 1$ , we get:
$$
\begin{array}{l} R _ {T} \leq \mathbb {E} \left[ 2 d K \gamma^ {2} \sum_ {t = 1} ^ {T} \eta_ {t} \ell_ {t} ^ {2} (A _ {t}, X _ {t}) + 2 + \frac {K \log T}{\eta_ {T}} \right] \leq \mathbb {E} \left[ 2 d K \gamma^ {2} \sum_ {t = 1} ^ {T} \eta_ {t} \ell_ {t} (A _ {t}, X _ {t}) + 2 + \frac {K \log T}{\eta_ {T}} \right] \\ = 2 d K \gamma^ {2} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \eta_ {t} \ell_ {t} \left(A _ {t}, X _ {t}\right) \mathbb {1} \left\{\mathcal {E} _ {T} \right\} \right] + 2 d K \gamma^ {2} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \eta_ {t} \ell_ {t} ^ {2} \left(A _ {t}, X _ {t}\right) \mathbb {1} \left\{\overline {{\mathcal {E}}} _ {T} \right\} \right] + 2 + \mathbb {E} \left[ \frac {K \log T}{\eta_ {T}} \right] \\ \leq 2 \sqrt {d} K \gamma^ {2} \sum_ {t = 1} ^ {T} \frac {L _ {t} - L _ {t - 1}}{\sqrt {L _ {t}}} + \frac {1}{T} 2 \sqrt {d} K \gamma^ {2} T + 2 + \frac {K \log T}{\eta_ {T} ^ {\prime}} \\ = 2 \sqrt {d} K \gamma^ {2} \sum_ {t = 1} ^ {T} \frac {(\sqrt {L _ {t}} - \sqrt {L _ {t - 1}}) (\sqrt {L _ {t}} + \sqrt {L _ {t - 1}})}{\sqrt {L _ {t}}} + 2 \sqrt {d K} \gamma^ {2} + 2 + \frac {K \log T}{\eta_ {T} ^ {\prime}} \\ \leq 4 \sqrt {d} K \gamma^ {2} \sum_ {t = 1} ^ {T} \left(\sqrt {L _ {t}} - \sqrt {L _ {t - 1}}\right) + 2 \sqrt {d} K \gamma^ {2} + 2 + \frac {K \log T}{\eta_ {T} ^ {\prime}} \\ \leq 4 \sqrt {d} K \gamma^ {2} \sqrt {L _ {T}} + 2 \sqrt {d} K \gamma^ {2} + 2 + \frac {K \log T}{\eta_ {T} ^ {\prime}}, \\ \end{array}
$$
where in the equation above,  $\eta_t' = (100dK\gamma^2 + d(L_{t-1} + 1 + H_{t-1}'))^{-1/2}$  and  $H_t' = 8\sqrt{2L_{t-1}\ln T + 40\ln T} + 72\ln T$ . By solving the quadratic equation over  $L_T^*$ , we obtain the statement of the theorem.
D On the difference between LINEXP3 and CONTEXTEW
Consider the LINEXP3 algorithm of Neu and Olkhovskaya (2020), that draws actions after observing the context  $X_{t}$  with probability
$$
\pi_ {t} (a | X _ {t}) = (1 - \gamma) \frac {w _ {t} (X _ {t} , a)}{\sum_ {a ^ {\prime}} w _ {t} (X _ {t} , a ^ {\prime})} + \frac {\gamma}{K},
$$
where  $w_{t}(X_{t},a) = \exp \left(-\eta \sum_{s = 0}^{t - 1}\langle X_{t},\widehat{\theta}_{s,a}\rangle\right)$  and using the estimator
$$
\widetilde {\theta} _ {t, a} ^ {*} = \mathbb {1} \left\{A _ {t} = a \right\} S _ {t, a} ^ {- 1} X _ {t} \left\langle X _ {t}, \theta_ {t, a} \right\rangle ,
$$
where  $S_{t,a} = \mathbb{E}_t[\pi_t(a|X_t)X_tX_t^\top]$ . Since LINEXP3 uses implicit exploration with probability  $\gamma$ ,  $\lambda_{min}(S_{t,a}) \geq \lambda_{min}(\Sigma)\frac{\gamma}{K}$ . But then, setting  $\gamma = 0$ ,  $S_{t,a}$  is still invertible as no actions have  $\pi_t(a|X_t) = 0$ . But still, the smallest eigenvalue  $\lambda_{\mathrm{min}}(S_{t,a})$  can be arbitrary small. Then, the analysis of the variance term in LINEXP3 looks as:
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \sum_ {a = 1} ^ {K} \pi_ {t} (a | X _ {0}) \langle X _ {0}, \widehat {\theta} _ {t, a} \rangle^ {2} \right] = \mathbb {E} _ {t} \left[ \sum_ {a = 1} ^ {K} \pi_ {t} (a | X _ {0}) \left(X _ {0} ^ {\top} S _ {t, a} ^ {- 1} X _ {t} X _ {t} ^ {\top} \theta_ {t, a} \mathbb {1} \left\{A _ {t} = a \right\}\right) ^ {2} \right] \\ = \mathbb {E} \left[ \ell_ {t} (X _ {t}, A _ {t}) ^ {2} \mathrm {t r} \left(\pi_ {t} (a | X _ {0}) X _ {0} X _ {0} ^ {\top} S _ {t, a} ^ {- 1} X _ {t} X _ {0} ^ {\top} S _ {t, a} ^ {- 1}\right) \right]. \\ \end{array}
$$
We can define  $Var_{t}^{\prime}$  for LINEXP3 in direct analogy to  $Var_{t}$  for CONTEXTEW above, which gives (almost surely):
$$
\begin{array}{l} \mathbb {E} _ {X _ {0}} \left[ V a r _ {t} ^ {\prime} \right] = \mathbb {E} _ {X _ {0}} \left[ \operatorname {t r} \left(\pi_ {t} (a | X _ {0}) X _ {0} X _ {0} ^ {\top} S _ {t, a} ^ {- 1} X _ {t} X _ {t} ^ {\top} S _ {t, a} ^ {- 1}\right) \right] \\ = \mathbb {E} _ {X _ {0}} \left[ \mathrm {t r} \left(\Sigma_ {t, a} S _ {t, a} ^ {- 1} X _ {t} X _ {t} ^ {\intercal} S _ {t, a} ^ {- 1}\right) \right] = X _ {t} ^ {\intercal} S _ {t, a} ^ {- 1} X _ {t}, \\ \end{array}
$$
which can be arbitrary large.
Meanwhile, the smallest eigenvalue  $\lambda_{\mathrm{min}}(\Sigma_{t,a})$  can be arbitrary small too. But, as we showed above in the analysis of CONTEXTEW,  $Var_t$  is bounded by  $dK\gamma^2$  because of the log-concavity of  $Z_t(X_t)$  and step (6) of CONTEXTEW.