SlowGuess/NeurIPS_2023_OCR huggingface dataset:
XU, R., MIN, Y., WANG, T., JORDAN, M. I., WANG, Z. and YANG, Z. (2023). Finding regularized competitive equilibria of heterogeneous agent macroeconomic models via reinforcement learning. In International Conference on Artificial Intelligence and Statistics. PMLR.
YE, C., XIONG, W., GU, Q. and ZHANG, T. (2023). Corruption-robust algorithms with uncertainty weighting for nonlinear contextual bandits and markov decision processes. In International Conference on Machine Learning. PMLR.
YIN, M., DUAN, Y., WANG, M. and WANG, Y.-X. (2022). Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism. arXiv preprint arXiv:2203.05804.
ZHANG, T. (2022). Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science 4 834-857.
ZHANG, W., ZHOU, D., LI, L. and GU, Q. (2020). Neural thompson sampling. arXiv preprint arXiv:2010.00827.
ZHANG, Z., YANG, J., Ji, X. and DU, S. S. (2021). Improved variance-aware confidence sets for linear bandits and linear mixture mdp. Advances in Neural Information Processing Systems 34 4342-4355.
ZHAO, H., HE, J., ZHOU, D., ZHANG, T. and GU, Q. (2023). Variance-dependent regret bounds for linear bandits and reinforcement learning: Adaptivity and computational efficiency. arXiv preprint arXiv:2302.10371.
ZHAO, H., ZHOU, D., HE, J. and GU, Q. (2022). Bandit learning with general function classes: Heteroscedastic noise and variance-dependent regret bounds. arXiv preprint arXiv:2202.13603.
ZHOU, D. and GU, Q. (2022). Computationally efficient horizon-free reinforcement learning for linear mixture mdps. arXiv preprint arXiv:2205.11507.
ZHOU, D., GU, Q. and SZEPESVARI, C. (2021). Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory. PMLR.
A Regret analysis of Algorithm 2
A.1 Clarification of notation
Before elaborating on proofs, we delineate several shorthands to facilitate our discourse. For each  $t \in [T]$ , we establish the events  $\mathcal{E}_t^\theta$  and  $\mathcal{E}_t^{\mathrm{TS}}$ . These events are denoted as the inclusion of  $\widehat{\pmb{\theta}}_t$  and  $\pmb{\theta}_t^{\mathrm{TS}}$  in the pertinent confidence ellipse with some confidence radius  $\widehat{\beta}_t$ , respectively, i.e.,
$$
\mathcal {E} _ {t} ^ {\boldsymbol {\theta}} = \left\{\left| \mathbf {x} ^ {\top} \widehat {\boldsymbol {\theta}} _ {t} - \mathbf {x} ^ {\top} \boldsymbol {\theta} ^ {*} \right| \leq \widehat {\beta} _ {t} \| \mathbf {x} \| _ {\mathbf {V} _ {t} ^ {- 1}}, \forall \mathbf {x} \in \mathcal {X} _ {t} \right\}
$$
$$
\mathcal {E} _ {t} ^ {\mathrm {T S}} = \left\{\left| \mathbf {x} ^ {\mathsf {T}} \boldsymbol {\theta} _ {t} ^ {\mathrm {T S}} - \mathbf {x} ^ {\mathsf {T}} \widehat {\boldsymbol {\theta}} _ {t} \right| \leq 2 \widehat {\beta_ {t}} \sqrt {d \log t} \| \mathbf {x} \| _ {\mathbf {V} _ {t} ^ {- 1}}, \forall \mathbf {x} \in \mathcal {X} _ {t} \right\}.
$$
We proceed to define the set  $\mathcal{S}(t)$  representing the saturated arms at round  $t$  as follows:
$$
\mathcal {S} (t) = \left\{\mathbf {x} \in \mathcal {X} _ {t}: \Delta_ {t} (\mathbf {x}) > \left(2 \sqrt {d \log t} + 1\right) \widehat {\beta} _ {t} \| \mathbf {x} \| _ {\mathbf {V} _ {t} ^ {- 1}} \right\}, \tag {A.1}
$$
where we denote  $\Delta_t(\mathbf{x}) = \langle \mathbf{x}_t^* - \mathbf{x}, \pmb{\theta}^* \rangle$  as the suboptimality of  $\mathbf{x} \in \mathcal{X}_t$ . Moreover, we present a filtration, denoted by  $\{\mathcal{H}_t\}_{t \geq 1}$ , which is defined as
$$
\mathscr {H} _ {t} = \left\{\mathscr {X} _ {1}, \boldsymbol {\theta} _ {1} ^ {\mathrm {T S}}, x _ {1}, \overline {{\sigma}} _ {1}, r _ {1}, \dots , \mathscr {X} _ {t - 1}, \boldsymbol {\theta} _ {t - 1} ^ {\mathrm {T S}}, x _ {t - 1}, \overline {{\sigma}} _ {t - 1}, r _ {t - 1}, \mathscr {X} _ {t} \right\}.
$$
This filtration encapsulates the information that has been gleaned up until the observation of the set of context vectors  $\mathcal{X}_t$  at each particular round.
Concentration inequalities are widely used in literature (Fei and Xu, 2022b; Min et al., 2022b; He et al., 2022; Lu et al., 2022; Xu et al., 2023), and here we introduce a Bernstein-type inequality applicable to vector-valued martingales. This theorem, when coalesced with the approach of weighted ridge regression, enables us to construct stringent bounds that are dependent on the variance  $\{\sigma_t^2\}_{t\geq 1}$ .
Theorem A.1 (Zhou and Gu (2022), Theorem 4.3). Let  $\{\mathcal{H}_t\}_{t=1}^\infty$  be a filtration,  $\{\mathbf{y}_t, \varrho_t\}_{t \geq 1}$  a stochastic process so that for each  $t \geq 1$ ,  $\mathbf{y}_t \in \mathbb{R}^d$  is  $\mathcal{H}_t$ -measurable and  $\varrho_t \in \mathbb{R}$  is  $\mathcal{H}_{t+1}$ -measurable. For each  $t \geq 1$ , define  $r_t := \langle \mathbf{y}_t, \boldsymbol{\theta}^* \rangle + \varrho_t$  and suppose that  $\varrho_t$ ,  $\mathbf{y}_t$  satisfy
$$
\| \mathbf {y} _ {t} \| \leq L, \quad | \varrho_ {t} | \leq R, \quad \mathbb {E} [ \varrho_ {t} \mid \mathcal {H} _ {t} ] = 0, \quad \mathbb {E} [ \varrho_ {t} ^ {2} \mid \mathcal {H} _ {t} ] \leq \sigma^ {2}
$$
for some constants  $L, R, \sigma, \epsilon > 0$  and  $\pmb{\theta}^{*} \in \mathbb{R}^{d}$ . Then, for any  $0 < \delta < 1$ , with probability at least  $1 - \delta$  we have for all  $t > 0$  that
$$
\left\| \sum_ {s = 1} ^ {t} \varrho_ {s} \mathbf {y} _ {s} \right\| _ {\mathbf {U} _ {t} ^ {- 1}} \leq \beta_ {t}, \quad \| \boldsymbol {\theta} _ {t} - \boldsymbol {\theta} ^ {*} \| _ {\mathbf {U} _ {t}} \leq \beta_ {t} + \sqrt {\lambda} \| \boldsymbol {\theta} ^ {*} \|,
$$
where  $\mathbf{U}_t = \lambda \mathbf{I}_d + \sum_{s=1}^t \mathbf{y}_t \mathbf{y}_t^\top$ ,  $\pmb{\nu}_t = \sum_{s=1}^t r_s \mathbf{y}_t$ ,  $\pmb{\theta}_t = \mathbf{U}_t^{-1} \pmb{\nu}_t$ , and
$$
\begin{array}{l} \beta_ {t} = 1 2 \sigma \sqrt {d \cdot \log \left(1 + \frac {t L ^ {2}}{d \lambda}\right) \log \left(3 2 \left(1 + \log \frac {R}{\epsilon}\right) \frac {t ^ {2}}{\delta}\right)} + 6 \log \left(3 2 \left(1 + \log \frac {R}{\epsilon}\right) \frac {t ^ {2}}{\delta}\right) \epsilon \\ + 2 4 \log \left(3 2 \Big (1 + \log \frac {R}{\epsilon} \Big) \frac {t ^ {2}}{\delta}\right) \max _ {s \in [ t ]} \{| \varrho_ {s} | \min \{1, \| \mathbf {y} _ {s} \| _ {\mathbf {U} _ {s - 1} ^ {- 1}} \} \}. \\ \end{array}
$$
A.2 Proof of supporting lemmata
The forthcoming Lemmas A.2 and A.3 show that the events  $\mathcal{E}_t^\theta$  and  $\mathcal{E}_t^{\mathrm{TS}}$  hold with high probability for all  $t\in [T]$ . The anti-concentration argument can then be developed in a manner analogous to the approach employed by Agrawal and Goyal (2013), which is often not necessary in UCB-type of arguments (Fei and Xu, 2022a; Chu et al., 2011; Min et al., 2023; Zhao et al., 2023).
Lemma A.2. Under Assumption 3.1, for any  $\delta \in (0,1)$ , if we define for all  $t\geq 1$  that
$$
\begin{array}{l} \widehat {\beta} _ {t} = 1 2 \sqrt {d \cdot \log \left(1 + \frac {t}{d \lambda \alpha^ {2}}\right) \log \left(6 4 \left(1 + \log \frac {\gamma^ {2}}{\alpha}\right) \frac {t ^ {2}}{\delta}\right)} \tag {A.2} \\ + \frac {3 0 R}{\gamma^ {2}} \cdot \log \left(6 4 \left(1 + \log \frac {\gamma^ {2}}{\alpha}\right) \frac {t ^ {2}}{\delta}\right) + \sqrt {\lambda} \\ \end{array}
$$
with the coefficients  $\alpha, \gamma, \lambda$  introduced in Algorithm 2, then the estimate  $\widehat{\pmb{\theta}}_t$  given by the associated weighted regressions satisfy
$$
\mathbb {P} \left(\left| \mathbf {x} ^ {\mathsf {T}} \widehat {\boldsymbol {\theta}} _ {t} - \mathbf {x} ^ {\mathsf {T}} \boldsymbol {\theta} ^ {*} \right| \leq \widehat {\beta} _ {t} \| \mathbf {x} \| _ {\mathbf {V} _ {t} ^ {- 1}}, \forall \mathbf {x} \in \mathcal {X} _ {t}, \forall t \geq 1\right) \geq 1 - \delta / 2.
$$
Proof. Observe that for any  $t \geq 1$  and  $\mathbf{x} \in \mathcal{X}_t$ , by Cauchy-Schwarz inequality we have
$$
\left| \mathbf {x} ^ {\top} \widehat {\boldsymbol {\theta}} _ {t} - \mathbf {x} ^ {\top} \boldsymbol {\theta} ^ {*} \right| \leq \left\| \widehat {\boldsymbol {\theta}} _ {t} - \boldsymbol {\theta} ^ {*} \right\| _ {\mathbf {V} _ {t}} \| \mathbf {x} \| _ {\mathbf {V} _ {t} ^ {- 1}}, \tag {A.3}
$$
so it suffices to bound each  $\| \mathbf{x}\|_{\mathbf{V}_{t}^{-1}}$ . By Assumption 3.1, the following conditions on  $\varepsilon_t$  and  $\mathbf{x}_t$  hold:
$$
\left| \frac {\varepsilon_ {t}}{\overline {{\sigma}} _ {t}} \right| \leq \frac {R}{\alpha}, \quad \mathbb {E} [ \varepsilon_ {t} \mid \mathcal {H} _ {t} ] = 0, \quad \mathbb {E} \left[ \left(\frac {\varepsilon_ {t}}{\overline {{\sigma}} _ {t}}\right) ^ {2} \mid \mathcal {H} _ {t} \right] \leq 1, \quad \left\| \frac {\mathbf {x} _ {t}}{\overline {{\sigma}} _ {t}} \right\| \leq \frac {1}{\alpha}.
$$
Moreover, we can obtain following  $\overline{\sigma}_t = \max \{\sigma_t,\alpha ,\gamma \| \mathbf{x}_t\|_{\mathbf{V}_t^{-1}}^{1 / 2}\}$  that
$$
\left| \frac {\varepsilon_ {t}}{\bar {\sigma} _ {t}} \right| \cdot \min  \left\{1, \left\| \frac {\mathbf {x} _ {t}}{\bar {\sigma} _ {t}} \right\| _ {\mathbf {V} _ {t} ^ {- 1}} \right\} \leq \frac {R}{\bar {\sigma} _ {t} ^ {2}} \| \mathbf {x} _ {t} \| _ {\mathbf {V} _ {t} ^ {- 1}} \leq \frac {R}{\gamma^ {2}}.
$$
Therefore, by applying Theorem A.1 with the stochastic process  $\{\mathbf{x}_t / \overline{\sigma}_t,\varepsilon_t / \overline{\sigma}_t\}_{t\geq 1}$  and  $\epsilon = R / \gamma^2$  we deduce that with probability at least  $1 - \delta /2$  for all  $t\geq 1$
$$
\begin{array}{l} \left\| \widehat {\boldsymbol {\theta}} _ {t} - \boldsymbol {\theta} ^ {*} \right\| _ {\mathbf {V} _ {t}} \leq 1 2 \sqrt {d \cdot \log \left(1 + \frac {t}{d \lambda \alpha^ {2}}\right) \log \left(6 4 \left(1 + \log \frac {\gamma^ {2}}{\alpha}\right) \frac {t ^ {2}}{\delta}\right)} \\ + \frac {3 0 R}{\gamma^ {2}} \cdot \log \left(6 4 \left(1 + \log \frac {\gamma^ {2}}{\alpha}\right) \frac {t ^ {2}}{\delta}\right) + \sqrt {\lambda} \| \boldsymbol {\theta} ^ {*} \|. \\ \end{array}
$$
Since  $\| \pmb{\theta}^{*}\|_{2}\leq 1$  by Assumption 3.1, it follows from the definition of  $\widehat{\beta}_t$  that  $\| \widehat{\pmb{\theta}}_t - \pmb{\theta}^*\|_{\mathbf{V}_t}\leq \widehat{\beta}_t$ . Combining this with (A.3), we conclude that with probability at least  $1 - \delta /2$
$$
\left| \mathbf {x} ^ {\top} \widehat {\boldsymbol {\theta}} _ {t} - \mathbf {x} ^ {\top} \boldsymbol {\theta} ^ {*} \right| \leq \widehat {\beta} _ {t} \| \mathbf {x} \| _ {\mathbf {V} _ {t} ^ {- 1}}
$$
for all  $t \geq 1$  and  $\mathbf{x} \in \mathcal{X}_t$ .
The following lemma establishes the control over  $|\mathbf{x}^{\mathrm{T}}\pmb{\theta}_{t}^{\mathrm{TS}} - \mathbf{x}^{\mathrm{T}}\widehat{\pmb{\theta}}_{t}|$ .
Lemma A.3. Under Assumption 3.1, for any  $\delta \in (0,1)$ , let  $\widehat{\beta}_t$  be as defined in Lemma A.2 for all  $t \geq 1$ . Then the sampled variable  $\pmb{\theta}_t^{\mathrm{TS}}$  in Algorithm 2 satisfies
$$
\mathbb {P} \left(| \mathbf {x} ^ {\mathsf {T}} \pmb {\theta} _ {t} ^ {\mathrm {T S}} - \mathbf {x} ^ {\mathsf {T}} \widehat {\pmb {\theta}} _ {t} | \leq \widehat {\beta} _ {t} \sqrt {2 d \log (4 t ^ {2} / \delta)} \cdot \| \mathbf {x} \| _ {\mathbf {V} _ {t} ^ {- 1}}, \forall \mathbf {x} \in \mathcal {X} _ {t}, \forall t \geq 1\right) \geq 1 - \frac {\delta}{2}.
$$
Proof. By definition,  $\pmb{\theta}_t^{\mathrm{TS}}\sim \mathcal{N}(\widehat{\pmb{\theta}}_t,\pmb {\Sigma}_t)$  where  $\pmb {\Sigma}_t = \widehat{\beta}_t^2\mathbf{V}_t^{-1}$ , thus  $\| \pmb{\theta}_t^{\mathrm{TS}} - \widehat{\pmb{\theta}}_t\|_{\pmb{\Sigma}_t^{-1}}\sim \mathcal{N}(0,1)$ . Then using the concentration inequality in Lemma C.1 for standard normal distribution, for every  $t\in [T]$ , we have with probability at least  $1 - \frac{\delta}{4t^2}$  that
$$
\begin{array}{l} | \mathbf {x} ^ {\mathsf {T}} \pmb {\theta} _ {t} ^ {\mathrm {T S}} - \mathbf {x} ^ {\mathsf {T}} \widehat {\pmb {\theta}} _ {t} | = | \mathbf {x} ^ {\mathsf {T}} \mathbf {V} _ {t} ^ {- 1 / 2} \mathbf {V} _ {t} ^ {1 / 2} (\pmb {\theta} _ {t} ^ {\mathrm {T S}} - \widehat {\pmb {\theta}} _ {t}) | \\ \leq \widehat {\beta} _ {t} \left\| \frac {\mathbf {V} _ {t} ^ {1 / 2} \left(\boldsymbol {\theta} _ {t} ^ {\mathrm {T S}} - \widehat {\boldsymbol {\theta}} _ {t}\right)}{\widehat {\beta} _ {t}} \right\| _ {2} \| \mathbf {x} \| _ {\mathbf {V} _ {t} ^ {- 1}} \\ \leq \widehat {\beta} _ {t} \sqrt {2 d \log (4 t ^ {2} / \delta)} \| \mathbf {x} \| _ {\mathbf {V} _ {t} ^ {- 1}}, \\ \end{array}
$$
where the first inequality follows from Cauchy-Schwarz inequality. Applying a union bound over  $t \in [T]$  and utilizing the fact that  $\sum_{t=1}^{\infty} \frac{1}{t^2} \leq 2$ , we obtain the final result.
In the following, we delve into the anti-concentration argument presented in Lemmas A.4 to A.6, which constitute the upper bound on expected regret.
Lemma A.4. For any  $t \geq 1$ , if  $\mathcal{H}_t$  is a filtration such that  $\mathcal{E}_t^\theta$  holds, we have a constant probability that the reward estimate  $\langle \mathbf{x}_t^*, \boldsymbol{\theta}_t^{\mathrm{TS}} \rangle$  of the optimal arm constitutes an upper confidence bound, i.e.,
$$
\mathbb {P} \left(\mathbf {x} _ {t} ^ {* \top} \boldsymbol {\theta} _ {t} ^ {\mathrm {T S}} > \mathbf {x} _ {t} ^ {* \top} \boldsymbol {\theta} ^ {*} \mid \mathcal {H} _ {t}\right) \geq \frac {1}{4 e \sqrt {\pi}}.
$$
Proof. Note that we can bound the quantity  $J_{t} = (\mathbf{x}_{t}^{*\top}\pmb{\theta}^{*} - \mathbf{x}_{t}^{*\top}\widehat{\pmb{\theta}}_{t}) / (\widehat{\beta}_{t}\| \mathbf{x}_{t}^{*}\|_{\mathbf{V}_{t}^{-1}})$  as follows:
$$
\begin{array}{l} | J _ {t} | = \left| \frac {\mathbf {x} _ {t} ^ {* \mathsf {T}} \boldsymbol {\theta} ^ {*} - \mathbf {x} _ {t} ^ {* \mathsf {T}} \widehat {\boldsymbol {\theta}} _ {t}}{\widehat {\beta} _ {t} \| \mathbf {x} _ {t} ^ {*} \| _ {\mathbf {V} _ {t} ^ {- 1}}} \right| \\ \leq \frac {\widehat {\beta} _ {t} \| \mathbf {x} _ {t} ^ {*} \| _ {\mathbf {V} _ {t} ^ {- 1}}}{\widehat {\beta} _ {t} \| \mathbf {x} _ {t} ^ {*} \| _ {\mathbf {V} _ {t} ^ {- 1}}} \\ = 1, \tag {A.4} \\ \end{array}
$$
where the inequality is due to Lemma A.2. Recalling that  $\pmb{\theta}_t^{\mathrm{TS}}$  has a mean of  $\widehat{\pmb{\theta}}_t$ , we can establish that the probability of the reward estimate  $\mathbf{x}_t^{*\top}\pmb{\theta}_t^{\mathrm{TS}}$  serving as an upper bound for the true reward is lower-bounded by
$$
\begin{array}{l} \mathbb {P} \left(\mathbf {x} _ {t} ^ {* ^ {\intercal}} \boldsymbol {\theta} _ {t} ^ {\mathrm {T S}} > \mathbf {x} _ {t} ^ {* ^ {\intercal}} \boldsymbol {\theta} ^ {*} \mid \mathscr {H} _ {t}\right) = \mathbb {P} \left(\mathbf {x} _ {t} ^ {* ^ {\intercal}} \boldsymbol {\theta} _ {t} ^ {\mathrm {T S}} - \mathbf {x} _ {t} ^ {* ^ {\intercal}} \widehat {\boldsymbol {\theta}} _ {t} > \mathbf {x} _ {t} ^ {* ^ {\intercal}} \boldsymbol {\theta} ^ {*} - \mathbf {x} _ {t} ^ {* ^ {\intercal}} \widehat {\boldsymbol {\theta}} _ {t} \mid \mathscr {H} _ {t}\right) \\ = \mathbb {P} \left(\frac {\mathbf {x} _ {t} ^ {* \mathsf {T}} \boldsymbol {\theta} _ {t} ^ {\mathrm {T S}} - \mathbf {x} _ {t} ^ {* \mathsf {T}} \widehat {\boldsymbol {\theta}} _ {t}}{\widehat {\beta} _ {t} \| \mathbf {x} _ {t} ^ {*} \| _ {\mathbf {V} _ {t} ^ {- 1}}} > \frac {\mathbf {x} _ {t} ^ {* \mathsf {T}} \boldsymbol {\theta} ^ {*} - \mathbf {x} _ {t} ^ {* \mathsf {T}} \widehat {\boldsymbol {\theta}} _ {t}}{\widehat {\beta} _ {t} \| \mathbf {x} _ {t} ^ {*} \| _ {\mathbf {V} _ {t} ^ {- 1}}} \Bigg |   \mathcal {H} _ {t}\right) \\ \geq \frac {1}{4 \sqrt {\pi}} e ^ {- J _ {t} ^ {2}}. \tag {A.5} \\ \end{array}
$$
The last inequality follows from the anti-concentration inequality in Lemma C.1 for standard normal distribution and the fact that
$$
\frac {\mathbf {x} _ {t} ^ {* \mathsf {T}} \boldsymbol {\theta} _ {t} ^ {\mathrm {T S}} - \mathbf {x} _ {t} ^ {* \mathsf {T}} \widehat {\boldsymbol {\theta}} _ {t}}{\widehat {\beta} _ {t} \| \mathbf {x} _ {t} ^ {*} \| _ {\mathbf {V} _ {t} ^ {- 1}}} \sim \mathcal {N} (0, 1).
$$
Consequently, combining (A.4) and (A.5), it follows that the desired probability is lower bounded by a constant, i.e.,  $\mathbb{P}(\mathbf{x}_t^{*\top}\pmb{\theta}_t^{\mathrm{TS}} > \mathbf{x}_t^{*\top}\pmb{\theta}^*\mid \mathcal{H}_t)\geq \frac{1}{4e\sqrt{\pi}}.$
Lemma A.5. For any  $t \geq 1$ , if  $\mathcal{H}_t$  is a filtration such that  $\mathcal{E}_t^\theta$  holds, we have with a constant probability that the chosen arm  $\mathbf{x}_t$  is not a saturated arm, i.e.,
$$
\mathbb {P} (\mathbf {x} _ {t} \notin \mathcal {S} (t) \mid \mathcal {H} _ {t}) \geq \frac {1}{4 e \sqrt {\pi}} - \frac {1}{t ^ {2}}.
$$
Proof. Recall the definition of the set of saturated arms  $\mathcal{S}(t)$  in (A.1) and that the selected arm  $\mathbf{x}_t = \arg \max_{\mathbf{x} \in \mathcal{X}_t} \mathbf{x}^\top \boldsymbol{\theta}_t^{\mathrm{TS}}$  aims to maximize the estimated reward. Consequently,  $\mathbf{x}_t \notin \mathcal{S}(t)$  if  $\mathbf{x}_t^{*\top} \boldsymbol{\theta}_t^{\mathrm{TS}} \geq \mathbf{x}^\top \boldsymbol{\theta}_t^{\mathrm{TS}}$  for all saturated arms  $\mathbf{x} \in \mathcal{S}(t)$ , implying that the estimated reward for the optimal arm surpasses the estimated rewards of all saturated arms. It follows that:
$$
\mathbb {P} \left(\mathbf {x} _ {t} \notin \mathcal {S} (t) \mid \mathcal {H} _ {t}\right) \geq \mathbb {P} \left(\mathbf {x} _ {t} ^ {* \mathsf {T}} \boldsymbol {\theta} _ {t} ^ {\mathrm {T S}} \geq \mathbf {x} ^ {\mathsf {T}} \boldsymbol {\theta} _ {t} ^ {\mathrm {T S}}, \forall \mathbf {x} \in \mathcal {S} (t) \mid \mathcal {H} _ {t}\right).
$$
Further note that by the definition of saturated arms, we have
$$
\mathbf {x} ^ {\intercal} \boldsymbol {\theta} _ {t} ^ {\mathrm {T S}} \leq \mathbf {x} ^ {\intercal} \boldsymbol {\theta} ^ {*} + (2 \sqrt {d \log t} + 1) \widehat {\beta} _ {t} \| \mathbf {x} \| _ {\mathbf {V} _ {t} ^ {- 1}} <   \mathbf {x} ^ {\intercal} \boldsymbol {\theta} ^ {*} + \Delta_ {t} (\mathbf {x}) = \mathbf {x} _ {t} ^ {* \intercal} \boldsymbol {\theta} ^ {*}
$$
when both  $\mathcal{E}_t^\theta$  and  $\mathcal{E}_t^{\mathrm{TS}}$  hold. Therefore,
$$
\mathbb {P} \left(\mathbf {x} _ {t} \notin \mathcal {S} (t) \mid \mathcal {H} _ {t}\right) \geq \mathbb {P} \left(\mathbf {x} _ {t} ^ {* ^ {\intercal}} \boldsymbol {\theta} _ {t} ^ {\mathrm {T S}} > \mathbf {x} _ {t} ^ {* ^ {\intercal}} \boldsymbol {\theta} ^ {*} \mid \mathcal {H} _ {t}\right) - \mathbb {P} \left(\overline {{\mathcal {E} ^ {\mathrm {T S}}}} (t) \mid \mathcal {H} _ {t}\right),
$$
The last inequality is a consequence of Lemmas A.3 and A.4.
Lemma A.6. For any  $t \geq 1$ , if  $\mathcal{H}_t$  is a filtration such that both  $\mathcal{E}_t^\theta$  and  $\mathcal{E}_t^{\mathrm{TS}}$  hold, we have
$$
\mathbb {E} \left[ \Delta_ {t} \mid \mathcal {H} _ {t} \right] \leq \left(\frac {2}{\frac {1}{4 e \sqrt {\pi}} - \frac {1}{t ^ {2}}} + 1\right) \left(2 \sqrt {d \log t} + 1\right) \widehat {\beta} _ {t} \mathbb {E} \left[ \| \mathbf {x} _ {t} \| _ {\mathbf {V} _ {t} ^ {- 1}} \mid \mathcal {H} _ {t} \right] + \frac {2}{t ^ {2}}.
$$
Proof. Let  $\mathbf{x}_t^\dagger$  represent the unsaturated arm with the smallest  $\| \cdot \|_{\mathbf{V}_t^{-1}}$  norm, i.e.,
$$
\mathbf{x}_{t}^{\dagger} = \operatorname *{arg  min}_{\mathbf{x}\notin \mathcal{S}(t)}\| \mathbf{x}\|_{\mathbf{V}_{t}^{-1}}.
$$
The existence of such an unsaturated arm  $\mathbf{x}_t^\dagger$  is guaranteed since  $\mathbf{x}_t^* \notin \mathcal{S}(t)$ . When both  $\mathcal{E}_t^\theta$  and  $\mathcal{E}_t^{\mathrm{TS}}$  hold, we can express the suboptimality as follows:
$$
\begin{array}{l} \Delta_ {t} = \Delta_ {t} (\mathbf {x} _ {t} ^ {\dagger}) + \mathbf {x} _ {t} ^ {\dagger \mathsf {T}} \boldsymbol {\theta} ^ {*} - \mathbf {x} _ {t} ^ {\mathsf {T}} \boldsymbol {\theta} ^ {*} \\ \leq \Delta_ {t} \left(\mathbf {x} _ {t} ^ {\dagger}\right) + \left(\mathbf {x} _ {t} ^ {\dagger} ^ {\mathsf {T}} \boldsymbol {\theta} _ {t} ^ {\mathrm {T S}} + (2 \sqrt {d \log t} + 1) \widehat {\beta_ {t}} \| \mathbf {x} _ {t} ^ {\dagger} \| _ {\mathbf {V} _ {t} ^ {- 1}}\right) \\ - \left(\mathbf {x} _ {t} ^ {\top} \boldsymbol {\theta} _ {t} ^ {\mathrm {T S}} - (2 \sqrt {d \log t} + 1) \widehat {\beta} _ {t} \| \mathbf {x} _ {t} \| _ {\mathbf {V} _ {t} ^ {- 1}}\right) \\ \end{array}
$$
Then since  $\mathbf{x}_t$  is the optimal arm under  $\theta_t^{\mathrm{TS}}$ , we further have
$$
\begin{array}{l} \Delta_ {t} \leq \Delta_ {t} (\mathbf {x} _ {t} ^ {\dagger}) + (2 \sqrt {d \log t} + 1) \widehat {\beta} _ {t} \| \mathbf {x} _ {t} ^ {\dagger} \| _ {\mathbf {V} _ {t} ^ {- 1}} + (2 \sqrt {d \log t} + 1) \widehat {\beta} _ {t} \| \mathbf {x} _ {t} \| _ {\mathbf {V} _ {t} ^ {- 1}} \\ \leq (2 \sqrt {d \log t} + 1) \widehat {\beta} _ {t} (2 \| \mathbf {x} _ {t} ^ {\dagger} \| _ {\mathbf {V} _ {t} ^ {- 1}} + \| \mathbf {x} _ {t} \| _ {\mathbf {V} _ {t} ^ {- 1}}), \\ \end{array}
$$
where the second inequality results from  $\mathbf{x}_t^\dagger \notin \mathcal{S}(t)$ . Note that  $\| \mathbf{x}_t\|_{\mathbf{V}_t^{-1}}\geq \| \mathbf{x}_t^\dagger \|_{\mathbf{V}_t^{-1}}$  with constant probability, i.e.,
$$
\begin{array}{l} \mathbb {E} [ \| \mathbf {x} _ {t} \| _ {\mathbf {V} _ {t} ^ {- 1}} \mid \mathcal {H} _ {t} ] \geq \mathbb {E} [ \| \mathbf {x} _ {t} \| _ {\mathbf {V} _ {t} ^ {- 1}} \mid \mathcal {H} _ {t}, \mathbf {x} _ {t} \notin \mathcal {S} (t) ] \cdot \mathbb {P} (\mathbf {x} _ {t} \notin \mathcal {S} (t) \mid \mathcal {H} _ {t}) \\ \geq \left(\frac {1}{4 e \sqrt {\pi}} - \frac {1}{t ^ {2}}\right) \mathbb {E} \left[ \left\| \mathbf {x} _ {t} ^ {\dagger} \right\| _ {\mathbf {V} _ {t} ^ {- 1}} \mid \mathcal {H} _ {t} \right]. \tag {A.6} \\ \end{array}
$$
The last inequality is due to Lemma A.5 and the definition of  $\mathbf{x}_t^\dagger$  as the unsaturated arm with the smallest  $\| \cdot \|_{\mathbf{V}_t^{-1}}$ . Consequently,
$$
\begin{array}{l} \mathbb {E} [ \Delta_ {t} \mid \mathcal {H} _ {t} ] \leq \mathbb {E} [ (2 \sqrt {d \log t} + 1) \widehat {\beta} _ {t} (2 \| \mathbf {x} _ {t} ^ {\dagger} \| _ {\mathbf {V} _ {t} ^ {- 1}} + \| \mathbf {x} _ {t} \| _ {\mathbf {V} _ {t} ^ {- 1}}) \mid \mathcal {H} _ {t} ] + 2 \mathbb {P} (\overline {{\mathcal {E} _ {t} ^ {\mathrm {T S}}}}) \\ \leq \left(\frac {2}{\frac {1}{4 e \sqrt {\pi}} - \frac {1}{t ^ {2}}} + 1\right) (2 \sqrt {d \log t} + 1) \widehat {\beta} _ {t} \mathbb {E} [ \| \mathbf {x} _ {t} \| _ {\mathbf {V} _ {t} ^ {- 1}} \mid \mathcal {H} _ {t} ] + \frac {2}{t ^ {2}}, \\ \end{array}
$$
where the first inequality is due to  $\Delta_t \leq 2$  for all  $t$ , the second inequality follows from applying (A.6), Lemmas A.3 and A.4.
A.3 Proof of Theorem 4.1
Let us define a shorthand  $\mathcal{E}^{\theta} = \cap_{t=1}^{T} \mathcal{E}_{t}^{\theta}$ . The expected regret can be decomposed as follows:
$$
\begin{array}{l} \mathbb {E} [ \mathcal {R} (T) ] = \sum_ {t = 1} ^ {T} \mathbb {E} [ \Delta_ {t} \mid \mathcal {E} ^ {\boldsymbol {\theta}} ] \mathbb {P} (\mathcal {E} ^ {\boldsymbol {\theta}}) + \sum_ {t = 1} ^ {T} \mathbb {E} [ \Delta_ {t} \mid \overline {{\mathcal {E} ^ {\boldsymbol {\theta}}}} ] \mathbb {P} (\overline {{\mathcal {E} ^ {\boldsymbol {\theta}}}}) \\ \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \Delta_ {t} \mid \mathcal {H} _ {t}, \mathcal {E} ^ {\boldsymbol {\theta}} \right] + 2 T \cdot \frac {\delta}{2} \\ \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \min  \left\{2, (2 \sqrt {d \log t} + 1) C \widehat {\beta} _ {t} \| \mathbf {x} _ {t} \| _ {\mathbf {V} _ {t} ^ {- 1}} \right\} \mid \mathcal {H} _ {t} \right] + \sum_ {t = 1} ^ {T} \frac {2}{t ^ {2}} + \delta T \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \min  \left\{2, (2 \sqrt {d \log t} + 1) C \widehat {\beta} _ {t} \| \mathbf {x} _ {t} \| _ {\mathbf {V} _ {t} ^ {- 1}} \right\} \middle | \mathcal {E} ^ {\boldsymbol {\theta}} \right] + \delta T + 4, \\ \end{array}
$$