SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {s \in \Psi_ {t, \ell}} w _ {s} ^ {2} \left(r _ {s} - \left\langle \mathbf {x} _ {s}, \widehat {\boldsymbol {\theta}} _ {t, \ell} \right\rangle\right) ^ {2} \leq \sum_ {s \in \Psi_ {t, \ell}} w _ {s} ^ {2} \left(r _ {s} - \left\langle \mathbf {x} _ {s}, \boldsymbol {\theta} ^ {*} \right\rangle\right) ^ {2} + 2 ^ {- 2 \ell} \| \boldsymbol {\theta} ^ {*} \| _ {2} ^ {2} \\ \leq \sum_ {s \in \Psi_ {t + 1, \ell}} w _ {s} ^ {2} \varepsilon_ {s} ^ {2} + 2 ^ {- 2 \ell}, \\ \end{array}
$$
where the first inequality follows from  $\theta_{t,\ell}$  being the minimizer of the weighted ridge regression problem. Combining the inequality with (B.2) yields the desired result.
Lemma B.5. Suppose that  $\mathcal{E}_{t,\ell}^{\theta}$  and  $\mathcal{E}_{t,\ell}^{\mathrm{TS}}$  hold for all  $\ell \in [L]$  and  $t\in \Psi_{t + 1,\ell}$ . If all the elements in series  $\{\widetilde{\beta}_{t,\ell}\}_{t,\ell}$  and  $\{\widehat{\beta}_{t,\ell}\}_{t,\ell}$ , as defined in (B.1) and (5.3) respectively, satisfy
$$
\widetilde {\beta} _ {t, \ell} \leq \widehat {\beta} _ {t, \ell},
$$
then for all  $t \geq 1$  and  $\ell \in [L]$  such that  $\mathcal{X}_{t,\ell}$  exists, it holds  $\mathbf{x}_t^* \in \mathcal{X}_{t,\ell}$ .
Proof. We prove the statement by induction. If  $\ell = 1$ , then  $\mathbf{x}_t^* \in \mathcal{X}_{t,\ell} = \mathcal{X}_t$  trivially holds. Suppose  $\mathbf{x}_t^* \in \mathcal{X}_{t,\ell}$  for some  $\ell \in \mathbb{Z}_+$  and  $\mathcal{X}_{t,\ell+1}$  exists, we have for all  $\mathbf{x} \in \mathcal{X}_{t,\ell}$  that
$$
\begin{array}{l} \left| \langle \mathbf {x}, \boldsymbol {\theta} _ {t, \ell} ^ {\mathrm {T S}} - \boldsymbol {\theta} ^ {*} \rangle \right| \leq \left| \langle \mathbf {x}, \widehat {\boldsymbol {\theta}} _ {t, \ell} - \boldsymbol {\theta} ^ {*} \rangle \right| + \left| \langle \mathbf {x}, \boldsymbol {\theta} _ {t, \ell} ^ {\mathrm {T S}} - \widehat {\boldsymbol {\theta}} _ {t, \ell} \rangle \right| \\ \leq \left(\sqrt {2 d \log \left(4 t ^ {2} L / \delta\right)} + 1\right) \widehat {\beta} _ {t, \ell} \| \mathbf {x} \| _ {\mathbf {V} _ {t, \ell} ^ {- 1}} \\ \leq 2 ^ {- \ell} (\sqrt {2 d \log (4 t ^ {2} L / \delta)} + 1) \widehat {\beta} _ {t, \ell}, \\ \end{array}
$$
where the second inequality is due to the assumption  $\widetilde{\beta}_{t,\ell} \leq \widehat{\beta}_{t,\ell}$ , Lemmas B.2 and B.3, and the last inequality is due to  $\| \mathbf{x} \|_{\mathbf{V}_{t,\ell}^{-1}} \leq 2^{-\ell}$  for all  $\mathbf{x} \in \mathcal{X}_{t,\ell}$ . If we define  $\widetilde{\mathbf{x}}_t = \arg \max_{\mathbf{x} \in \mathcal{X}_{t,\ell}} \langle \mathbf{x}, \boldsymbol{\theta}_{t,\ell}^{\mathrm{TS}} \rangle$ , the it holds that
$$
\begin{array}{l} \langle \widetilde {\mathbf {x}} _ {t} - \mathbf {x} _ {t} ^ {*}, \boldsymbol {\theta} _ {t, \ell} ^ {\mathrm {T S}} \rangle \leq \langle \widetilde {\mathbf {x}} _ {t} - \mathbf {x} _ {t} ^ {*}, \boldsymbol {\theta} ^ {*} \rangle + \langle \widetilde {\mathbf {x}} _ {t} - \mathbf {x} _ {t} ^ {*}, \boldsymbol {\theta} _ {t, \ell} ^ {\mathrm {T S}} - \boldsymbol {\theta} ^ {*} \rangle \\ \leq | \langle \tilde {\mathbf {x}} _ {t}, \boldsymbol {\theta} _ {t, \ell} ^ {\mathrm {T S}} - \boldsymbol {\theta} ^ {*} \rangle | + | \langle \mathbf {x} _ {t} ^ {*}, \boldsymbol {\theta} _ {t, \ell} ^ {\mathrm {T S}} - \boldsymbol {\theta} ^ {*} \rangle | \\ \leq 2 \cdot 2 ^ {- \ell} \left(\sqrt {2 d \log \left(4 t ^ {2} L / \delta\right)} + 1\right) \widehat {\beta} _ {t, \ell}, \\ \end{array}
$$
where the second inequality follows from  $\langle \widetilde{\mathbf{x}}_t - \mathbf{x}_t^*,\pmb {\theta}^*\rangle \leq 0$  . Hence, the optimal action  $\mathbf{x}_t^*\in \mathcal{X}_{t,\ell +1}$  and the statement follows from induction.
Lemma B.6. Suppose that  $\widetilde{\beta}_{t,\ell} \leq \widehat{\beta}_{t,\ell}$ ,  $\mathcal{E}_{t,\ell}^{\theta}$ , and  $\mathcal{E}_{t,\ell}^{\mathrm{TS}}$  hold for all  $\ell \in [L]$  and  $t \in \Psi_{t+1,\ell}$ . For any  $\ell \in [L] \setminus \{1\}$ , we have
$$
\sum_ {t \in \Psi_ {T + 1, \ell}} \Delta_ {t} \leq 1 6 d \cdot 2 ^ {\ell} (\sqrt {2 d \log (4 T ^ {2} L / \delta)} + 1) \widehat {\beta} _ {T, \ell - 1} \cdot \log (1 + 2 ^ {2 \ell} T / d).
$$
Proof. For all  $t \in \Psi_{T+1,\ell}$ , we have  $\mathbf{x}_t, \mathbf{x}_t^* \in \mathcal{X}_{t,\ell}$  following Lemma B.5 and the assumptions on  $\widetilde{\beta}_{t,\ell} \leq \widehat{\beta}_{t,\ell}, \mathcal{E}_{t,\ell}^{\theta}$ , and  $\mathcal{E}_{t,\ell}^{\mathrm{TS}}$ . Due to Line 11 in Algorithm 3, it holds
$$
\left\langle \mathbf {x} _ {t} ^ {*}, \boldsymbol {\theta} _ {t, \ell - 1} ^ {\mathrm {T S}} \right\rangle - \left\langle \mathbf {x} _ {t}, \boldsymbol {\theta} _ {t, \ell - 1} ^ {\mathrm {T S}} \right\rangle \leq 2 \cdot 2 ^ {- \ell + 1} \left(\sqrt {2 d \log \left(4 \left(t ^ {2} L / \delta\right) + 1\right)} \widehat {\beta} _ {t, \ell - 1}. \right. \tag {B.3}
$$
In addition, for any  $t \in \Psi_{T+1,\ell}$ , we have both  $\| \mathbf{x}_t\|_{\mathbf{V}_{t,\ell-1}^{-1}} \leq 2^{-\ell+1}$  and  $\| \mathbf{x}_t^*\|_{\mathbf{V}_{t,\ell-1}^{-1}} \leq 2^{-\ell+1}$ . It follows that
$$
\begin{array}{l} \Delta_ {t} \leq \langle \mathbf {x} _ {t} ^ {*}, \boldsymbol {\theta} _ {t, \ell - 1} ^ {\mathrm {T S}} \rangle - \langle \mathbf {x} _ {t}, \boldsymbol {\theta} _ {t, \ell - 1} ^ {\mathrm {T S}} \rangle + | \langle \mathbf {x} _ {t} ^ {*}, \boldsymbol {\theta} _ {t, \ell - 1} ^ {\mathrm {T S}} - \boldsymbol {\theta} ^ {*} \rangle | + | \langle \mathbf {x} _ {t}, \boldsymbol {\theta} _ {t, \ell - 1} ^ {\mathrm {T S}} - \boldsymbol {\theta} ^ {*} \rangle | \\ \leq 2 \cdot 2 ^ {- \ell + 1} (\sqrt {2 d \log (4 t ^ {2} L / \delta)} + 1) \widehat {\beta} _ {t, \ell - 1} + 2 \cdot 2 ^ {- \ell + 1} (\sqrt {2 d \log (4 t ^ {2} L / \delta)} + 1) \widehat {\beta} _ {t, \ell - 1} \\ \leq 8 \cdot 2 ^ {- \ell} \left(\sqrt {2 d \log \left(4 t ^ {2} L / \delta\right)} + 1\right) \hat {\beta} _ {t, \ell - 1}, \\ \end{array}
$$
where the second inequality is due to (B.3), Lemmas B.2 and B.3. Summing over  $t \in \Psi_{T+1,\ell}$ , we have
$$
\begin{array}{l} \sum_ {t \in \Psi_ {T + 1, \ell}} \Delta_ {t} \leq 8 \cdot 2 ^ {- \ell} (\sqrt {2 d \log (4 T ^ {2} L / \delta)} + 1) \widehat {\beta} _ {T, \ell - 1} | \Psi_ {T + 1, \ell} | \\ \leq 8 \cdot 2 ^ {\ell} (\sqrt {2 d \log (4 T ^ {2} L / \delta)} + 1) \widehat {\beta} _ {T, \ell - 1} \cdot \sum_ {t \in \Psi_ {T + 1, \ell}} \| w _ {t} \mathbf {x} _ {t} \| _ {\mathbf {V} _ {t, \ell} ^ {- 1}} ^ {2} \\ \leq 8 \cdot 2 ^ {\ell} \left(\sqrt {2 d \log \left(4 T ^ {2} L / \delta\right)} + 1\right) \widehat {\beta} _ {T, \ell - 1} \cdot 2 d \log \left(1 + 2 ^ {2 \ell} T / d\right), \\ \end{array}
$$
where the second inequality is due to  $\| w_t\mathbf{x}_t\|_{\mathbf{V}_{t,\ell}^{-1}} = 2^{-\ell}$  for all  $t\in \Psi_{T + 1,\ell}$ , and the last inequality is due to Lemma C.2.
B.3 Proof of Theorem 5.1
With probability at least  $1 - 2\delta$ , all the statements of Lemmas B.2 and B.3 hold, and the rest of our proof will build on such an event. Let  $\ell_0 = \lceil \frac{1}{2}\log_2\log (8(T + 1)^2 L / \delta)\rceil +8$  and we have  $\widetilde{\beta}_{t,\ell}\leq \widehat{\beta}_{t,\ell}$  for all  $t\in [T]$  and  $\ell \in [L]\backslash [\ell_0]$  following Lemma B.4. It then follows Lemma B.6 that for all  $\ell \in [L]\backslash [\ell_0]$
$$
\begin{array}{l} \sum_ {t \in \Psi_ {T + 1, \ell}} \Delta_ {t} \leq \widetilde {\mathcal {O}} \left(d ^ {3 / 2} 2 ^ {\ell} \widehat {\beta} _ {T, \ell - 1}\right) \\ \leq \widetilde {\mathcal {O}} \left(d ^ {3 / 2} \sqrt {\sum_ {t = 1} ^ {T} w _ {t} ^ {2} \left(r _ {t} - \langle \mathbf {x} _ {t} , \widehat {\boldsymbol {\theta}} _ {t + 1 , \ell} \rangle\right) ^ {2} + R ^ {2} + 1} + R\right) \\ \leq \widetilde {\mathcal {O}} \left(d ^ {3 / 2} \sqrt {\sum_ {t = 1} ^ {T} \sigma_ {t} ^ {2}} + d ^ {3 / 2} R + d ^ {3 / 2}\right), \\ \end{array}
$$
where the first inequality is due to Lemma B.6, the second inequality is due to the definition of  $\widehat{\beta}_{T,\ell -1}$  and the last inequality is due to Lemma B.4. For any layer  $\ell \in [\ell_0]$  and  $t\in \Psi_{T + 1,\ell}$ , we have
$$
\begin{array}{l} \sum_ {t \in \Psi_ {T + 1, \ell}} \Delta_ {t} \leq 2 | \Psi_ {T + 1, \ell} | \\ = 2 ^ {2 \ell + 1} \sum_ {t \in \Psi_ {T + 1, \ell}} \| w _ {t} \mathbf {x} _ {t} \| _ {\mathbf {V} _ {t, \ell}} ^ {2} \\ \leq \widetilde {\mathcal {O}} (d), \\ \end{array}
$$
where the first inequality follows from  $\langle \mathbf{x},\pmb{\theta}^*\rangle \leq 1$  for all  $\mathbf{x}\in \mathcal{X}_t$  and  $t\in [T]$ , the equality is due to  $\| w_{t}\mathbf{x}_{t}\|_{\mathbf{V}_{t,\ell}^{-1}} = 2^{-\ell}$  for all  $t\in \Psi_{T + 1,\ell}$ , and the last inequality is due to Lemma C.2 and  $2^{\ell_0}\lesssim \sqrt{\log(TL / \delta)}$ .
For any  $t \in [T] \backslash (\cup_{\ell \in [L]} \Psi_{T + 1,\ell})$ , Algorithm 3 selects an arm at some  $\ell_t$ -th layer, and it follows that
$$
\begin{array}{l} \sum_ {t \in [ T ] \backslash (\cup_ {\ell \in [ L ]} \Psi_ {T + 1, \ell})} \Delta_ {t} \leq \sum_ {t \in [ T ] \backslash (\cup_ {\ell \in [ L ]} \Psi_ {T + 1, \ell})} \left(\left\langle \mathbf {x} _ {t} ^ {*}, \boldsymbol {\theta} _ {t, \ell_ {t}} ^ {\mathrm {T S}} \right\rangle - \left\langle \mathbf {x} _ {t}, \boldsymbol {\theta} ^ {*} \right\rangle\right) \\ + \sum_ {t \in [ T ] \backslash (\cup_ {\ell \in [ L ]} \Psi_ {T + 1, \ell})} \sqrt {2 d \log (4 t ^ {2} L / \delta)} + 1) \widehat {\beta} _ {t, \ell_ {t}} \| \mathbf {x} _ {t} ^ {*} \| _ {\mathbf {V} _ {t, \ell_ {t}} ^ {- 1}} \\ \leq \sum_ {t \in [ T ] \backslash (\cup_ {\ell \in [ L ]} \Psi_ {T + 1, \ell})} \left(\left\langle \mathbf {x} _ {t}, \boldsymbol {\theta} _ {t, \ell_ {t}} ^ {\mathrm {T S}} - \boldsymbol {\theta} ^ {*} \right\rangle + \alpha \left(\sqrt {2 d \log \left(4 t ^ {2} L / \delta\right)} + 1\right) \widehat {\beta} _ {t, \ell_ {t}}\right) \\ \leq \sum_ {t \in [ T ] \backslash (\cup_ {\ell \in [ L ]} \Psi_ {T + 1, \ell})} \left(\alpha \| \boldsymbol {\theta} _ {t, \ell_ {t}} ^ {\mathrm {T S}} - \boldsymbol {\theta} ^ {*} \| _ {\mathbf {V} _ {t, \ell_ {t}}} + \alpha \left(\sqrt {2 d \log \left(4 t ^ {2} L / \delta\right)} + 1\right) \widehat {\beta} _ {t, \ell_ {t}}\right) \\ \leq \sum_ {t \in [ T ] \backslash (\cup_ {\ell \in [ L ]} \Psi_ {T + 1, \ell})} 2 \alpha \left(\sqrt {2 d \log \left(4 t ^ {2} L / \delta\right)} + 1\right) \widehat {\beta} _ {t, \ell_ {t}} \\ \leq T \cdot \widetilde {\mathcal {O}} (\sqrt {d} / T) \\ \leq \widetilde {\mathcal {O}} (\sqrt {d}), \\ \end{array}
$$
where the first inequality is due to  $\widetilde{\beta}_{t,\ell} \leq \widehat{\beta}_{t,\ell}$ , Lemmas B.2, B.3 and B.5, the second inequality is due to the arm selection rule of Algorithm 3, the third inequality is due to Cauchy-Schwarz inequality, and the fourth inequality is due to  $\widetilde{\beta}_{t,\ell} \leq \widehat{\beta}_{t,\ell}$ , Lemmas B.2 and B.3, and the fifth inequality follows from the definition of  $\alpha$  and  $\widehat{\beta}_{t,\ell_t} \leq \widetilde{\mathcal{O}}(R\sqrt{T})$ .
Combining the above inequalities together, we conclude that
$$
\begin{array}{l} \mathcal {R} (T) = \sum_ {t = 1} ^ {T} \Delta_ {t} \\ \leq \widetilde {\mathcal {O}} \left(d ^ {3 / 2} + d ^ {3 / 2} R + d ^ {3 / 2} \sqrt {\sum_ {t = 1} ^ {T} \sigma_ {t} ^ {2}}\right), \\ \end{array}
$$
where the inequality follows from  $L \lesssim \log T$  and  $\ell_0 = \lceil \frac{1}{2} \log_2 \log (8(T + 1)^2 L / \delta) \rceil + 8$ .
C Auxiliary Lemmata
Lemma C.1 (Abramowitz et al. (1964)). For a Gaussian distributed random variable  $X$  with mean  $m$  and variance  $\sigma^2$ , it holds that for any  $x \geq 1$  that
$$
\frac {1}{2 \sqrt {\pi} x} \exp (- x ^ {2} / 2) \leq \mathbb {P} (| X - m | > x \sigma) \leq \frac {1}{\sqrt {\pi} x} \exp (- x ^ {2} / 2).
$$
Lemma C.2 (Abbasi-Yadkori et al. (2011), Lemma 11). Let  $\{\mathbf{X}_t\}_{t=1}^{\infty}$  be a sequence in  $\mathbb{R}^d$ ,  $\mathbf{V} \in \mathbb{R}^{d \times d}$  positive definite matrix, and  $\overline{\mathbf{V}}_t = \mathbf{V} + \sum_{s=1}^{t} \mathbf{X}_s \mathbf{X}_s^\intercal$ . Then we have that
$$
\log \left(\frac {\operatorname* {d e t} (\overline {{\mathbf {V}}} _ {n})}{\operatorname* {d e t} (\mathbf {V})}\right) \leq \sum_ {t = 1} ^ {n} \| \mathbf {X} _ {t} \| _ {\overline {{\mathbf {V}}} _ {t - 1}} ^ {2}.
$$
Further, if  $\| \mathbf{X}_t\| _2\leq L$  for all  $t$ , then
$$
\begin{array}{l} \sum_ {t = 1} ^ {n} \min  \left\{1, \| \mathbf {X} _ {t} \| _ {\overline {{\mathbf {V}}} _ {t - 1} ^ {- 1}} ^ {2} \right\} \leq 2 (\log \det  (\overline {{\mathbf {V}}} _ {n}) - \log \det  (\mathbf {V})) \\ \leq 2 \left(d \log \left(\left(\operatorname {T r} (\mathbf {V}) + n L ^ {2}\right) / d\right) - \log \det  (\mathbf {V})\right), \\ \end{array}
$$
and finally, if  $\lambda_{\min}(\mathbf{V}) \geq \max(1, L^2)$  then
$$
\sum_ {t = 1} ^ {n} \left\| \mathbf {X} _ {t} \right\| _ {\overline {{\mathbf {V}}} _ {t - 1} ^ {- 1}} ^ {2} \leq 2 \log \frac {\det  (\overline {{\mathbf {V}}} _ {n})}{\det  (\mathbf {V})}.
$$
D Numerical Results
In our simulation, we test our algorithm on a linear contextual bandit problem with different sets of reward perturbations. Specifically, the simulation is carried out over  $T = 2000$  rounds with an ambient dimension of  $d = 25$ . For each selection round  $t \in [T]$ , the environment yields a set of contexts  $\mathcal{X}_t$ , which contains  $K = 50$  context vectors. The context vectors are randomly drawn and structured as truncated Gaussian vectors, ensuring their norms remain bounded by 1. Within each trial, a random ground truth vector  $\theta^*$  is generated as another Gaussian vector. The reward associated with every context vector is then computed based on the expected reward, perturbed by Gaussian noise. We employ our proposed algorithms, LinVDTS and LinNATS, to scrutinize their performance within this configuration.
Figure 1: Performance comparison of LinVDTS, LinNATS, and the vanilla TS algorithms on the linear contextual bandit problem. The blue (LinVDTS), orange (LinNATS), and green (vanilla TS) lines depict the average regret for each method. Dashed lines represent the average regret in a constant variance scenario, while solid lines indicate the average regret under diminishing noise variance. Confidence bands, plotted for each method, are derived from 50 random trials.
Figure Fig. 1 presents the cumulative regrets of the LinVDTS and LinNATS algorithms, using the vanilla TS as a benchmark. The algorithms are evaluated in two distinct bandit environments: one characterized by constant variance and the other by a quadratically decaying variance. Performance under the constant variance setup is captured by dashed lines, whereas the decaying variance scenario is depicted by solid lines, with the observations being grounded on 50 randomized trials. It is noteworthy that while LinVDTS is privy to the ground truth variance, LinNATS operates without this knowledge in our experimental framework. This distinction underscores LinNATS's commendable adaptability in environments where the variance is elusive. Given its access to variance data, the superior performance of LinVDTS vis-a-vis LinNATS in fluctuating variance situations is anticipated. The presented outcomes showcase the variance-awareness of our proposed algorithms and the inherent flexibility of these algorithms in modulating their regret according to the prevailing variance dynamics.
Future directions and broader impact. The recent progress in understanding the training dynamics of deep neural networks (Jacot et al., 2018; Song et al., 2021) has paved the way for investigations at the intersection of deep learning and reinforcement learning (Lillicrap et al., 2015; Chen et al., 2021; Xu et al., 2021). This integrative approach promises to yield transformative insights and substantially enhance the performance of existing learning architectures. Further, the incorporation of non-linear function approximation into our frameworks emerges as a pivotal research direction,
with the potential to significantly improve their robustness (Ling et al., 2019; Chen et al., 2020; Min et al., 2021a; Ye et al., 2023). We believe that engaging in these research endeavors is important for the refinement of our methodologies.