SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Concretely, suppose  $X_{t}$  falls in bin  $B$  at level  $r$ , and let  $\pi_{\mathrm{t}}^{\mathrm{oracle}}(B)$  be the arm selected at round  $t$  by  $\pi_{\mathrm{t}}^{\mathrm{oracle}}$  in bin  $B$ . Then, as mean rewards are Lipschitz, each policy  $\pi_{\mathrm{t}}^{\mathrm{oracle}}$  suffers regret:
$$
\max  _ {a \in [ K ]} f _ {t} ^ {a} (X _ {t}) - f _ {t} ^ {\pi_ {\mathrm {t}} ^ {\mathrm {o r a l e}} (B)} (X _ {t}) \leq \max  _ {a \in [ K ]} \mathbb {E} \left[ f _ {t} ^ {a} (X _ {t}) - f _ {t} ^ {\pi_ {\mathrm {t}} ^ {\mathrm {o r a l e}} (B)} (X _ {t}) \mid X _ {t} \in B \right] + r = r.
$$
Thus, the sequence of policies  $\{\pi_{\mathrm{t}}^{\mathrm{oracle}}\}_{t}$  achieves dynamic regret (in the sense of Definition 2)
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \max  _ {a \in [ K ]} f _ {t} ^ {a} (X _ {t}) - f _ {t} ^ {\pi_ {\mathrm {t}} ^ {\mathrm {o r a c l e}} (X _ {t})} (X _ {t}) \right] \lesssim (L + 1) \cdot \left(\frac {T}{L + 1}\right) \cdot \left(\frac {K}{(L + 1) T}\right) ^ {\frac {1}{2 + d}} \propto (L + 1) ^ {\frac {1}{2 + d}} \cdot T ^ {\frac {1 + d}{2 + d}}.
$$
Thus, it suffices to minimize dynamic regret in the sense of (8) to this oracle policy  $\pi_t^{\mathrm{oracle}}$ . The state-of-the-art adaptive guarantee in this setting is that of the ADA-ILTCB algorithm of Chen et al. [2019], which achieves a dynamic regret of  $\sqrt{KLT\log(|\Pi|)}$ . Thus, it remains to compute  $|\Pi|$ .
We first observe that we need only consider levels in  $\mathcal{R}$  of size at least  $(K / T)^{\frac{1}{2 + d}}$ , which is the oracle choice of level for one stationary phase of length  $T$ . Thus, the size of the policy class  $\Pi$  is
$$
| \Pi | = \sum_ {r \in \mathcal {R}} K ^ {r ^ {- d}} \propto K ^ {(T / K) ^ {\frac {d}{2 + d}}} \Rightarrow \log (| \Pi |) = \left(\frac {T}{K}\right) ^ {\frac {d}{2 + d}} \log (K).
$$
Plugging this into  $\sqrt{KLT\log(|\Pi|)}$  gives a regret rate of  $K^{\frac{1}{2 + d}}\cdot L^{1 / 2}T^{\frac{1 + d}{2 + d}}$ , which has a worse dependence on the global number of shifts  $L$  than the minimax optimal rate of  $L^{\frac{1}{2 + d}}\cdot T^{\frac{1 + d}{2 + d}}$  (see Theorem 1).
2While there are matters of efficiency and what offline learning guarantees may be assumed in this broader agnostic setting, we do not discuss these here, and readers are deferred to Langford and Zhang [2008], Dudik et al. [2011], Agarwal et al. [2014].
- Parametrizing in Terms of Total-Variation  $V_{T}$ . Fix any positive real number  $V \in [T^{-\frac{3 + d}{2 + d}}, T]$ . Then, the lower bound construction of Theorem 1 reveals that there exists an environment with  $L + 1 = T / \Delta$  stationary phases of length  $\Delta \doteq \left\lceil \left(\frac{T}{V}\right)^{\frac{2 + d}{3 + d}} \right\rceil$  and total-variation of order  $V$ .
Then, the earlier defined oracle sequence of policies  $\{\pi_{t}^{\mathrm{oracle}}\}_{t}$  attains the optimal dynamic regret rate in terms of  $V_{T}$  (see Theorem 1) since
$$
(L + 1) ^ {\frac {1}{2 + d}} \cdot T ^ {\frac {1 + d}{2 + d}} \propto T ^ {\frac {2 + d}{3 + d}} \cdot V ^ {\frac {1 + d}{3 + d}}.
$$
Meanwhile, the state-of-the-art adaptive regret guarantee in this parametrization is Theorem 2 of Chen et al. [2019], which shows ADA-ILTCB's regret bound is:
$$
\left(K \cdot \log (| \Pi |) \cdot V\right) ^ {1 / 3} T ^ {2 / 3} + \sqrt {K \log (| \Pi |) \cdot T} \propto K ^ {\frac {2}{3 (2 + d)}} \cdot V ^ {\frac {1}{3}} \cdot T ^ {\frac {2 + d}{3 + d} + \frac {d}{3 (2 + d) (3 + d)}} + K ^ {\frac {1}{2 + d}} \cdot T ^ {\frac {1 + d}{2 + d}}.
$$
We claim this rate is no better than our rate in Corollary 5, in all parameters  $V, K, T$ . For  $K \geq T$ , both rates imply linear regret. Assume  $K < T$ . Then, note by elementary calculations that for all  $d \in \mathbb{N} \cup \{0\}$ :
$$
\frac {2}{3} + \frac {d}{3 (2 + d)} = \frac {2 + d}{3 + d} + \frac {1}{3 + d} - \frac {2}{3 (2 + d)}.
$$
Then, it follows that rate of Corollary 5 is smaller using the fact that  $K < T$ :
$$
K ^ {\frac {2}{3 (2 + d)}} \cdot V ^ {1 / 3} \cdot T ^ {\frac {2}{3} + \frac {d}{3 (2 + d)}} \geq K ^ {\frac {2}{3 (2 + d)}} \cdot V ^ {\frac {1}{3 + d}} \cdot T ^ {\frac {2 + d}{3 + d}} \cdot K ^ {\frac {1}{3 + d} - \frac {2}{3 (2 + d)}} \geq (K V) ^ {\frac {1}{3 + d}} \cdot T ^ {\frac {2 + d}{3 + d}}.
$$
A.2 Realizable Contextual Bandits
Lipschitz contextual bandits is also a special case of contextual bandits with realizability. In this broader setting, the learner is given a function class  $\Phi$  which contains the true regression function  $\phi_t^*:\mathcal{X}\times [K]\to [0,1]$  describing mean rewards of context-arm pairs at round  $t$ . The goal is to compete with the time-varying benchmark of policies  $\pi_{\phi_t^*}(x)\doteq \operatorname{argmax}_{a\in [K]}\phi_t^* (x,a)$ , using calls to a regression oracle over  $\Phi$ .
While the natural choice for  $\Phi$  is the infinite class of all Lipschitz functions from  $\mathcal{X} \times [K] \to [0,1]$ , the state-of-the-art non-stationary algorithm only provides guarantees for finite  $\Phi$  [Wei and Luo, 2021, Appendix I.7].
However, it is still possible to recover the Lipschitz contextual bandit setting, by defining  $\Phi$  similarly to how we defined the finite class of policies  $\Pi$  above. Let  $\Phi$  be the class of all piecewise constant functions which depends on a level  $r\in \mathcal{R}$ , and are constant on bins  $B\in \mathcal{T}_r$  at level  $r$ , taking values which are multiples of  $T^{-\frac{1}{2 + d}}$  (there are  $O(T)$  many such values in [0, 1]). Here,  $\Phi$  is essentially the class of different discretization-based regression estimates for the true mean rewards, as appears in prior works [Rigollet and Zeevi, 2010, Perchet and Rigollet, 2013].
For this specification of  $\Phi$ , the realizability assumption is false. Rather, this is a mildly misspecified regression class which is allowed by the stationary guarantees of FALCON [Simchi-Levi and Xu, 2021, Section 3.2]. In particular, by smoothness, at each round  $t \in [T]$  there is a function  $\phi_t^* \in \Phi$  such that
$$
\sup_{x\in \mathcal{X},a\in [K]}\left|\phi_{t}^{*}(x,a) - f_{t}^{a}(x)\right|\lesssim \left(\frac{1}{T}\right)^{\frac{1}{2 + d}}.
$$
Specifically, we can let  $\phi_t^* (x,a)\propto \mathbb{E}_X[f_t^a (X)|X\in B]$  be the smoothed version of  $f_{t}^{a}(x)$  in the bin  $B$  at level  $T^{-\frac{1}{2 + d}}$  containing  $x$ . Then, the above misspecification introduces an additive term in the regret bound of FALCON of order  $T^{\frac{1 + d}{2 + d}}$  which is of the right order in our setting.
In this setting, the current state-of-the-art MASTER black-box algorithm using FALCON Simchi-Levi and Xu [2021] as a base algorithm can obtain dynamic regret upper bounded by [see Wei and Luo, 2021, Theorem 2]:
$$
\min  \left\{\sqrt {\log (| \Phi |) \cdot L \cdot T}, \log^ {1 / 3} (| \Phi |) \cdot \Delta^ {1 / 3} \cdot T ^ {2 / 3} + \sqrt {\log (| \Phi |) \cdot T} \right\}.
$$
As  $\Phi$  is essentially the same size as the policy class  $\Pi$  defined in the previous section, the above regret bound specializes to similar rates as those of ADA-ILTCB derived above, and are ultimately suboptimal in light of Theorem 1.
Throughout the appendix,  $c_{1}, c_{2}, \ldots$  will denote universal positive constants not depending on  $T$ ,  $K$  or any of the significant shifts  $\{\tau_{i}(\mathbf{X}_{T})\}_{i}$ .
B.1 Concentration of Aggregate Gap over an Interval within a Bin
We'll first establish some concentration bounds for the local gap estimators  $\hat{\delta}_s^B (a',a)$  defined in (4). For this purpose, we recall Freedman's inequality.
Lemma 6 (Theorem 1 of Beygelzimer et al. [2011]). Let  $X_{1},\ldots ,X_{n}\in \mathbb{R}$  be a martingale difference sequence with respect to some filtration  $\mathcal{F}_0,\mathcal{F}_1,\dots$  Assume for all  $t$  that  $X_{t}\leq R$  a.s.. Then for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ , we have:
$$
\sum_ {i = 1} ^ {n} X _ {i} \leq (e - 1) \left(\sqrt {\log (1 / \delta) \sum_ {i = 1} ^ {n} \mathbb {E} \left[ X _ {i} ^ {2} \mid \mathcal {F} _ {t - 1} \right]} + R \log (1 / \delta)\right). \tag {9}
$$
Recall from Section 4 that for round  $t$ , the local gap estimate  $\hat{\delta}_t^B(a', a)$  in bin  $B$  at round  $t$  between arms  $a', a$  is:
$$
\hat {\delta} _ {t} ^ {B} \left(a ^ {\prime}, a\right) \doteq \left| \mathcal {A} _ {t} \right| \cdot \left(Y _ {t} \left(a ^ {\prime}\right) \cdot \mathbf {1} \left\{\pi_ {t} = a ^ {\prime} \right\} - Y _ {t} (a) \cdot \mathbf {1} \left\{\pi_ {t} = a \right\}\right) \cdot \mathbf {1} \left\{a \in \mathcal {A} _ {t} \right\} \cdot \mathbf {1} \left\{X _ {t} \in B \right\}.
$$
We next apply Lemma 6 to our aggregate estimator from Section 4.
Proposition 7. With probability at least  $1 - 1 / T^2$  w.r.t. the randomness of  $\mathbf{Y}_T$ ,  $\{\pi_t\}_t \mid \mathbf{X}_T$ , we have for all bins  $B \in \mathcal{T}$  and rounds  $s_1 < s_2$  and all arms  $a \in [K]$  that for large enough  $c_1 > 0$ :
$$
\left| \sum_ {s = s _ {1}} ^ {s _ {2}} \hat {\delta} _ {s} ^ {B} \left(a ^ {\prime}, a\right) - \sum_ {s = s _ {1}} ^ {s _ {2}} \mathbb {E} \left[ \hat {\delta} _ {s} ^ {B} \left(a ^ {\prime}, a\right) \mid \mathcal {F} _ {s - 1} \right] \right| \leq c _ {1} \left(\sqrt {K \log (T) \cdot n _ {B} ([ s _ {1} , s _ {2} ])} + K \log (T)\right), \tag {10}
$$
where  $\mathcal{F} \doteq \{\mathcal{F}_t\}_{t=1}^T$  is the filtration with  $\mathcal{F}_t$  generated by  $\{\pi_s, Y_s^{\pi_s}\}_{s=1}^t$ .
Proof. The martingale difference  $\hat{\delta}_s^B (a',a) - \mathbb{E}[\hat{\delta}_s^B (a',a)\mid \mathcal{F}_{s - 1}]$  is clearly bounded above by  $2K$  for all bins  $B$ , rounds  $s$ , and all arms  $a,a'$ . We also have a cumulative variance bound:
$$
\begin{array}{l} \sum_ {s = s _ {1}} ^ {s _ {2}} \mathbb {E} \left[ \left(\hat {\delta} _ {s} ^ {B} \left(a ^ {\prime}, a\right)\right) ^ {2} \mid \mathcal {F} _ {s - 1} \right] \leq \sum_ {s = s _ {1}} ^ {s _ {2}} \mathbf {1} \left\{X _ {s} \in B \right\} \cdot \left| \mathcal {A} _ {s} \right| ^ {2} \cdot \mathbb {E} \left[ \mathbf {1} \left\{\pi_ {s} = a \text {o r} a ^ {\prime} \right\} \mid \mathcal {F} _ {s - 1} \right] \\ \leq \sum_ {s = s _ {1}} ^ {s _ {2}} \mathbf {1} \left\{X _ {s} \in B \right\} \cdot 2 | \mathcal {A} _ {s} | \\ \leq 2 K \cdot n _ {B} ([ s _ {1}, s _ {2} ]). \\ \end{array}
$$
Then, the result follows from (9), and taking union bounds over bins  $B$  (note there are at most  $T$  levels and at most  $T$  bins per level), arms  $a, a'$ , and rounds  $s_1, s_2$ .
Since the error probability of Proposition 7 is negligible with respect to regret, we assume going forward in the analysis that (10) holds for all arms  $a, a' \in [K]$  and rounds  $s_1, s_2$ . Specifically, let  $\mathcal{E}_1$  be the good event over which the bounds of Proposition 7 hold for all arms and intervals  $[s_1, s_2]$ .
B.2 Concentration of Context Counts
We'll next establish concentration w.r.t. the distribution of contexts  $\mathbf{X}_T$ . This will ensure that all bins  $B \in \mathcal{T}$  have sufficient observed data.
Notation. To ease notation throughout the analysis, we'll henceforth use  $\mu(\cdot)$  to refer to the context marginal distribution  $\mu_X(\cdot)$ .
Lemma 8. Let  $\{i_t\}_{t=1}^T$  be a random sequence of arms whose distribution depends on  $X_T$ . With probability at least  $1 - 1 / T^2$  w.r.t. the randomness of  $X_T$ , we have for all bins  $B \in \mathcal{T}$ , all arms  $a', a \in [K]$ , and rounds  $s_1 < s_2$ , for some large enough  $c_2 > 0$  the following inequalities hold:
$$
\left| n _ {B} \left(\left[ s _ {1}, s _ {2} \right]\right) - \left(s _ {2} - s _ {1} + 1\right) \cdot \mu (B) \right| \leq c _ {2} \left(\log (T) + \sqrt {\log (T) \mu (B) \cdot \left(s _ {2} - s _ {1} + 1\right)}\right) \tag {11}
$$
$$
\left| \sum_ {s = s _ {1}} ^ {s _ {2}} \delta_ {s} \left(i _ {s}, a\right) \cdot \left(\mathbf {1} \left\{X _ {s} \in B \right\} - \mu_ {s} (B)\right) \right| \leq c _ {2} \left(\log (T) + \sqrt {\log (T) \mu (B) \cdot \left(s _ {2} - s _ {1} + 1\right)}\right) \tag {12}
$$
$$
\left| \sum_ {s = s _ {1}} ^ {s _ {2}} \delta_ {s} (a) \cdot \left(\mathbf {1} \left\{X _ {s} \in B \right\} - \mu_ {s} (B)\right) \right| \leq c _ {2} \left(\log (T) + \sqrt {\log (T) \mu (B) \cdot \left(s _ {2} - s _ {1} + 1\right)}\right) \tag {13}
$$
Proof. The first inequality (11) follows from Lemma 6 since  $\sum_{s=s_1}^{s_2} \mathbf{1}\{X_s \in B\} - \mu(B)$  is a martingale, which has predictable quadratic variation is at most  $(s_2 - s_1 + 1) \cdot \mu(B)$ .
The other two inequalities are trickier since the corresponding sums are not necessarily martingales. Indeed, note  $\delta_s(a)$  depends on  $X_{s}$  while  $\delta_s(i_s,a)$  may not even be adapted to the canonical filtration generated by  $\mathbf{X}_T$  (i.e.,  $i_{s}$  may depend on  $X_{t}$  for  $t > s$ ). Nevertheless, we observe that for any random variable  $W_{s} = W_{s}(\mathbf{X}_{T})\in [-1,1]$ :
$$
- \left(\mathbf {1} \left\{X _ {s} \in B \right\} - \mu (B)\right) \leq W _ {s} \cdot \left(\mathbf {1} \left\{X _ {s} \in B \right\} - \mu (B)\right) \leq \mathbf {1} \left\{X _ {s} \in B \right\} - \mu (B).
$$
The upper and lower bounds above are both martingale differences with respect to the canonical filtration of  $\mathbf{X}_T$  and thus, summing the above over  $s$  we have via Lemma 6:
$$
\begin{array}{l} \left| \sum_ {s = s _ {1}} ^ {s _ {2}} W _ {s} \cdot \left(\mathbf {1} \{X _ {t} \in B \} - \mu (B)\right) \right| \leq \left| \sum_ {s = s _ {1}} ^ {s _ {2}} \mathbf {1} \{X _ {s} \in B \} - \mu (B) \right| \\ \leq c _ {2} \left(\log (T) + \sqrt {\log (T) \mu (B) \cdot \left(s _ {2} - s _ {1} + 1\right)}\right). \\ \end{array}
$$
Then, taking union bounds over rounds  $s_1, s_2$ , bins  $B \in \mathcal{T}$ , and arms  $a \in [K]$  gives the result.
Notation 2 (good event). Recall from earlier that  $\mathcal{E}_1$  is the good event over which the bounds of Proposition 7 hold for all rounds  $s_1, s_2 \in [T]$  and arms  $a', a \in [K]$ . Thus, on  $\mathcal{E}_1$ , our estimated gaps in each bin will concentrate around their conditional means.
Let  $\mathcal{E}_2$  be the good event on which bounds of Lemma 8 hold for all bins  $B$ , arms  $a \in [K]$ , rounds  $s_1, s_2 \in [T]$ . Thus, on  $\mathcal{E}_2$ , our covariate counts  $n_B([s_1, s_2])$  will concentrate and we will be able to relate the empirical quantities  $\sum_{s=s_1}^{s_2} \delta_s(a) \cdot \mathbf{1}\{X_s \in B\}$  with their expectations.
Next, we establish a lemma which allows us to relate significant regret  $(\star)$  and thus our eviction criterion (5) between different bins and levels.
Lemma 9 (Relating Aggregate Gaps Between Levels). On event  $\mathcal{E}_2$ , if for rounds  $s_1 < s_2$ , bin  $B'$  at level  $r_{s_2 - s_1}$  and arm  $a$ , for some  $c_3 > 0$ :
$$
\sum_ {s = s _ {1}} ^ {s _ {2}} \delta_ {s} (a) \cdot \mathbf {1} \{X _ {s} \in B ^ {\prime} \} \leq c _ {3} \left(\sqrt {K \log (T) \cdot \left(n _ {B ^ {\prime}} ([ s _ {1} , s _ {2} ]) \vee K \log (T)\right)} + r (B ^ {\prime}) \cdot n _ {B ^ {\prime}} ([ s _ {1}, s _ {2} ])\right),
$$
then for any bin  $B \subseteq B'$  and some  $c_4 > 0$ :
$$
\begin{array}{l} \sum_ {s = s _ {1}} ^ {s _ {2}} \delta_ {s} (a) \cdot \mathbf {1} \{X _ {s} \in B \} \leq c _ {4} \left(\log^ {1 / 2} (T) \cdot r (B) ^ {d} \cdot K ^ {\frac {1}{2 + d}} \cdot \left(s _ {2} - s _ {1}\right) ^ {\frac {1 + d}{2 + d}} \right. \\ \left. + K \log (T) + \sqrt {\log (T) \mu (B) \left(s _ {2} - s _ {1} + 1\right)}\right). \\ \end{array}
$$
The same applies for  $\delta_s(a)$  replaced with  $\delta_s(a', a)$  for any fixed arm  $a' \in [K]$ .
Proof. We have using (13) and the strong density assumption (Assumption 2):
$$
\begin{array}{l} \sum_ {s = s _ {1}} ^ {s _ {2}} \delta_ {s} (a) \cdot \mathbf {1} \{X _ {s} \in B \} \leq \sum_ {s = s _ {1}} ^ {s _ {2}} \delta_ {s} (a) \cdot \mu (B) \\ + c _ {2} \left(\log (T) + \sqrt {\log (T) \mu (B) \cdot \left(s _ {2} - s _ {1} + 1\right)}\right) \\ \leq \frac {C _ {d} \cdot r (B) ^ {d}}{c _ {d} \cdot r \left(B ^ {\prime}\right) ^ {d}} \sum_ {s = s _ {1}} ^ {s _ {2}} \delta_ {s} (a) \cdot \mu \left(B ^ {\prime}\right) \\ + c _ {2} \left(\log (T) + \sqrt {\log (T) \mu (B) \cdot \left(s _ {2} - s _ {1} + 1\right)}\right) \tag {14} \\ \end{array}
$$
Again using (13)
$$
\begin{array}{l} \sum_ {s = s _ {1}} ^ {s _ {2}} \delta_ {s} (a) \cdot \mu_ {s} \left(B ^ {\prime}\right) \leq \sum_ {s = s _ {1}} ^ {s _ {2}} \delta_ {s} (a) \cdot \mathbf {1} \left\{X _ {s} \in B ^ {\prime} \right\} + c _ {2} \left(\log (T) + \sqrt {\log (T) \mu \left(B ^ {\prime}\right) \cdot \left(s _ {2} - s _ {1} + 1\right)}\right) \\ \leq c _ {5} \left(\sqrt {K \log (T) \cdot \left(n _ {B ^ {\prime}} ([ s _ {1} , s _ {2} ]) \vee K \log (T)\right)} + r \left(B ^ {\prime}\right) \cdot n _ {B ^ {\prime}} ([ s _ {1}, s _ {2} ]) \right. \\ \left. + \log (T) + \sqrt {\log (T) \mu \left(B ^ {\prime}\right) \cdot \left(s _ {2} - s _ {1} + 1\right)}\right). \\ \end{array}
$$
Next, applying (11) to  $n_{B'}([s_1, s_2])$  and using the strong density assumption (Assumption 2) to bound the mass  $\mu(B')$  above by  $C_d \cdot r(B')^d$ , the above R.H.S. is further upper bounded by
$$
c _ {6} \left(\log^ {1 / 2} (T) K ^ {\frac {1 + d}{2 + d}} \cdot \left(s _ {2} - s _ {1}\right) ^ {\frac {1}{2 + d}} + K \log (T)\right). \tag {15}
$$
Finally, plugging (15) into (14) and using the fact that  $(r(B') / 2)^d \geq (K / (s_2 - s_1))^{\frac{d}{2 + d}}$ , we have that (14) is of the desired order. The proof of the same inequalities with  $\delta_s(a', a)$  is analogous.
The following lemma relating the bias and variance terms in the notion of significant regret  $(\star)$  will serve useful many places in the analysis. They all follow from concentration and similar calculations via the strong density assumption (Assumption 2) as done previously.
Lemma 10 (Relating Bias and Variance Error Terms via Strong Density Assumption). Let  $r \doteq r_{s_2 - s_1}$  for some  $s_2 > s_1$ . Then, on event  $\mathcal{E}_2$ , for any bin  $B \in T_r$ :
$$
\begin{array}{l} \sqrt {(s _ {2} - s _ {1} + 1) \cdot \mu (B)} \leq c _ {7} (s _ {2} - s _ {1}) ^ {\frac {1}{2 + d}} \cdot K ^ {\frac {d / 2}{2 + d}} \\ \sqrt {n _ {B} ([ s _ {1} , s _ {2} ])} \leq c _ {8} \left((s _ {2} - s _ {1}) ^ {\frac {1}{2 + d}} \cdot K ^ {\frac {d / 2}{2 + d}} + \log^ {1 / 2} (T) \right. \\ \left. + \log^ {1 / 4} (T) \cdot K ^ {\frac {d / 4}{2 + d}} \cdot \left(s _ {2} - s _ {1}\right) ^ {\frac {1 / 2}{2 + d}}\right) \\ \end{array}
$$
B.3 Useful Facts about Levels  $r \in \mathcal{R}$  and their Durations in Play
The following basic facts about the level selection procedure on Line 2 of Algorithm 2 will be useful as we will decompose the analysis into the blocks, or different periods of rounds, where different levels are used.
Definition 8. Namely, for  $r \in \mathcal{R}$ , let  $s_{\ell}(r)$  and  $e_{\ell}(r)$  denote the first and last rounds when level  $r$  is used by the master Base-Alg in episode  $[t_{\ell}, t_{\ell+1})$ , i.e. rounds  $t \in [t_{\ell}, t_{\ell+1})$  such that  $r_{t - t_{\ell}} = r$ . Then, we call  $[s_{\ell}(r), e_{\ell}(r)]$  a block.
The proofs of the following facts all follow from the definition of the level  $r_n$  (see Notation 1) and basic calculations.
Fact 1 (Relating Level to Interval Length). The level  $r_{s_2 - s_1} = 2^{-m}$  satisfies for  $s_2 - s_1 \geq K$ :
$$
2 ^ {- (m - 1)} > \left(\frac {K}{s _ {2} - s _ {1}}\right) ^ {\frac {1}{2 + d}} \geq 2 ^ {- m},
$$
$$
K \cdot 2 ^ {(m - 1) (2 + d)} <   s _ {2} - s _ {1} \leq K \cdot 2 ^ {m (2 + d)}.
$$
Fact 2 (First Block). The first block  $[s_{\ell}(1), e_{\ell}(1)]$  consists of rounds  $[t_{\ell}, t_{\ell} + K]$ .
Fact 3 (Start and End Times of a Block). For  $r < 1$ , the start time or first round  $s_{\ell}(r)$  of the block corresponding to level  $r$  in episode  $[t_{\ell}, t_{\ell + 1})$  is  $s_{\ell}(r) = t_{\ell} + \left[ K \cdot (2r)^{-(2 + d)} \right]$  and the anticipated end time, or last round of the block if no new episode is triggered in said block, is  $e_{\ell}(r) = t_{\ell} + \left[ K \cdot r^{-(2 + d)} \right] - 1$ .
Fact 4 (Length of a Block). Each block  $[s_{\ell}(r), e_{\ell}(r)]$  is at least  $K$  rounds long. For the first block  $[s_{\ell}(1), e_{\ell}(1)]$ , this is already clear. Otherwise, suppose  $r < 1$  in which case:
$$
\begin{array}{l} e _ {\ell} (r) - s _ {\ell} (r) + 1 = \left[ K \cdot r ^ {- (2 + d)} \right] - \left[ K \cdot (2 r) ^ {- (2 + d)} \right] \\ \geq K \cdot r ^ {- (2 + d)} (1 - 2 ^ {- (2 + d)}) - 1 \\ \geq K \cdot 2 ^ {2 + d} (1 - 2 ^ {- (2 + d)}) - 1. \\ \end{array}
$$
In particular, since  $2^{2 + d}(1 - 2^{-(2 + d)}) \geq 2$  for all  $d \geq 0$ , we have the above is at least  $K$ .
We also have the above implies
$$
2 \cdot \left(e _ {\ell} (r) - s _ {\ell} (r)\right) \geq \frac {K \cdot r ^ {- (2 + d)} \cdot \left(1 - 2 ^ {- (2 + d)}\right)}{2}.
$$
Rearranging, this becomes for some constant  $c_9 > 0$  depending only on  $d$ :
$$
c _ {9} ^ {- 1} \cdot r \leq \left(\frac {K}{e _ {\ell} (r) - s _ {\ell} (r)}\right) ^ {\frac {1}{2 + d}} <   c _ {9} \cdot r.
$$
Note we can make  $c_9$  large enough so that the above also holds for level  $r = 1$ .
The above along with the definition of  $r_{t - t_\ell}$  (see Notation 1) implies that the block length  $e_{\ell}(r) - s_{\ell}(r)$  and the episode length  $e_{\ell}(r) - t_{\ell}(r)$  up to the end of block  $[s_{\ell}(r), e_{\ell}(r)]$  can be conflated up to constants
$$
c _ {1 0} ^ {- 1} \cdot \left(e _ {\ell} (r) - s _ {\ell} (r)\right) \leq e _ {\ell} (r) - t _ {\ell} \leq c _ {1 0} \cdot \left(e _ {\ell} (r) - s _ {\ell} (r)\right).
$$
C Proof of Oracle Regret Bound (Proposition 2)
Recall that  $\mathcal{E}_2$  is the good event on which our covariate counts concentrate by Lemma 8. It suffices to show our desired regret bound for any fixed context sequence  $\mathbf{X}_T$  on this event.
Fix a phase  $[\tau_i, \tau_{i+1})$  and let  $r \doteq r_{\tau_{i+1} - \tau_i}$ . Fix a bin  $B \in \mathcal{T}_r$  and let  $\tau_i^a$  be the last round  $t \in [\tau_i, \tau_{i+1})$  such that  $X_t \in B$  and arm  $a$  is included in  $\mathcal{G}_t$ . If  $a$  is never excluded from  $\mathcal{G}_t$  for all such  $t$ , let  $\tau_i^a \doteq \tau_{i+1} - 1$ . WLOG suppose  $\tau_i^1 \leq \tau_i^2 \leq \dots \leq \tau_i^K$ . Then, letting  $B'$  be the bin at level  $r_{\tau_i^a - \tau_i}$  containing covariate  $X_{\tau_i^a}$ , we have by ( $\star$ ) that:
$$
\sum_ {t = \tau_ {i}} ^ {\tau_ {i} ^ {a}} \delta_ {t} (a) \cdot \mathbf {1} \{X _ {t} \in B ^ {\prime} \} \leq \sqrt {K \cdot n _ {B ^ {\prime}} ([ \tau_ {i} , \tau_ {i} ^ {a} ])} + r (B ^ {\prime}) \cdot n _ {B ^ {\prime}} ([ \tau_ {i}, \tau_ {i} ^ {a} ]).
$$
From Lemma 9, we conclude  $\sum_{t = \tau_i}^{\tau_i^a}\frac{\delta_t(a)\mathbf{1}\{X_t\in B\}}{|\mathcal{G}_t|}$  is at most
$$
\frac {c _ {4} \left(\log^ {1 / 2} (T) r ^ {d} K ^ {\frac {1}{2 + d}} (\tau_ {i + 1} ^ {a} - \tau_ {i}) ^ {\frac {1 + d}{2 + d}} + K \log (T) + \sqrt {\log (T) \mu (B) (\tau_ {i} ^ {a} - \tau_ {i} + 1)}\right)}{K + 1 - a},
$$
where we use the fact that  $|\mathcal{G}_t| \geq K + 1 - a$  for  $t \leq \tau_i^a$  such that  $X_t \in B$ . Summing over arms  $a \in [K]$  with  $\sum_{a \in [K]} \frac{1}{K + 1 - a} \leq \log(K)$ , we obtain from summing the above over  $K$ :
$$
c _ {4} \log (K) \left(\log^ {1 / 2} (T) r ^ {d} K ^ {\frac {1}{2 + d}} \left(\tau_ {i + 1} - \tau_ {i}\right) ^ {\frac {1 + d}{2 + d}} + K \log (T) + \sqrt {\log (T) \mu (B) \cdot \left(\tau_ {i} ^ {a} - \tau_ {i} + 1\right)}\right). \tag {16}
$$
Next, we claim that each significant phase  $\left[\tau_{i},\tau_{i + 1}\right)$  is at least  $K$  rounds long or  $K\leq \tau_{i + 1} - \tau_{i}$ . This follows from the definition of significant regret  $(\star)$  since for  $[s_1,s_2]\subseteq [\tau_i,\tau_{i + 1})$ :
$$
n _ {B} ([ s _ {2}, s _ {2} ]) \geq \sum_ {s = s _ {1}} ^ {s _ {2}} \delta_ {s} (a) \cdot \mathbf {1} \{X _ {s} \in B \} \geq \sqrt {K \cdot n _ {B} ([ s _ {1} , s _ {2} ])} \Rightarrow \tau_ {i + 1} - \tau_ {i} \geq n _ {B} ([ s _ {1}, s _ {2} ]) \geq K.
$$
Then  $K \leq \tau_{i+1} - \tau_i$  implies (via Fact 1 about the level  $r_{\tau_{i+1} - \tau_i}$ )