SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {s \in I} \delta_ {s} ^ {a} \left(X _ {\tau_ {i}}\right) \cdot \mathbf {1} \left\{X _ {s} \in B \right\} \geq \sum_ {s \in I} \delta_ {s} (a) \cdot \mathbf {1} \left\{X _ {s} \in B \right\} - r (B) \sum_ {s \in I} \mathbf {1} \left\{X _ {s} \in B \right\} > 0.
$$
Thus,  $\tilde{L} \leq L + 1$ , the global count of shifts, and can in fact be much smaller. On the other hand, so long as an experienced significant shift does not occur, there will be arms safe to play at each context  $X_{t}$ . Thus, procedures need not restart exploration so long as unsafe arms can be quickly ruled out.
As a warmup to presenting our main regret bounds and algorithms, we'll first consider an oracle procedure which knows the experienced significant shift times  $\tau_{i}$ . The strategy will be to mimic a successive elimination algorithm, restarted at each experienced significant shift.
Definition 7 (Oracle Procedure). For each round  $t$  in phase  $[\tau_i, \tau_{i+1})$ , define a good arm set  $\mathcal{G}_t$  as the set of safe arms, i.e., arms which do not yet satisfy  $(\star)$  in the bin  $T_r(X_t)$  at level  $r = r_{\tau_{i+1} - \tau_i}$  containing  $X_t$ . Here, recall from Subsection 2.3 that  $r_{\tau_{i+1} - \tau_i}$  is the oracle choice of level over phase  $[\tau_i, \tau_{i+1})$ . Then, define an oracle procedure  $\pi$ : playing a uniformly random arm  $a \in \mathcal{G}_t$  at time  $t$ .
We then claim such an oracle procedure attains an enhanced dynamic regret rate in terms of the significant shifts  $\{\tau_i\}_i$  which recovers the minimax lower bound in terms of global shifts  $L$  and total variation  $V_{T}$  from before. The following proposition (see Appendix C for proof) captures this.
Proposition 2 (Sanity Check). We have the oracle procedure  $\pi$  of Definition 7 satisfies with probability at least  $1 - 1 / T^2$  w.r.t. the randomness of  $X_{T}$ : for some  $C > 0$
$$
\mathbb {E} _ {\pi} \left[ R _ {T} \left(\pi , \boldsymbol {X} _ {T}\right) \mid \boldsymbol {X} _ {T} \right] \leq C \log (K) \log (T) \sum_ {i = 0} ^ {\tilde {L} \left(\boldsymbol {X} _ {T}\right)} \left(\tau_ {i + 1} \left(\boldsymbol {X} _ {T}\right) - \tau_ {i} \left(\boldsymbol {X} _ {T}\right)\right) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}}.
$$
By Jensen's inequality the above regret rate is at most  $(\tilde{L}(\mathbf{X}_T) + 1)^{\frac{1}{2 + d}} \cdot T^{\frac{1 + d}{2 + d}} \ll (L + 1)^{\frac{1}{2 + d}} \cdot T^{\frac{1 + d}{2 + d}}$ . At the same time, the rate is also faster than  $V_T^{\frac{1}{3 + d}} T^{\frac{2 + d}{3 + d}}$  (see Corollary 5). Thus, the oracle procedure above attains the dynamic regret lower bound of Theorem 1. We next aim to design an algorithm which can attain the same order regret without knowledge of  $\tau_i$  or  $\tilde{L}$ .
3.3 Main Results: Adaptive Upper-bounds
Our main result is a dynamic regret upper bound of similar order to Proposition 2 without knowledge of the environment, e.g., the significant shift times, or the number of significant phases. It is stated for our algorithm CMETA (Algorithm 1 of Section 4), which, for simplicity, requires knowledge of the time horizon  $T$  (knowledge of  $T$  removable using doubling tricks).
Theorem 3. Let  $\{\tau_i(X_T)\}_{i=0}^{L+1}$  denote the unknown experienced significant shifts (Definition 6). We then have w.p. at least  $1 - 1/T^2$  w.r.t. the randomness of  $X_T$ , for some  $C > 0$ :
$$
\mathbb {E} \left[ R _ {T} (\mathsf {C M E T A}, \boldsymbol {X} _ {T}) \mid \boldsymbol {X} _ {T} \right] \leq C \log (K) \log^ {3} (T) \sum_ {i = 1} ^ {\tilde {L} (\boldsymbol {X} _ {T})} \left(\tau_ {i} (\boldsymbol {X} _ {T}) - \tau_ {i - 1} (\boldsymbol {X} _ {T})\right) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}}.
$$
Corollary 4 (Adapting to Experienced Significant Shifts). Under the conditions of Theorem 3, with probability at least  $1 - 1 / T^2$  w.r.t. the randomness in  $\mathbf{X}_T$ :
$$
\mathbb {E} \left[ R _ {T} (\mathsf {C M E T A}, \boldsymbol {X} _ {T}) \mid \boldsymbol {X} _ {T} \right] \leq C \log (K) \log^ {3} (T) \cdot \left(K \cdot \left(\tilde {L} (\boldsymbol {X} _ {T}) + 1)\right)\right) ^ {\frac {1}{2 + d}} \cdot T ^ {\frac {1 + d}{2 + d}}.
$$
Note, this is tighter than the earlier mentioned  $(L + 1)^{\frac{1}{2 + d}}T^{\frac{1 + d}{2 + d}}$  rate. The next corollary asserts that Theorem 3 also recovers the optimal rate in terms of total-variation  $V_{T}$ .
Corollary 5 (Adapting to Total Variation). Under the conditions of Theorem 3:
$$
\mathbb {E} \left[ R _ {T} (\mathsf {C M E T A}, \boldsymbol {X} _ {T}) \right] \leq C \log (K) \log^ {3} (T) \left(T ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}} + \left(V _ {T} \cdot K\right) ^ {\frac {1}{3 + d}} \cdot T ^ {\frac {2 + d}{3 + d}}\right).
$$
Remark 5. Our regret bound can straightforwardly be generalized to  $\min \{(\tilde{L} +1)^{\frac{\beta}{2\beta + d}}\cdot T^{\frac{\beta + d}{2\beta + d}}\cdot K^{\frac{\beta}{2\beta + d}},(V_T\cdot K)^{\frac{\beta}{3\beta + d}}\cdot T^{\frac{2\beta + d}{3\beta + d}} + T^{\frac{\beta + d}{2\beta + d}}\cdot K^{\frac{\beta}{2\beta + d}}\}$  for  $\beta$  -Holder reward functions with  $\beta \leq 1$  , provided the notion  $(\star)$  is modified to take into account a bias of  $r^\beta (B)$
We take a similar algorithmic approach to Suk and Kpotufe [2022], with several important modifications for our setting. The high-level strategy is to schedule multiple copies of a base algorithm (Algorithm 2) at random times and durations, in order to ensure updated and reliable estimation of the gaps in  $(\star)$ . This allows fast enough detection of unknown experienced significant shifts.
Overview of Algorithm Hierarchy. Our main algorithm CMETA (Algorithm 1) proceeds in episodes, each of which begins by playing according to an initially scheduled base algorithm of possible duration equal to the number of rounds left till  $T$ . Base algorithms occasionally activate their own base algorithms of varying durations (Line 9 of Algorithm 2), called replays, according to a random schedule (set via variables  $\{Z_{m,s}\}$ ). We refer to the currently playing base algorithm as the active base algorithm. This induces a hierarchy of base algorithms, from parent to child instances.
Choice of Level. Focusing on a single base algorithm now, each Base-Alg manages its own discretization of the context space  $\mathcal{X} = [0,1]^d$ , corresponding to a level  $r \in \mathcal{R}$  (see Definition 3). Within each bin  $B \in \mathcal{T}_r$  at the level  $r$ , candidate arms, maintained in a set  $\mathcal{A}(B)$ , are evicted according to importance-weighted estimates (4) of local gaps.
As discussed in Subsection 2.3, key in algorithmic design is determining the optimal level  $r \in \mathcal{R}$ . An immediate difficulty is that the oracle procedure's choice of level (see Definition 7) depends on the unknown significant phase length  $\tau_{i+1} - \tau_i$ . To circumvent this, and as in previous works on Lipschitz contextual bandits [Perchet and Rigollet, 2013, Slivkins, 2014], we rely on an adaptive time-varying choice of level  $r_t$ . Specifically, each base algorithm uses the level  $r_{t-t_{\mathrm{start}}}$  based on the time elapsed since the time  $t_{\mathrm{start}}$  it was first activated.
Managing Multiple Base Algorithms. Instances of Base-Alg and CMETA share information, in the form of global variables as listed below:
- All variables defined in CMETA:  $t_{\ell}, t, \{A_{\text{master}}(B)\}_{B \in \mathcal{T}}, \{Z_{m,t}\}$  (see Lines 3-6 of Algorithm 1).
- The choice of arm played at each round  $t$ , along with observed rewards  $Y_{t}^{a}$ , and the candidate arm set  $\mathcal{A}_{t}$  which takes value the set  $\mathcal{A}(B)$  of the active Base-Alg at round  $t$  and bin  $B = T_{r}(X_{t})$  used.
By sharing these global variables, any Base-Alg can trigger a new episode: once an arm is evicted from a Base-Alg's  $\mathcal{A}(B)$ , it is also evicted from  $\mathcal{A}_{\mathrm{master}}(B)$ , which is essentially the candidate arm set for the current episode. A new episode is triggered at time  $t$  when  $\mathcal{A}_{\mathrm{master}}(B)$  becomes empty for some bin  $B$  (necessarily a currently experienced bin), i.e., there is no safe arm left to play at the context  $X_{t}$  in the sense of Definition 6. Note that  $\mathcal{A}(B)$  are local variables internal to each Base-Alg (the owner of which will be clear from context in usage).
To ensure consistent behavior while using a time-varying choice of level, we enforce further regularity in arm evictions across  $\mathcal{X}$ : arms evicted from  $\mathcal{A}(B')$  are also evicted from child bins  $B \subseteq B'$  to ensure  $\mathcal{A}(B) \subseteq \mathcal{A}(B')$ .
Algorithm 1: Contextual Meta-Elimination while Tracking Arms (CMETA)
Input: horizon  $T$ , set of arms  $[K]$ , tree  $\mathcal{T}$  with levels  $r \in \mathcal{R}$ .  
1 Initialize: round count  $t \gets 1$ .  
2 Episode Initialization (setting global variables):  
3  $t_{\ell} \gets t$ . //  $t_{\ell}$  indicates start of  $\ell$ -th episode.  
4 For each bin  $B \in \mathcal{T}$ , set  $\mathcal{A}_{\mathrm{master}}(B) \gets [K]$ . // Initialize master candidate arm sets  
5 For each  $m = 2, 4, \ldots, 2^{\lceil \log(T) \rceil}$  and  $s = t_{\ell} + 1, \ldots, T$ :  
6 Sample and store  $Z_{m,s} \sim \text{Bernoulli}\left(\left(\frac{1}{m}\right)^{\frac{1}{2+d}} \cdot \left(\frac{1}{s - t_{\ell}}\right)^{\frac{1+d}{2+d}}\right)$ . // Set replay schedule.  
7 Run Base-Alg  $(t_{\ell}, T + 1 - t_{\ell})$ .  
8 if  $t < T$  then restart from Line 2 (i.e. start a new episode).;
Algorithm 2: Base-Alg  $(t_{\mathrm{start}},m_0)$ : Adaptively Binned Elimination with randomized arm-pulls
Input: starting round  $t_\mathrm{start}$  , scheduled duration  $m_0$    
1 Initialize:  $t\gets t_{\mathrm{start}}$  For each bin  $B$  at any level in  $\mathcal{T}$  , set  $\mathcal{A}(B)\leftarrow [K]$    
2 while  $t\leq t_{\mathrm{start}} + m_0$  do   
3 Choose level in  $\mathcal{R}$  ..  $r\gets r_{t - t_{\mathrm{start}}}$  . Let  $\mathcal{A}_t\gets \mathcal{A}(B)$  and let  $B\gets T_r(X_t)$  . Play a random arm  $a\in \mathcal{A}_t$  selected with probability  $1 / |\mathcal{A}_t|$    
6 Increment  $t\gets t + 1$    
7 if  $\exists m$  such that  $Z_{m,t} > 0$  then   
8 Let  $m\stackrel {\cdot}{=}\max \{m\in \{2,4,\dots,2^{\lceil \log (T)\rceil}\} :Z_{m,t} > 0\}$  // Set maximum replay length.   
9 Run Base-Alg  $(t,m)$  // Replay interrupts.   
10 Evict bad arms in bin B:   
11  $\mathcal{A}(B)\gets \mathcal{A}(B)\backslash \{a\in [K]:$  ∃rounds  $[s_1,s_2]\subseteq [t_{\mathrm{start}},t)$  s.t.(5) holds for bin  $T_{s_2 - s_1}(X_t)$    
12  $\mathcal{A}_{\mathrm{master}}(B)\gets \mathcal{A}_{\mathrm{master}}(B)\backslash \{a\in [K]:$  ∃rounds  $[s_1,s_2]\subseteq [t_\ell ,t)$  s.t.(5) holds for bin  $T_{s_2 - s_1}(X_t)$    
13 Refine candidate arms: //Discard arms previously discarded in ancestor bins.   
14  $\mathcal{A}(B)\gets \cap_{B'\in \mathcal{T},B\subseteq B'}\mathcal{A}(B')$    
15  $\mathcal{A}_{\mathrm{master}}(B)\gets \cap_{B'\in \mathcal{T},B\subseteq B'}\mathcal{A}_{\mathrm{master}}(B')$    
16 Restart criterion: if  $\mathcal{A}_{\mathrm{master}}(B) = \emptyset$  for some bin  $B$  then RETURN.;   
17 RETURN.
Estimating Aggregate Local Gaps.  $\sum_{s=s_1}^{s_2} \delta_s(a', a) \cdot \mathbf{1}\{X_s \in B\}$  is estimated by  $\sum_{s=s_1}^{s_2} \hat{\delta}_s^B(a', a)$ , where  $\delta_s(a', a) \cdot \mathbf{1}\{X_s \in B\}$  is estimated by importance weighting as:
$$
\hat {\delta} _ {s} ^ {B} \left(a ^ {\prime}, a\right) \doteq \left| \mathcal {A} _ {t} \right| \cdot \left(Y _ {s} ^ {a ^ {\prime}} \cdot \mathbf {1} \left\{\pi_ {s} = a ^ {\prime} \right\} - Y _ {s} ^ {a} \cdot \mathbf {1} \left\{\pi_ {s} = a \right\}\right) \cdot \mathbf {1} \left\{a \in \mathcal {A} _ {s} \right\} \cdot \mathbf {1} \left\{X _ {s} \in B \right\}. \tag {4}
$$
Note that the above is an unbiased estimate of  $\delta_s(a', a) \cdot \mathbf{1}\{X_s \in B\}$  whenever  $a'$  and  $a$  are both in  $\mathcal{A}_s$  at time  $s$ , conditional on the context  $X_s$ . It then follows that, conditional on  $\mathbf{X}_T$ , the difference  $\sum_{s=s_1}^{s_2} \left( \hat{\delta}_s^B(a', a) \cdot \mathbf{1}\{X_s \in B\} - \delta_s(a', a) \right)$  is a martingale that concentrates at a rate of order roughly  $\sqrt{K \cdot n_B([s_1, s_2])}$ , where recall from earlier that  $n_B(I) \doteq \sum_{s \in I} \mathbf{1}\{X_s \in I\}$  is the context count in bin  $B$  over interval  $I$ .
An arm  $a$  is then evicted at round  $t$  if, for some fixed  $C_0 > 0^1$ ,  $\exists$  rounds  $s_1 < s_2 \leq t$  such that at level  $r_{s_2 - s_1}$  and letting  $B \doteq T_{s_2 - s_1}(X_t)$  (i.e., the bin at level  $r_{s_2 - s_1}$  containing  $X_t$ )
$$
\max  _ {a ^ {\prime} \in [ K ]} \sum_ {s = s _ {1}} ^ {s _ {2}} \hat {\delta} _ {s} ^ {B} \left(a ^ {\prime}, a\right) > \sqrt {C _ {0} \cdot K \log (T) \cdot \left(n _ {B} \left(\left[ s _ {1} , s _ {2} \right]\right) \vee K \log (T)\right)} + r _ {s _ {2} - s _ {1}} \cdot n _ {B} ([ s _ {1}, s _ {2} ]). \tag {5}
$$
5 Key Technical Highlights of Analysis
While a full analysis is deferred to Appendix D, we highlight key novelties and intuitive calculations.
- Local Safety in Bins implies Safe Total Regret. We first argue that the notion of significant regret  $(\star)$  within a bin  $B$  captures the total allowable regret rates  $T^{\frac{1 + d}{2 + d}}$  we wish to compete with over  $T$  "safe" rounds. If  $(\star)$  holds for no intervals  $[s_1, s_2]$  in all bins  $B$ , arm  $a$  would be safe and incur little regret over any  $[s_1, s_2]$ . As it turns out, bounding the per-bin regret by  $(\star)$  implies a total regret of  $T^{\frac{1 + d}{2 + d}}$  as seen from the following rough calculation: via concentration and the strong density assumption (Assumption 2) to conflate  $n_B([1, T]) \approx r(B)^d \cdot T$  and the fact that there are  $\approx r^{-d}$  bins at level  $r$ , we have:
$$
\sum_ {B \in T _ {r}} \sqrt {K \cdot n _ {B} ([ 1 , T ])} + r \cdot n _ {B} ([ 1, T ]) \leq K ^ {1 / 2} \cdot T ^ {1 / 2} \cdot r ^ {- d / 2} + T \cdot r. \tag {6}
$$
In particular taking  $r \propto (K / T)^{\frac{1}{2 + d}}$  makes the above R.H.S. the desired rate  $K^{\frac{1}{2 + d}}T^{\frac{1 + d}{2 + d}}$ .
- Significant Regret Threshold is Estimation Error. At the same time, the R.H.S. of  $(\star)$  is a standard variance and bias bound on the regression error of estimating the cumulative regret  $\sum_{s=s_1}^{s_2} \delta_s^a(x) \cdot 1\{X_s \in B\}$  at any context  $x \in B$ , conditional on  $\mathbf{X}_T$  (see Lemma 6). Thus, intuitively, large gaps of magnitude above the threshold  $\sqrt{K \cdot n_B(I)} + r(B) \cdot n_B(I)$  in  $(\star)$  are detectable via the estimates of (4).
Combining the above two points, we conclude that the notion of significant regret  $(\star)$  balances both (1) detection of unsafe arms and (2) regret of playing non-evicted arms. We next argue the randomized scheduling of multiple base algorithms is suitable for detecting experienced significant shifts.
- A New Balanced Replay Scheduling. As mentioned earlier in Subsection 3.1, previous adaptive works on contextual bandits fail to attain the optimal regret in this setting due to an inappropriate frequency of scheduling re-exploration. We introduce a novel scheduling (Line 6 of Algorithm 1) of replays which carefully balances exploration and fast detection of significant regret in the sense of  $(\star)$ . In particular, the determined probability  $(1 / m)^{\frac{1}{2 + d}}(1 / t)^{\frac{1 + d}{2 + d}}$  of scheduling a new Base-Alg  $(t,m)$  comes from the following intuitive calculation: a single replay of duration  $m$  will, if scheduled, incur an additional regret of about  $m^{\frac{1 + d}{2 + d}}$ . Then, summing over all possible replays, the total extra regret incurred due to scheduled replays is roughly upper bounded by
$$
\sum_ {t = 1} ^ {T} \sum_ {m = 2, 4, \dots , T} \left(\frac {1}{m}\right) ^ {\frac {1}{2 + d}} \left(\frac {1}{t}\right) ^ {\frac {1 + d}{2 + d}} \cdot m ^ {\frac {1 + d}{2 + d}} \lesssim \sum_ {t = 1} ^ {T} T ^ {\frac {d}{2 + d}} \cdot (1 / t) ^ {\frac {1 + d}{2 + d}} \lesssim T ^ {\frac {1 + d}{2 + d}}.
$$
${}^{1}{C}_{0} > 0$  needs to be sufficiently large, but is a universal constant free of the horizon  $T$  or any distributional parameters.
In other words, the cost of replays only incurs extra constants in the regret. Surprisingly, we find this scheduling rate is also sufficient for detecting significant regret in any experienced subregion  $B$  of the context space  $\mathcal{X}$ , i.e. there is no need to do additional exploration on a localized per-bin basis.
Key in this observation is the fact that, to detect significant regret over interval  $I$  in any bin  $B$ , it suffices to check it at the critical level  $r_I \propto (K / |I|)^{\frac{1}{2 + d}}$ , where  $|I|$  is the length of  $I$ . In particular, a well-timed Base-Alg running on the interval  $I$  will use this level  $r_I$  (Line 2 of Algorithm 2) and is, thus, equipped to detect significant regret at all experienced bins over  $I$ .
- Suffices to Only Check  $(\star)$  at Critical Levels  $r_I$ . At first glance, detecting experienced significant shifts (Definition 6) appears difficult as an arm  $a$  may incur significant regret over a different bin  $B'$  from the bin  $B$  that is currently being used by the algorithm.
We in fact show that it suffices to only estimate the R.H.S. of  $(\star)$  in bins  $B^{\prime}$  at the critical level  $r_I$  (Lemma 9 and Proposition 15). We give a rough argument for why this is the case: first, note that  $(\star)$  may be rewritten as
$$
\frac {1}{n _ {B} (I)} \sum_ {s \in I} \delta_ {s} (a) \cdot \mathbf {1} \{X _ {s} \in B \} \geq \sqrt {\frac {K}{n _ {B} (I)}} + r (B). \tag {7}
$$
We next relate the two sides of the above display across different levels  $r(B)$ .
- By concentration and the strong density assumption (Assumption 2), the R.H.S. of (7) is in fact of order  $\propto 1 / \sqrt{|I| \cdot r(B)^d} + r(B)$ , which is minimized at the critical level  $r(B) \propto r_I$ .
- The L.H.S. of (7) is an estimate of the average gap  $\frac{1}{|I|}\sum_{s\in I}\delta_s^a (x)$  at any particular  $x\in B$  using nearby contexts. In fact, by concentration, the two can be conflated up to error terms of order the R.H.S. of (7).
Combining the above two points, we see that if (7) holds for some bin  $B$ , then it will also hold for the "critical bin"  $B'$ , with  $B' \cap B \neq \emptyset$ , at the critical level  $r_I \propto (K / |I|)^{\frac{1}{2 + d}}$ . In other words, significant regret at any experienced bin implies significant regret at the critical level, thus allowing us to constrain attention to these critical levels.
On the other hand, we observe that the calculations in (6) would hold if we only checked  $(\star)$  for bins  $B$  at level  $r_I$ . Thus, it also suffices to only use the levels  $r_I$  for regret minimization over intervals  $I$  with no experienced significant shift.
Yet, even still, the analysis is challenging as there may be "missing data problems": arms  $a \in \mathcal{A}(B)$  in contention at  $B$  may have been evicted from sibling bins inside the parent  $B' \supset B$  at the critical level. In other words, it is not a priori obvious how to do reliable estimation of arms  $a \in \mathcal{A}(B)$  across a larger bin  $B'$  which may contain sub-regions where  $a$  has already been evicted. We show it is in fact possible to identify a subclass of intervals of rounds (Definition 12) and an associated class of replays (Definition 13) which can quickly evict arm  $a$  in the critical bin  $B'$  before there are missing data problems for bin  $B \subseteq B'$ . The details of this can be found in Proposition 15 of Appendix D.2.
We have shown that it is possible to adapt optimally to an unknown number of experienced significant shifts - a new notion introduced here - which captures severe changes in best-arm, only at observed contexts. An interesting future direction is to explore other notions of experienced shifts which may yield even more optimistic rates. For example, suppose changes in best arm occur at every round, but are localized to a sub-region  $\Xi$  of the context space  $\mathcal{X}$ . Then, a procedure which discretizes  $\mathcal{X}$  into bins at level  $T^{-\frac{1}{2 + d}}$  and runs local instantiations of a suitable non-stationary MAB algorithm (e.g., META of Suk and Kpotufe [2022]) can attain faster rates than those of Theorem 3 for some choices of  $\Xi$ . At the same time, such a strategy cannot always attain the  $\tilde{L} (\mathbf{X}_T)^{\frac{1}{2 + d}}\cdot T^{\frac{1 + d}{2 + d}}$  rate in general as using the level  $T^{-\frac{1}{2 + d}}$  is insufficient to detect experienced significant shifts occurring at short intervals  $I$  of length  $|I|\ll T$  (which require larger levels). This prompts the questions of whether there exists a unified notion of experienced shift which captures the most optimistic rates in these scenarios and whether such a notion can be adapted to.
Yuhong Yang, Dan Zhu, et al. Randomized allocation with nonparametric estimation for a multiarmed bandit problem with covariates. The Annals of Statistics, 30(1):100-121, 2002.
Huozhi Zhou, Jinglin Chen, Lav R. Varshney, and Ashish Jagmohan. Nonstationary reinforcement learning with linear function approximation. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
A Details for Specializing Previous Contextual Bandit Results to Lipschitz Contextual Bandits
A.1 Finite Policy Class Contextual Bandits
In the finite policy class setting $^2$ , one is given access to a known finite class  $\Pi$  of policies  $\pi : \mathcal{X} \to [K]$  and in the non-stationary variant, seeks to minimize regret to the time-varying benchmark of best policies  $\pi_t^* \doteq \operatorname{argmax}_{\pi \in \Pi} \mathbb{E}_{(X,Y) \in \mathcal{D}_t}[Y(\pi(X))]$ . In other words, the "dynamic regret" in this setting is defined by (for chosen policies  $\{\hat{\pi}_t\}_t$ )
$$
\sum_ {t = 1} ^ {T} \max  _ {\pi \in \Pi} \mathbb {E} _ {(X, Y) \in \mathcal {D} _ {t}} [ Y (\pi (X)) ] - \sum_ {t = 1} ^ {T} \mathbb {E} [ Y _ {t} (\hat {\pi} _ {t}) ]. \tag {8}
$$
We can in fact recover the Lipschitz contextual bandit setting and relate the above to our notion of dynamic regret (Definition 2). To do so, we let  $\Pi$  be the class of policies which uses a level  $r\in \mathcal{R}$  and discretizes decision-making across individual bins  $B\in \mathcal{T}_r$ . Then, we claim there is an oracle sequence of policies  $\{\pi_{\mathrm{t}}^{\mathrm{oracle}}\}_{t}$  which attains the minimax regret rate of Theorem 1. So, it remains to bound the regret to the sequence  $\{\pi_{\mathrm{t}}^{\mathrm{oracle}}\}_{t}$  in the sense above.
- Parametrizing in Terms of Global Number  $L$  of Shifts. Suppose there are  $L + 1$  stationary phases of length  $T / (L + 1)$ . Then, we first claim there is an oracle sequence of policies  $\pi_{\mathrm{t}}^{\mathrm{oracle}}$  which attains  $\text{reget} \left( L + 1 \right)^{\frac{1}{2 + d}} \cdot T^{\frac{1 + d}{2 + d}}$ .
First, recall from Subsection 2.3 the oracle choice of level  $r_n$  for a stationary period of  $n$  rounds, or the level  $r_n \propto (K / n)^{\frac{1}{2 + d}}$ . Now, define  $\{\pi_{\mathrm{t}}^{\mathrm{oracle}}\}_{t}$  as follows: at each round  $t$ ,  $\pi_{\mathrm{t}}^{\mathrm{oracle}}$  uses the oracle level  $r \doteq r_{T / (L + 1)} \propto \left(\frac{K(L + 1)}{T}\right)^{\frac{1}{2 + d}}$  and plays in each bin  $B \in \mathcal{T}_r$ , the arm maximizing the average reward in that bin  $\mathbb{E}[f_t^a(X_t) \mid X_t \in B]$ . As this is a biased version of the actual bandit problem  $\{f_t^a(X_t)\}_{a \in [K]}$  at context  $X_t$ , it will follow that  $\pi_{\mathrm{t}}^{\mathrm{oracle}}$  incurs regret of order the bias of estimation in  $B$  which is  $r$ .