SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Joe Suk
joe.suk@columbia.edu
Samory Kpotufe
samory@columbia.edu
We study nonparametric contextual bandits where Lipschitz mean reward functions may change over time. We first establish the minimax dynamic regret rate in this less understood setting in terms of number of changes  $L$  and total-variation  $V$ , both capturing all changes in distribution over context space, and argue that state-of-the-art procedures are suboptimal in this setting.
Next, we tend to the question of an adaptivity for this setting, i.e. achieving the minimax rate without knowledge of  $L$  or  $V$ . Quite importantly, we posit that the bandit problem, viewed locally at a given context  $X_{t}$ , should not be affected by reward changes in other parts of context space  $\mathcal{X}$ . We therefore propose a notion of change, which we term experienced significant shifts, that better accounts for locality, and thus counts considerably less changes than  $L$  and  $V$ . Furthermore, similar to recent work on non-stationary MAB [Suk and Kpotufe, 2022], experienced significant shifts only count the most significant changes in mean rewards, e.g., severe best-arm changes relevant to observed contexts.
Our main result is to show that this more tolerant notion of change can in fact be adapted to.
Contextual bandits model sequential decision making problems where the reward of a chosen action depends on an observed context  $X_{t}$  at time  $t$ , e.g., a consumer's profile, a medical patient's history. The goal is to maximize the total rewards over time of chosen actions, as informed by seen contexts. As such, one suitable measure of performance is that of dynamic regret, which compares earned rewards to a time-varying oracle maximizing mean rewards at  $X_{t}$ . While it is often assumed in the bulk of works in this setting that rewards distributions remain stationary over time, it is understood that in practice, environmental changes induce nontrivial changes in rewards.
In fact, the problem of non-stationary environments has received a surge of attention in the simpler non-contextual Multi-Arm-Bandits (MAB) setting, while the more challenging contextual case remains ill-understood. In particular in the contextual case, some recent works [Wu et al., 2018, Luo et al., 2018, Chen et al., 2019, Wei and Luo, 2021] consider parametric settings, i.e. where reward functions belong to fixed parametric family, and show that one may achieve rates adaptive to an unknown number of  $L$  of shifts in rewards or to a notion of total-variation  $V$ , both accounting for all changes over time and context space. Instead here, we consider a much larger class of reward functions, namely Lipschitz rewards, corresponding to the natural assumption that closeby contexts have similar rewards even as reward distributions change.
As a first result for this nonparametric setting, we establish some minimax lower-bounds as a baseline in terms of either  $L$  or  $V$ , and argue that state-of-the-art procedures for the parametric case—extended to the class of Lipschitz functions—do not achieve these baselines.
We then turn attention to whether such baselines may be achieved adaptively, i.e., without knowledge of  $L$  or  $V$ . The answer as we show is affirmative, and more importantly, some much weaker notions of change may be adapted to; for intuition, while  $L$  or  $V$  accounts for any change at any time over the context space (say  $\mathcal{X}$ ), it may be that all changes are relegated to parts of the space irrelevant to observed contexts  $X_{t}$  at the time they are played. For instance, suppose at time  $t$ , we observe  $X_{t} = x_{0}$ , then it may not make sense to count changes that happen at some other  $x_{1}$  far from  $x_{0}$ , or changes that happened at  $x_{0}$  itself but far back in time.
We therefore propose a new parameterization of change, termed experienced significant shifts that better accounts for the locality of changes in time and space, and as such may register much less changes than either  $L$  or  $V$ . As a sanity check, we show that an oracle policy which restarts only at experienced significant shifts can attain enhanced regret rates in terms of the number  $\tilde{L} = \tilde{L}(X_1, \ldots, X_T)$  of such experienced shifts (Proposition 2), a rate always no worse that the baseline we first established in terms of  $L$  and  $V$ .
Our main result is to show that experienced significant shifts can be adapted to (Theorem 3), i.e., with no prior knowledge of such shifts. Importantly, the result holds in both stochastic environments, and in (oblivious) adversarial ones with no change to our notion, algorithmic approach, nor analysis. Furthermore, similar to recent advances in the non-contextual case [Abbasi-Yadkori et al., 2022, Suk and Kpotufe, 2022], an experienced shift is only triggered under severe changes such as changes of best arms locally at a context  $X_{t}$ . An added difficulty in the contextual case is that we cannot hope to observe rewards for a given arm (action) repeatedly at  $X_{t}$  as the context may only appear once, and have to rely on carefully chosen nearby points to identify unknown shifts in reward at  $X_{t}$ .
1.1 Other Related Work
Nonparametric Contextual Bandits. The stationary bandits with covariates (where rewards and contexts follow a joint distribution) was first introduced in a one-armed bandit problem [Woodroofoe, 1979, Sarkar, 1991], with the nonparametric model first studied by Yang et al. [2002]. Minimax regret rates, based on a margin condition, were first established for the two-armed bandit in Rigollet and Zeevi [2010] and generalized to any finite number of arms in Perchet and Rigollet [2013], with further insights thereafter [Qian and Yang, 2016a,b, Reeve et al., 2018, Guan and Jiang, 2018, Hu et al., 2020, Arya and Yang, 2020, Suk and Kpotufe, 2021, Gur et al., 2022, Cai et al., 2022]. However, the mentioned works all assume a stationary distribution of rewards over contexts. Blanchard et al. [2023] studies non-stationary nonparametric contextual bandits, but in the much-different context of universal learning, concerning when sublinear regret is achievable asymptotically. Lipschitz contextual bandits also appears as part of studies on broader infinite-armed settings [Lu et al., 2009, Krishnamurthy et al., 2019]. Related, Slivkins [2014] allows for non-stationary (i.e., obliviously adversarial) environments, but only studies regret to the (per-context) best arm in hindsight. Realizable contextual bandits posits that the regression function capturing mean rewards in contexts lies in some known class of regressors  $\mathcal{F}$ , over which one can do empirical risk minimization [Foster et al., 2018, Foster and Rakhlin, 2020, Simchi-Levi and Xu, 2021]. While this setting recovers Lipschitz contextual bandits, the only applicable non-stationary guarantee to our knowledge is Wei and Luo [2021], which yields suboptimal dynamic regret (see Table 1).
Non-Stationary Bandits and RL. In the simpler non-contextual bandits, changing reward distributions (a.k.a. switching bandits) was introduced in Garivier and Moulines [2011] and further explored with various assumptions and formulations [Karnin and Anava, 2016, Allesiardo et al., 2017, Liu et al., 2018, Wei and Srivatsva, 2018, Besbes et al., 2019, Cao et al., 2019, Mukherjee and Maillard, 2019, Besson et al., 2022]. While these earlier works focused on algorithmic design assuming knowledge of non-stationarity, such a strong assumption was removed via the adaptive procedures of Auer et al. [2019], Chen et al. [2019]. In followup works, Abbasi-Yadkori et al. [2022], Suk and Kpotufe [2022] show that tighter dynamic regret rates are possible, scaling only with severe changes in best arm. The ideas from non-stationary MAB were extended to various contextual bandit settings by Wu et al. [2018] (for linear mean rewards in contexts), Luo et al. [2018], Chen et al. [2019] (for finite policy classes), and Wei and Luo [2021] (for realizable mean reward functions). There have also been extensions to various reinforcement learning setups [Jaksch et al., 2010, Gajane et al., 2018, Chi Cheung et al., 2019, Ortner et al., 2020, Fei et al., 2020, Cheung et al., 2020, Touati and Vincent, 2020, Domingues et al., 2021, Mao et al., 2021, Domingues et al., 2021, Wei and Luo, 2021, Zhou et al., 2022, Lykouris et al., 2021, Wei et al., 2022, Chen and Luo, 2022, Ding and Lavaei, 2023].
Of these works, only Domingues et al. [2021] can recover Lipschitz contextaul bandits, whereupon we find their dynamic regret bounds are suboptimal (see Table 1). Again, the typical aim of the aforementioned works on contextual bandits or RL is to minimize a notion of dynamic regret in terms of the number of changes  $L$  or total-variation  $V$ . As such, the guarantees of such works don't involve tighter notions of experienced non-stationarity, such as those studied in this work.
2.1 Contextual Bandits with Changing Rewards
Preliminaries. We assume a finite set of arms  $[K] \doteq \{1, 2, \ldots, K\}$ . Let  $Y_{t} \in [0, 1]^{K}$  denote the vector of rewards for arms  $a \in [K]$  at round  $t \in [T]$  (horizon  $T$ ), and  $X_{t}$  the observed context at that round, lying in  $\mathcal{X} \doteq [0, 1]^{d}$ , which have joint distribution  $(X_{t}, Y_{t}) \sim \mathcal{D}_{t}$ . We let  $\mathbf{X}_{t} \doteq \{X_{s}\}_{s \leq t}, \mathbf{Y}_{t} \doteq \{Y_{s}\}_{s \leq t}$  denote the observed contexts and (observed and unobserved) rewards from rounds 1 to  $t$ . In our setting, an oblivious adversary decides a sequence of (independent) distributions on  $\{(X_{t}, Y_{t})\}_{t \in [T]}$  before the first round.
Notation. The reward function  $f_{t}:\mathcal{X}\to [0,1]^{K}$  is  $f_{t}^{a}(x)\doteq \mathbb{E}[Y_{t}^{a}|X_{t} = x], a\in [K]$ , and captures the mean rewards of arm  $a$  at context  $x$  and time  $t$ .
A policy chooses actions at each round  $t$ , based on observed contexts (up to round  $t$ ) and passed rewards, whereby at each round  $t$  only the reward  $Y_{t}^{a}$  of the chosen action  $a$  is revealed. Formally:
Definition 1 (Policy). A policy  $\pi \doteq \{\pi_t\}_{t\in \mathbb{N}}$  is a random sequence of functions  $\pi_t:\mathcal{X}^t\times [K]^{t - 1}\times [0,1]^{t - 1}\to [K]$ . A randomized policy  $\pi_t$  maps to distributions on  $[K]$ . In an abuse of notation, in the context of a sequence of observations till round  $t$ , we'll let  $\pi_t\in [\bar{K} ]$  denote the (possibly random) action chosen at round  $t$ .
The performance of a policy is evaluated using the dynamic regret, defined as follows:
Definition 2. Fix a context sequence  $X_{T}$ . Define the dynamic regret of a policy  $\pi$ , as
$$
R _ {T} (\pi , X _ {T}) \doteq \sum_ {t = 1} ^ {T} \max  _ {a \in [ K ]} f _ {t} ^ {a} (X _ {t}) - f _ {t} ^ {\pi_ {t}} (X _ {t}).
$$
So, we aim to minimize  $\mathbb{E}[R_T(\pi ,\mathbf{X}_T)]$  where the expectation is over  $\mathbf{X}_T$ ,  $\mathbf{Y}_T$ , and randomness in  $\pi$ .
Notation. As much of our analysis focuses on the gaps in mean rewards between arms at observed contexts  $X_{t}$ , the following notation will serve useful. Let  $\delta_t(a', a) \doteq f_t^{a'}(X_t) - f_t^a(X_t)$  denote the relative gap of arms  $a$  to  $a'$  at round  $t$  at context  $X_{t}$ . Define the worst gap of arm  $a$  as  $\delta_t(a) \doteq \max_{a' \in [K]} \delta_t(a', a)$ , corresponding to the instantaneous regret of playing  $a$  at round  $t$  and context  $X_{t}$ . Thus, the dynamic regret can be written as  $\sum_{t \in [T]} \mathbb{E}[\delta_t(\pi_t)]$ . Additionally, we will occasionally talk about the gap functions in context  $x \in \mathcal{X}$ . In an abuse of notation, let  $\delta_t^{a', a}(x) \doteq f_t^{a'}(x) - f_t^a(x)$  and  $\delta_t^a(x) \doteq \max_{a' \in [K]} \delta_t^{a', a}(x)$ . Generally, when a context  $x$  is not specified, as in the quantities  $\delta_t(a', a), \delta_t(a)$ , it should be assumed that the context in question is  $X_{t}$ .
2.2 Nonparametric Setting
We assume, as in prior work on nonparametric contextual bandits [Rigollet and Zeevi, 2010, Perchet and Rigollet, 2013, Slivkins, 2014, Reeve et al., 2018, Guan and Jiang, 2018, Suk and Kpotufe, 2021], that the reward function is 1-Lipschitz.
Assumption 1 (Lipschitz  $f_{t}$ ). For all rounds  $t \in \mathbb{N}$ ,  $a \in [K]$  and  $x, x' \in \mathcal{X}$ ,
$$
\left| f _ {t} ^ {a} (x) - f _ {t} ^ {a} \left(x ^ {\prime}\right) \right| \leq \| x - x ^ {\prime} \| _ {\infty}. \tag {1}
$$
For ease of presentation, we assume the contextual marginal distribution  $\mu_{X}$  remains the same across rounds. Furthermore, we make a standard strong density assumption on  $\mu_{X}$ , which is typical in this nonparametric setting [Audibert and Tsybakov, 2007, Perchet and Rigollet, 2013, Qian and Yang, 2016a,b, Gur et al., 2022, Hu et al., 2020, Arya and Yang, 2020, Cai et al., 2022]. This holds, e.g. if  $\mu_{X}$  has a continuous Lebesgue density on  $[0,1]^{d}$ , and ensures good coverage of the context space.
Assumption 2 (Strong Density Condition). There exist  $C_d, c_d > 0$  s.t.  $\forall \ell_{\infty}$  balls  $B \subset [0,1]^d$  of diameter  $r \in (0,1]$ :
$$
C _ {d} \cdot r ^ {d} \geq \mu_ {X} (B) \geq c _ {d} \cdot r ^ {d}. \tag {2}
$$
Remark 1. We can in fact relax Assumption 2 so that  $\mu_{X,t}(\cdot)$  is changing in  $t$  and (2) is satisfied with different constants  $C_{d,t}, c_{d,t}$ . Our procedures in the end will not require knowledge of any  $C_{d,t}, c_{d,t}$ .
A common algorithmic approach in nonparametric contextual bandits, starting from earlier work [Rigollet and Zeevi, 2010, Perchet and Rigollet, 2013], is to discretize or partition the context space  $\mathcal{X}$  into bins where we can maintain local reward estimates. These bins have a natural hierarchical tree structure which we first elaborate.
Definition 3 (Partition Tree). Let  $\mathcal{R} \doteq \{2^{-i} : i \in \mathbb{N} \cup \{0\}\}$ , and let  $\mathcal{T}_r, r \in \mathcal{R}$  denote a regular partition of  $[0,1]^d$  into hypercubes (which we refer to as bins) of side length (a.k.a. bin size)  $r$ . We then define the dyadic tree  $\mathcal{T} \doteq \{\mathcal{T}_r\}_{r \in \mathcal{R}}$ , i.e., a hierarchy of nested partitions of  $[0,1]^d$ . We will refer to the level  $r$  of  $\mathcal{T}$  as the collection of bins in partition  $\mathcal{T}_r$ . The parent of a bin  $B \in \mathcal{T}_r, r < 1$  is the bin  $B' \in \mathcal{T}_{2r}$  containing  $B$ ; child, ancestor and descendant relations follow naturally. We'll use  $\mathcal{T}_r(x)$  to refer to the bin at level  $r$  containing  $x$ , while  $r(B)$  is the side length of bin  $B$ .
Note that, while in the above definition,  $\mathcal{T}$  has infinite levels  $r\in \mathcal{R}$ , at any round  $t$  in a procedure, we implicitly only operate on the subset of  $\mathcal{T}$  containing data.
Key in securing good regret is then finding the optimal level  $r \in \mathcal{R}$  of discretization (balancing local regression bias and variance), which over  $T$  stationary rounds is known to be  $\propto (K / T)^{\frac{1}{2 + d}}$  [Rigollet and Zeevi, 2010]. The intuition for this choice is that a bias of  $T^{-\frac{1}{2 + d}}$  is safe to pay for  $T$  rounds to maintain a minimax regret rate of  $T^{\frac{1 + d}{2 + d}}$  (see rates in Theorem 1). We introduce the following general notation, useful later in the non-stationary problem, for associating a level with an intervals of rounds.
Notation 1 (Level). For  $n \in \mathbb{N} \cup \{0\}$ , let  $r_n$  be the largest  $2^{-m} \in \mathcal{R}$  such that  $(K / n)^{\frac{1}{2 + d}} \geq 2^{-m}$ . When  $I$  is an interval of rounds, we use  $r_I$  as shorthand for  $r_{|I|}$ , where  $|I|$  is the length of  $I$ .
We use  $\mathcal{T}_m, T_m(x)$  as shorthand to denote (respectively) the tree  $\mathcal{T}_r$  of level  $r = r_m$  and the (unique) bin at level  $r_m$  containing  $x$ .
3 Results Overview
3.1 Minimax Lower Bounds Under Global Shifts
Dynamic Regret Upper Bound
ADA-ILTCB [Chen et al., 2019]	\(\left(L^{1/2} \cdot T^{\frac{1+d}{2+d}}\right) \wedge \left(V_T^{1/3} \cdot T^{\frac{2+d}{3+d} + \frac{d}{3(2+d)(3+d)}}\right)\)
MASTER with FALCON [Wei and Luo, 2021]	\(\left(L^{1/2} \cdot T^{\frac{1+d}{2+d}}\right) \wedge \left(V_T^{1/3} \cdot T^{\frac{2+d}{3+d} + \frac{d}{3(2+d)(3+d)}}\right)\)
KeRNS [Domingues et al., 2021] (non-adaptive)	\(V_T^{1/3}T^{\frac{2+d}{3+d} + O(1/d)}\)
Minimax Lower-Bound	\(\left(L^{\frac{1}{2+d}}T^{\frac{1+d}{2+d}}\right) \wedge \left(V_T^{\frac{1}{3+d}}T^{\frac{2+d}{3+d}}\right)\)
Table 1: Existing dynamic regret upper-bounds are suboptimal in the Lipschitz setting.
As a baseline, we start with some basic lower-bounds under the simplest parametrizations of changes in rewards which have appeared in the literature, namely a global number of shifts, and total variation.
Definition 4 (Global Number of Shifts). Let  $L \doteq \sum_{t=2}^{T} \mathbf{1}\{\exists x \in \mathcal{X}, a \in [K] : f_t^a(x) \neq f_{t-1}^a(x)\}$  be the number of global shifts, i.e., it counts every change in mean-reward overtime and over  $\mathcal{X}$  space.
Definition 5 (Total Variation). Define  $V_{T} \doteq \sum_{t=2}^{T} \| \mathcal{D}_{t} - \mathcal{D}_{t-1} \|_{TV}$  where recall  $\mathcal{D}_{t} \in \mathcal{X} \times [0,1]^{K}$  is the joint distribution on context and rewards at time  $t$ .
We have the following initial result (for two-armed bandits) to serve as baseline for this study.
Theorem 1 (Dynamic Regret Lower Bound). Suppose there are  $K = 2$  arms. For  $V, L \in [0, T]$ , let  $\mathcal{P}(V, L, T)$  be the family of joint distributions  $\mathcal{D} \doteq \{\mathcal{D}_t\}_{t \in [T]}$  with either total variation  $V_T \leq V$  or at most  $L$  global shifts. Then, there exists a constant  $c > 0$  such that for any policy  $\pi$ :
$$
\sup  _ {\mathcal {D} \in \mathcal {P} (V, L, T)} \mathbb {E} _ {\mathcal {D}} \left[ R (\pi , \boldsymbol {X} _ {T}) \right] \geq c \left(T ^ {\frac {1 + d}{2 + d}} + T ^ {\frac {2 + d}{3 + d}} \cdot V ^ {\frac {1}{3 + d}}\right) \wedge \left(\left(L + 1\right) ^ {\frac {1}{2 + d}} T ^ {\frac {1 + d}{2 + d}}\right). \tag {3}
$$
Remark 2. Note setting  $d = 0$  in Theorem 1 recovers the minimax rate  $(\sqrt{T} + T^{2/3} V_T^{1/3}) \wedge \sqrt{(L + 1) \cdot T}$  for non-contextual bandits [Besbes et al., 2019].
Achieability of Minimax Lower-Bound (3). We are interested in whether the rates of (3) are achievable, with, or without knowledge of relevant parameters. First, we note that no existing algorithm currently guarantees a rate that matches (3). See Table 1 for a rate comparison (details for specializing to our setting found in Appendix A).
In particular, the prior adaptive works [Chen et al., 2019, Wei and Luo, 2021] both rely on the approach of randomly scheduling replays of stationary algorithms to detect unknown non-stationarity. However, the scheduling rate is designed to safeguard against the parametric  $\sqrt{LT} \wedge V_T^{1/3}T^{2/3}$  regret rates and thus lead to suboptimal dependence on  $L$  and  $V_T$ .
However, a simple back of the envelope calculation indicates that the rate in (3) may be attainable, at least given some distributional knowledge: a minimax-optimal stationary procedure restarted at each shift will incur regret, over  $L$  equally spaced shifts,  $(L + 1) \cdot \left( \frac{T}{L + 1} \right)^{\frac{1 + d}{2 + d}} \approx (L + 1)^{\frac{1}{2 + d}} \cdot T^{\frac{1 + d}{2 + d}}$ .
As it turns out as we will show in the next section, (3) is indeed attainable, even adaptively; in fact, this is shown via a more optimistic problem parametrization as described next.
3.2 A New Problem Parametrization: Experienced Significant Shifts.
As discussed in Subsection 2.3, typical approaches [Rigollet and Zeevi, 2010, Perchet and Rigollet, 2013] in our setting discretize the context space  $\mathcal{X}$  into bins, each of which is treated as an MAB instance. At a high level, our new measure of non-stationarity will trigger an experienced significant shift when the observed context  $X_{t}$  arrives in a bin  $B\in \mathcal{T}$  where there has been a severe change in local best arm, w.r.t. the observed data in that bin.
We first define a notion of significant regret for an arm  $a \in [K]$  locally within a bin  $B \in \mathcal{T}$ . We say arm  $a$  incurs significant regret in bin  $B$  on interval  $I$  if:
$$
\sum_ {s \in I} \delta_ {s} (a) \cdot \mathbf {1} \left\{X _ {s} \in B \right\} \geq \sqrt {K \cdot n _ {B} (I)} + r (B) \cdot n _ {B} (I), \tag {\star}
$$
where  $n_B(I) \doteq \sum_{s \in I} \mathbf{1}\{X_s \in I\}$  and recall  $r(B)$  is the side length of bin  $B$ . The intuition for  $(\star)$  is as follows: suppose that, over  $n$  separate rounds, we observe the same context  $X_s = x_0$  in bin  $B$ . Then, arm  $a$  would be considered unsafe in the local bandit problem at context  $x_0$  if its regret exceeds  $\sqrt{K \cdot n}$  (i.e., the first term on the above R.H.S.), which is a safe regret to pay for the non-contextual problem. Our broader notion  $(\star)$  extends this over the bin  $B$  by also accounting for the bias (i.e., the second term on the above R.H.S.) of observing  $X_s$  near a given context  $x_0 \in B$ .
Remark 3 (Significant Regret Occurs at Critical Levels). If  $(\star)$  holds for some bin  $B$ , then in fact it also roughly holds for the bin  $B'$ , with  $B' \cap B \neq \emptyset$ , at the critical level  $r_{|I|}$  (see Notation 1) w.r.t. interval  $I$  (Lemma 9 and Proposition 15). Thus, it suffices to only check  $(\star)$  for the critical levels  $r_{|I|}$  which will be crucial in the analysis. Additionally, all such critical levels  $r_{|I|}$  are above the optimal level  $T^{-\frac{1}{2 + d}}$  for  $T$  stationary rounds (see Subsection 2.3). Thus, only  $O(\log(T))$  levels play a role in checking  $(\star)$  for all intervals of time  $I$  and bins  $B$ .
We then propose to record an experienced significant shift when we experience a context  $X_{t}$ , for which there is no safe arm to play in the sense of  $(\star)$ . The following notation will be useful.
Notation. For the sake of succinctly identifying regions of time with experienced significant shifts, we will conflate the closed, open, and half-closed intervals of real numbers  $[a,b]$ ,  $(a,b)$ , and  $[a,b)$ , respectively, with the corresponding rounds contained therein, i.e.  $[a,b] \equiv [a,b] \cap \mathbb{N}$ .
Definition 6. Fix the context sequence  $X_{1},X_{2},\ldots ,X_{T}$
- We say an arm  $a \in [K]$  is unsafe at context  $x \in \mathcal{X}$  on  $I$  if there exists a bin  $B \in \mathcal{T}$  containing  $x$  such that arm  $a$  incurs significant regret ( $\star$ ) in bin  $B$  on  $I$ .
We then have the following recursive definition:
- Let  $\tau_0 = 1$ . Define the  $(i + 1)$ -th experienced significant shift as the earliest time  $\tau_{i + 1} \in (\tau_i, T]$  such that every arm  $a \in [K]$  is unsafe at  $X_t$  on some interval  $I \subset [\tau_i, \tau_{i + 1}]$ . We refer to intervals  $[\tau_i, \tau_{i + 1})$ ,  $i \geq 0$ , as experienced significant phases. The unknown number of such shifts (by time  $T$ ) is denoted  $\tilde{L}$ , whereby  $[\tau_{\tilde{L}}, \tau_{\tilde{L} + 1})$ , for  $\tau_{\tilde{L} + 1} \doteq T + 1$ , is the last phase.
Figure 1: Shown are the gap functions  $\delta_t^{1,2}(\cdot)$  ( $K = 2$  arms) in time  $t$  at two contexts  $x$ ,  $x'$  assumed to be far apart. Suppose at times  $t_1$  and  $t_3$ , we observe context  $x$ , and at times  $t_2$  and  $t_4$ , we observe  $x'$ . Then, the change in best arm at context  $x$  is experienced. To contrast, the change in best arm at  $x'$  is not experienced and hence not counted in Definition 6.
Remark 4 (Definition 6 Only Counts Most Essential Changes). It's clear from Definition 6 that only changes in mean rewards  $f_{t}^{a}(X_{t})$  at experienced contexts  $X_{t}$  are counted, and only counted when experienced (see example in Figure 1).
An experienced significant shift  $\tau_{i}$  in fact implies a best-arm change at  $X_{\tau_i}$  since, by smoothness (Assumption 1) and  $(\star)$ , we have