Title: Tracking Most Significant Shifts in Nonparametric Contextual Bandits
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-armed bandits

non-stationary

contextual bandits

nonparametric

Lipschitz
Abstract: We study nonparametric contextual bandits where Lipschitz mean reward functions may change over time.
We first establish the minimax dynamic regret rate in this less understood setting in terms of number of changes $L$ and total-variation $V$, both capturing all changes in distribution over context space, and argue that state-of-the-art procedures are suboptimal in this setting.

Next, we tend to the question of an _adaptivity_ for this setting, i.e. achieving the minimax rate without knowledge of $L$ or $V$. Quite importantly, we posit that the bandit  problem, viewed locally at a given context $X_t$, should not be affected by reward changes in other parts of context space $\cal X$. We therefore propose a notion of _change_, which we term _experienced significant shifts_, that better accounts for locality, and thus counts considerably less changes than $L$ and $V$. Furthermore, similar to recent work on non-stationary MAB (Suk & Kpotufe, 2022), _experienced significant shifts_ only count the most _significant_ changes in mean rewards, e.g., severe best-arm changes relevant to observed contexts.

Our main result is to show that this more tolerant notion of change can in fact be adapted to.

Tracking Most Significant Shifts in Nonparametric Contextual Bandits

Abstract

We study nonparametric contextual bandits where Lipschitz mean reward functions may change over time. We first establish the minimax dynamic regret rate in this less understood setting in terms of number of changes L and total-variation V , both capturing all changes in distribution over context space, and argue that state-of-the-art procedures are suboptimal in this setting.

Next, we tend to the question of an adaptivity for this setting, i.e. achieving the minimax rate without knowledge of L or V . Quite importantly, we posit that the bandit problem, viewed locally at a given context X t , should not be affected by reward changes in other parts of context space X . We therefore propose a notion of change , which we term experienced significant shifts , that better accounts for locality, and thus counts considerably less changes than L and V . Furthermore, similar to recent work on non-stationary MAB [Suk and Kpotufe, 2022], experienced significant shifts only count the most significant changes in mean rewards, e.g., severe best-arm changes relevant to observed contexts.

Our main result is to show that this more tolerant notion of change can in fact be adapted to.

1 Introduction

Contextual bandits model sequential decision making problems where the reward of a chosen action depends on an observed context X t at time t , e.g., a consumer's profile, a medical patient's history. The goal is to maximize the total rewards over time of chosen actions, as informed by seen contexts. As such, one suitable measure of performance is that of dynamic regret , which compares earned rewards to a time-varying oracle maximizing mean rewards at X t . While it is often assumed in the bulk of works in this setting that rewards distributions remain stationary over time, it is understood that in practice, environmental changes induce nontrivial changes in rewards.

In fact, the problem of non-stationary environments has received a surge of attention in the simpler non-contextual Multi-Arm-Bandits (MAB) setting, while the more challenging contextual case remains ill-understood. In particular in the contextual case, some recent works [Wu et al., 2018, Luo et al., 2018, Chen et al., 2019, Wei and Luo, 2021] consider parametric settings , i.e. where reward functions belong to fixed parametric family, and show that one may achieve rates adaptive to an unknown number of L of shifts in rewards or to a notion of total-variation V , both acccounting for all changes over time and context space. Instead here, we consider a much larger class of reward functions, namely Lipschitz rewards, corresponding to the natural assumption that closeby contexts have similar rewards even as reward distributions change.

As a first result for this nonparametric setting, we establish some minimax lower-bounds as a baseline in terms of either L or V , and argue that state-of-the-art procedures for the parametric case-extended to the class of Lipschitz functions-do not achieve these baselines.

We then turn attention to whether such baselines may be achieved adaptively , i.e., without knowledge of L or V . The answer as we show is affirmative, and more importantly, some much weaker notions of change may be adapted to; for intuition, while L or V accounts for any change at any time over the context space (say X ), it may be that all changes are relegated to parts of the space irrelevant to observed contexts X t at the time they are played. For instance, suppose at time t , we observe X t = x 0 , then it may not make sense to count changes that happen at some other x 1 far from x 0 , or changes that happened at x 0 itself but far back in time.

We therefore propose a new parameterization of change, termed experienced significant shifts that better accounts for the locality of changes in time and space, and as such may register much less changes than either L or V . As a sanity check, we show that an oracle policy which restarts only at experienced significant shifts can attain enhanced regret rates in terms of the number ˜ L = ˜ L ( X 1 , . . . , X T ) of such experienced shifts (Proposition 2), a rate always no worse that the baseline we first established in terms of L and V .

Our main result is to show that experienced significant shifts can be adapted to (Theorem 3), i.e., with no prior knowledge of such shifts. Importantly, the result holds in both stochastic environments, and in (oblivious) adversarial ones with no change to our notion, algorithmic approach, nor analysis. Furthermore, similar to recent advances in the non-contextual case [Abbasi-Yadkori et al., 2022, Suk and Kpotufe, 2022], an experienced shift is only triggered under severe changes such as changes of best arms locally at a context X t . An added difficulty in the contextual case is that we cannot hope to observe rewards for a given arm (action) repeatedly at X t as the context may only appear once, and have to rely on carefuly chosen nearby points to identify unknown shifts in reward at X t .

1.1 Other Related Work

Nonparametric Contextual Bandits. The stationary bandits with covariates (where rewards and contexts follow a joint distribution) was first introduced in a one-armed bandit problem [Woodroofe, 1979, Sarkar, 1991], with the nonparametric model first studied by Yang et al. [2002]. Minimax regret rates, based on a margin condition, were first established for the two-armed bandit in Rigollet and Zeevi [2010] and generalized to any finite number of arms in Perchet and Rigollet [2013], with further insights thereafter [Qian and Yang, 2016a,b, Reeve et al., 2018, Guan and Jiang, 2018, Hu et al., 2020, Arya and Yang, 2020, Suk and Kpotufe, 2021, Gur et al., 2022, Cai et al., 2022]. However, the mentioned works all assume a stationary distribution of rewards over contexts. Blanchard et al. [2023] studies non-stationary nonparametric contextual bandits, but in the much-different context of universal learning, concerning when sublinear regret is achievable asymptotically. Lipschitz contextual bandits also appears as part of studies on broader infinite-armed settings [Lu et al., 2009, Krishnamurthy et al., 2019]. Related, Slivkins [2014] allows for non-stationary (i.e., obliviously adversarial) environments, but only studies regret to the (per-context) best arm in hindsight. Realizable contextual bandits posits that the regression function capturing mean rewards in contexts lies in some known class of regressors F , over which one can do empirical risk minimization [Foster et al., 2018, Foster and Rakhlin, 2020, Simchi-Levi and Xu, 2021]. While this setting recovers Lipschitz contextual bandits, the only applicable non-stationary guarantee to our knowledge is Wei and Luo [2021], which yields suboptimal dynamic regret (see Table 1).

Non-Stationary Bandits and RL. In the simpler non-contextual bandits, changing reward distributions (a.k.a. switching bandits ) was introduced in Garivier and Moulines [2011] and further explored with various assumptions and formulations [Karnin and Anava, 2016, Allesiardo et al., 2017, Liu et al., 2018, Wei and Srivatsva, 2018, Besbes et al., 2019, Cao et al., 2019, Mukherjee and Maillard, 2019, Besson et al., 2022]. While these earlier works focused on algorithmic design assuming knowledge of non-stationarity, such a strong assumption was removed via the adaptive procedures of Auer et al. [2019], Chen et al. [2019]. In followup works, Abbasi-Yadkori et al. [2022], Suk and Kpotufe [2022] show that tighter dynamic regret rates are possible, scaling only with severe changes in best arm. The ideas from non-stationary MAB were extended to various contextual bandit settings by Wu et al. [2018] (for linear mean rewards in contexts), Luo et al. [2018], Chen et al. [2019] (for finite policy classes), and Wei and Luo [2021] (for realizable mean reward functions). There have also been extensions to various reinforcement learning setups [Jaksch et al., 2010, Gajane et al., 2018, Chi Cheung et al., 2019, Ortner et al., 2020, Fei et al., 2020, Cheung et al., 2020, Touati and Vincent, 2020, Domingues et al., 2021, Mao et al., 2021, Domingues et al., 2021, Wei and Luo, 2021, Zhou et al., 2022, Lykouris et al., 2021, Wei et al., 2022, Chen and Luo, 2022, Ding and Lavaei, 2023].

Of these works, only Domingues et al. [2021] can recover Lipschitz contextaul bandits, whereupon we find their dynamic regret bounds are suboptimal (see Table 1). Again, the typical aim of the aforementioned works on contextual bandits or RL is to minimize a notion of dynamic regret in terms of the number of changes L or total-variation V . As such, the guarantees of such works don't involve tighter notions of experienced non-stationarity, such as those studied in this work.

2 Problem Formulation

2.1 Contextual Bandits with Changing Rewards

Preliminaries. We assume a finite set of arms [ K ] . = { 1 , 2 . . . , K } . Let Y t ∈ [0 , 1] K denote the vector of rewards for arms a ∈ [ K ] at round t ∈ [ T ] (horizon T ), and X t the observed context at that round, lying in X . = [0 , 1] d , which have joint distribution ( X t , Y t ) ∼ D t . We let X t . = { X s } s ≤ t , Y t . = { Y s } s ≤ t denote the observed contexts and (observed and unobserved) rewards from rounds 1 to t . In our setting, an oblivious adversary decides a sequence of (independent) distributions on { ( X t , Y t ) } t ∈ [ T ] before the first round.

Notation. The reward function f t : X → [0 , 1] K is f a t ( x ) . = E [ Y a t | X t = x ] , a ∈ [ K ] , and captures the mean rewards of arm a at context x and time t .

A policy chooses actions at each round t , based on observed contexts (up to round t ) and passed rewards, whereby at each round t only the reward Y a t of the chosen action a is revealed. Formally:

Definition 1 (Policy) . A policy π . = { π t } t ∈ N is a random sequence of functions π t : X t × [ K ] t -1 × [0 , 1] t -1 → [ K ] . A randomized policy π t maps to distributions on [ K ] , In an abuse of notation, in the context of a sequence of observations till round t , we'll let π t ∈ [ K ] denote the (possibly random) action chosen at round t .

The performance of a policy is evaluated using the dynamic regret , defined as follows:

Definition 2. Fix a context sequence X T . Define the dynamic regret of a policy π , as

So, we aim to minimize E [ R T ( π, X T )] where the expectation is over X T , Y T , and randomness in π .

Notation. As much of our analysis focuses on the gaps in mean rewards between arms at observed contexts X t , the following notation will serve useful. Let δ t ( a ′ , a ) . = f a ′ t ( X t ) -f a t ( X t ) denote the relative gap of arms a to a ′ at round t at context X t . Define the worst gap of arm a as δ t ( a ) . = max a ′ ∈ [ K ] δ t ( a ′ , a ) , corresponding to the instantaneous regret of playing a at round t and context X t . Thus, the dynamic regret can be written as ∑ t ∈ [ T ] E [ δ t ( π t )] . Additionally, we will occasionally talk about the gap functions in context x ∈ X . In an abuse of notation, let δ a ′ ,a t ( x ) . = f a ′ t ( x ) -f a t ( x ) and δ a t ( x ) . = max a ′ ∈ [ K ] δ a ′ ,a t ( x ) . Generally, when a context x is not specified, as in the quantities δ t ( a ′ , a ) , δ t ( a ) , it should be assumed that the context in question is X t .

2.2 Nonparametric Setting

We assume, as in prior work on nonparametric contextual bandits [Rigollet and Zeevi, 2010, Perchet and Rigollet, 2013, Slivkins, 2014, Reeve et al., 2018, Guan and Jiang, 2018, Suk and Kpotufe, 2021], that the reward function is 1 -Lipschitz.

Assumption 1 (Lipschitz f t ) . For all rounds t ∈ N , a ∈ [ K ] and x, x ′ ∈ X ,

For ease of presentation, we assume the contextual marginal distribution µ X remains the same across rounds. Furthermore, we make a standard strong density assumption on µ X , which is typical in this nonparametric setting [Audibert and Tsybakov, 2007, Perchet and Rigollet, 2013, Qian and Yang, 2016a,b, Gur et al., 2022, Hu et al., 2020, Arya and Yang, 2020, Cai et al., 2022]. This holds, e.g. if µ X has a continuous Lebesgue density on [0 , 1] d , and ensures good coverage of the context space.

Assumption 2 (Strong Density Condition) . There exist C d , c d > 0 s.t. ∀ ℓ ∞ balls B ⊂ [0 , 1] d of diameter r ∈ (0 , 1] :

Remark 1. We can in fact relax Assumption 2 so that µ X,t ( · ) is changing in t and (2) is satisfied with different constants C d,t , c d,t . Our procedures in the end will not require knowledge of any C d,t , c d,t .

2.3 Model Selection

A common algorithmic approach in nonparametric contextual bandits, starting from earlier work [Rigollet and Zeevi, 2010, Perchet and Rigollet, 2013], is to discretize or partition the context space X into bins where we can maintain local reward estimates. These bins have a natural hierarchical tree structure which we first elaborate.

Definition 3 (Partition Tree) . Let R . = { 2 -i : i ∈ N ∪ { 0 }} , and let T r , r ∈ R denote a regular partition of [0 , 1] d into hypercubes (which we refer to as bins ) of side length (a.k.a. bin size) r . We then define the dyadic tree T . = {T r } r ∈R , i.e., a hierarchy of nested partitions of [0 , 1] d . We will refer to the level r of T as the collection of bins in partition T r . The parent of a bin B ∈ T r , r < 1 is the bin B ′ ∈ T 2 r containing B ; child , ancestor and descendant relations follow naturally. We'll use T r ( x ) to refer to the bin at level r containing x , while r ( B ) is the side length of bin B .

Note that, while in the above definition, T has infinite levels r ∈ R , at any round t in a procedure, we implicitly only operate on the subset of T containing data.

Key in securing good regret is then finding the optimal level r ∈ R of discretization (balancing local regression bias and variance), which over T stationary rounds is known to be ∝ ( K/T ) 1 2+ d [Rigollet and Zeevi, 2010]. The intuition for this choice is that a bias of T -1 2+ d is safe to pay for T rounds to maintain a minimax regret rate of T 1+ d 2+ d (see rates in Theorem 1). We introduce the following general notation, useful later in the non-stationary problem, for associating a level with an intervals of rounds.

Notation 1 (Level) . For n ∈ N ∪ { 0 } , let r n be the largest 2 -m ∈ R such that ( K/n ) 1 2+ d ≥ 2 -m . When I is an interval of rounds, we use r I as shorthand for r | I | , where | I | is the length of I .

We use T m , T m ( x ) as shorthand to denote (respectively) the tree T r of level r = r m and the (unique) bin at level r m containing x .

3 Results Overview

3.1 Minimax Lower Bounds Under Global Shifts

As a baseline, we start with some basic lower-bounds under the simplest parametrizations of changes in rewards which have appeared in the literature, namely a global number of shifts , and total variation .

̸

Definition 4 (Global Number of Shifts) . Let L . = ∑ T t =2 1 {∃ x ∈ X , a ∈ [ K ] : f a t ( x ) = f a t -1 ( x ) } be the number of global shifts , i.e., it counts every change in mean-reward overtime and over X space.

Definition 5 (Total Variation) . Define V T . = ∑ T t =2 ∥D t -D t -1 ∥ TV where recall D t ∈ X × [0 , 1] K is the joint distribution on context and rewards at time t .

We have the following initial result (for two-armed bandits) to serve as baseline for this study.

Theorem 1 (Dynamic Regret Lower Bound) . Suppose there are K = 2 arms. For V, L ∈ [0 , T ] , let P ( V, L, T ) be the family of joint distributions D . = {D t } t ∈ [ T ] with either total variation V T ≤ V or at most L global shifts. Then, there exists a constant c > 0 such that for any policy π :

Remark 2. Note setting d = 0 in Theorem 1 recovers the minimax rate ( √ T + T 2 / 3 V 1 / 3 T ) ∧ √ ( L +1) · T for non-contextual bandits [Besbes et al., 2019].

Achievability of Minimax Lower-Bound (3). We are interested in whether the rates of (3) are achievable, with, or without knowledge of relevant parameters. First, we note that no existing algorithm currently guarantees a rate that matches (3). See Table 1 for a rate comparison (details for specializing to our setting found in Appendix A).

In particular, the prior adaptive works [Chen et al., 2019, Wei and Luo, 2021] both rely on the approach of randomly scheduling replays of stationary algorithms to detect unknown non-stationarity. However, the scheduling rate is designed to safeguard against the parametric √ LT ∧ V 1 / 3 T T 2 / 3 regret rates and thus lead to suboptimal dependence on L and V T .

However, a simple back of the envelope calculation indicates that the rate in (3) may be attainable, at least given some distributional knowledge: a minimax-optimal stationary procedure restarted at each shift will incur regret, over L equally spaced shifts, ( L +1) · ( T L +1 ) 1+ d 2+ d ≈ ( L +1) 1 2+ d · T 1+ d 2+ d .

As it turns out as we will show in the next section, (3) is indeed attainable, even adaptively; in fact, this is shown via a more optimistic problem parametrization as described next.

3.2 A New Problem Parametrization: Experienced Significant Shifts.

As discussed in Subsection 2.3, typical approaches [Rigollet and Zeevi, 2010, Perchet and Rigollet, 2013] in our setting discretize the context space X into bins, each of which is treated as an MAB instance. At a high level, our new measure of non-stationarity will trigger an experienced significant shift when the observed context X t arrives in a bin B ∈ T where there has been a severe change in local best arm, w.r.t. the observed data in that bin .

We first define a notion of significant regret for an arm a ∈ [ K ] locally within a bin B ∈ T . We say arm a incurs significant regret in bin B on interval I if:

where n B ( I ) . = ∑ s ∈ I 1 { X s ∈ I } and recall r ( B ) is the side length of bin B . The intuition for ( ⋆ ) is as follows: suppose that, over n separate rounds, we observe the same context X s = x 0 in bin B . Then, arm a would be considered unsafe in the local bandit problem at context x 0 if its regret exceeds √ K · n (i.e., the first term on the above R.H.S.), which is a safe regret to pay for the non-contextual problem. Our broader notion ( ⋆ ) extends this over the bin B by also accounting for the bias (i.e., the second term on the above R.H.S.) of observing X s near a given context x 0 ∈ B .

̸

Remark 3 (Significant Regret Occurs at Critical Levels) . If ( ⋆ ) holds for some bin B , then in fact it also roughly holds for the bin B ′ , with B ′ ∩ B = ∅ , at the critical level r | I | (see Notation 1) w.r.t. interval I (Lemma 9 and Proposition 15). Thus, it suffices to only check ( ⋆ ) for the critical levels r | I | , which will be crucial in the analysis. Additionally, all such critical levels r | I | are above the optimal level T -1 2+ d for T stationary rounds (see Subsection 2.3). Thus, only O (log( T )) levels play a role in in checking ( ⋆ ) for all intervals of time I and bins B .

We then propose to record an experienced significant shift when we experience a context X t , for which there is no safe arm to play in the sense of ( ⋆ ). The following notation will be useful.

Notation. For the sake of succinctly identifying regions of time with experienced significant shifts, we will conflate the closed, open, and half-closed intervals of real numbers [ a, b ] , ( a, b ) , and [ a, b ) , respectively, with the corresponding rounds contained therein, i.e. [ a, b ] ≡ [ a, b ] ∩ N .

Definition 6. Fix the context sequence X 1 , X 2 , . . . , X T .

We say an arm a ∈ [ K ] is unsafe at context x ∈ X on I if there exists a bin B ∈ T containing x such that arm a incurs significant regret ( ⋆ ) in bin B on I .

We then have the following recursive definition:

Let τ 0 = 1 . Define the ( i +1) -th experienced significant shift as the earliest time τ i +1 ∈ ( τ i , T ] such that every arm a ∈ [ K ] is unsafe at X t on some interval I ⊂ [ τ i , τ i +1 ] . We refer to intervals [ τ i , τ i +1 ) , i ≥ 0 , as experienced significant phases . The unknown number of such shifts (by time T ) is denoted ˜ L , whereby [ τ ˜ L , τ ˜ L +1 ) , for τ ˜ L +1 . = T +1 , is the last phase.

(#)

0

timet

Remark 4 (Definition 6 Only Counts Most Essential Changes) . It's clear from Definition 6 that only changes in mean rewards f a t ( X t ) at experienced contexts X t are counted, and only counted when experienced (see example in Figure 1).

An experienced significant shift τ i in fact implies a best-arm change at X τ i since, by smoothness (Assumption 1) and ( ⋆ ), we have

Thus, ˜ L ≤ L +1 , the global count of shifts, and can in fact be much smaller. On the other hand, so long as an experienced significant shift does not occur, there will be arms safe to play at each context X t . Thus, procedures need not restart exploration so long as unsafe arms can be quickly ruled out.

As a warmup to presenting our main regret bounds and algorithms, we'll first consider an oracle procedure which knows the experienced significant shift times τ i . The strategy will be to mimic a successive elimination algorithm, restarted at each experienced significant shift.

Definition 7 (Oracle Procedure) . For each round t in phase [ τ i , τ i +1 ) , define a good arm set G t as the set of safe arms, i.e., arms which do not yet satisfy ( ⋆ ) in the bin T r ( X t ) at level r = r τ i +1 -τ i containing X t . Here, recall from Subsection 2.3 that r τ i +1 -τ i is the oracle choice of level over phase [ τ i , τ i +1 ) ). Then, define an oracle procedure π : playing a uniformly random arm a ∈ G t at time t .

We then claim such an oracle procedure attains an enhanced dynamic regret rate in terms of the significant shifts { τ i } i which recovers the minimax lower bound in terms of global shifts L and total variation V T from before. The following proposition (see Appendix C for proof) captures this.

Proposition 2 (Sanity Check) . We have the oracle procedure π of Definition 7 satisfies with probability at least 1 -1 /T 2 w.r.t. the randomness of X T : for some C > 0

By Jensen's inequality the above regret rate is at most ( ˜ L ( X T ) + 1) 1 2+ d · T 1+ d 2+ d ≪ ( L +1) 1 2+ d · T 1+ d 2+ d . At the same time, the rate is also faster than V 1 3+ d T T 2+ d 3+ d (see Corollary 5). Thus, the oracle procedure above attains the dynamic regret lower bound of Theorem 1. We next aim to design an algorithm which can attain same order regret without knowledge of τ i or ˜ L .

3.3 Main Results: Adaptive Upper-bounds

Our main result is a dynamic regret upper bound of similar order to Proposition 2 without knowledge of the environment, e.g., the significant shift times, or the number of significant phases . It is stated for our algorithm CMETA (Algorithm 1 of Section 4), which, for simplicity, requires knowledge of the time horizon T (knowledge of T removable using doubling tricks).

Theorem 3. Let { τ i ( X T ) } ˜ L +1 i =0 denote the unknown experienced significant shifts (Definition 6). We then have w.p. at least 1 -1 /T 2 w.r.t. the randomness of X T , for some C > 0 :

Corollary 4 (Adapting to Experienced Significant Shifts) . Under the conditions of Theorem 3, with probability at least 1 -1 /T 2 w.r.t. the randomness in X T :

Note, this is tighter than the earlier mentioned ( L +1) 1 2+ d T 1+ d 2+ d rate. The next corollary asserts that Theorem 3 also recovers the optimal rate in terms of total-variation V T .

Corollary 5 (Adapting to Total Variation) . Under the conditions of Theorem 3:

Remark 5. Our regret bound can straightforwardly be generalized to min { ( ˜ L +1) β 2 β + d · T β + d 2 β + d · K β 2 β + d , ( V T · K ) β 3 β + d · T 2 β + d 3 β + d + T β + d 2 β + d · K β 2 β + d } for β -Hölder reward functions with β ≤ 1 , provided the notion ( ⋆ ) is modified to take into account a bias of r β ( B ) .

4 Algorithm

We take a similar algorithmic approach to Suk and Kpotufe [2022], with several important modifications for our setting. The high-level strategy is to schedule multiple copies of a base algorithm (Algorithm 2) at random times and durations, in order to ensure updated and reliable estimation of the gaps in ( ⋆ ). This allows fast enough detection of unknown experienced significant shifts.

Overview of Algorithm Hierarchy. Our main algorithm CMETA (Algorithm 1) proceeds in episodes, each of which begins by playing according to an initially scheduled base algorithm of possible duration equal to the number of rounds left till T . Base algorithms occasionally activate their own base algorithms of varying durations (Line 9 of Algorithm 2), called replays , according to a random schedule (set via variables { Z m,s } ). We refer to the currently playing base algorithm as the active base algorithm . This induces a hierarchy of base algorithms, from parent to child instances.

Choice of Level. Focusing on a single base algorithm now, each Base-Alg manages its own discretization of the context space X = [0 , 1] d , corresponding to a level r ∈ R (see Definition 3). Within each bin B ∈ T r at the level r , candidate arms, maintained in a set A ( B ) , are evicted according to importance-weighted estimates (4) of local gaps.

As discussed in Subsection 2.3, key in algorithmic design is determining the optimal level r ∈ R . An immediate difficulty is that the oracle procedure's choice of level (see Definition 7) depends on the unknown significant phase length τ i +1 -τ i . To circumvent this, and as in previous works on Lipschitz contextual bandits [Perchet and Rigollet, 2013, Slivkins, 2014], we rely on an adaptive time-varying choice of level r t . Specifically, each base algorithm uses the level r t -t start based on the time elapsed since the time t start it was first activated.

Managing Multiple Base Algorithms. Instances of Base-Alg and CMETA share information, in the form of global variables as listed below:

All variables defined in CMETA : t ℓ , t, {A master ( B ) } B ∈T , { Z m,t } (see Lines 3-6 of Algorithm 1).

The choice of arm played at each round t , along with observed rewards Y a t , and the candidate arm set A t which takes value the set A ( B ) of the active Base-Alg at round t and bin B = T r ( X t ) used.

By sharing these global variables, any Base-Alg can trigger a new episode: once an arm is evicted from a Base-Alg 's A ( B ) , it is also evicted from A master ( B ) , which is essentially the candidate arm set for the current episode. A new episode is triggered at time t when A master ( B ) becomes empty for some bin B (necessarily a currently experienced bin), i.e., there is no safe arm left to play at the context X t in the sense of Definition 6. Note that A ( B ) are local variables internal to each Base-Alg (the owner of which will be clear from context in usage).

To ensure consistent behavior while using a time-varying choice of level, we enforce further regularity in arm evictions across X : arms evicted from A ( B ′ ) are also evicted from child bins B ⊆ B ′ to ensure A ( B ) ⊆ A ( B ′ ) .

Algorithm 1: C ontextual M etaE limination while T racking A rms ( CMETA )

8 if t < T then restart from Line 2 (i.e. start a new episode). ;

Estimating Aggregate Local Gaps. ∑ s 2 s = s 1 δ s ( a ′ , a ) · 1 { X s ∈ B } is estimated by ∑ s 2 s = s 1 ˆ δ B s ( a ′ , a ) , where δ s ( a ′ , a ) · 1 { X s ∈ B } is estimated by importance weighting as:

Note that the above is an unbiased estimate of δ s ( a ′ , a ) · 1 { X s ∈ B } whenever a ′ and a are both in A s at time s , conditional on the context X s . It then follows that, conditional on X T , the difference ∑ s 2 s = s 1 ( ˆ δ B s ( a ′ , a ) · 1 { X s ∈ B } -δ s ( a ′ , a ) ) is a martingale that concentrates at a rate of order roughly √ K · n B ([ s 1 , s 2 ]) , where recall from earlier that n B ( I ) . = ∑ s ∈ I 1 { X s ∈ I } is the context count in bin B over interval I .

An arm a is then evicted at round t if, for some fixed C 0 > 0 1 , ∃ rounds s 1 < s 2 ≤ t such that at level r s 2 -s 1 and letting B . = T s 2 -s 1 ( X t ) (i.e., the bin at level r s 2 -s 1 containing X t )

5 Key Technical Highlights of Analysis

While a full analysis is deferred to Appendix D, we highlight key novelties and intuitive calculations.

Local Safety in Bins implies Safe Total Regret. We first argue that the notion of significant regret ( ⋆ ) within a bin B captures the total allowable regret rates T 1+ d 2+ d we wish to compete with over T 'safe' rounds. If ( ⋆ ) holds for no intervals [ s 1 , s 2 ] in all bins B , arm a would be safe and incur little regret over any [ s 1 , s 2 ] . As it turns out, bounding the per-bin regret by ( ⋆ ) implies a total regret of T 1+ d 2+ d as seen from the following rough calculation: via concentration and the strong density assumption (Assumption 2) to conflate n B ([1 , T ]) ≈ r ( B ) d · T and the fact that there are ≈ r -d bins at level r , we have:

In particular taking r ∝ ( K/T ) 1 2+ d makes the above R.H.S. the desired rate K 1 2+ d T 1+ d 2+ d .

Significant Regret Threshold is Estimation Error. At the same time, the R.H.S. of ( ⋆ ) is a standard variance and bias bound on the regression error of estimating the cumulative regret ∑ s 2 s = s 1 δ a s ( x ) · 1 { X s ∈ B } at any context x ∈ B , conditional on X T (see Lemma 6). Thus, intuitively, large gaps of magnitude above the threshold √ K · n B ( I ) + r ( B ) · n B ( I ) in ( ⋆ ) are detectable via the estimates of (4).

Combining the above two points, we conclude that the notion of significant regret ( ⋆ ) balances both (1) detection of unsafe arms and (2) regret of playing non-evicted arms. We next argue the randomized scheduling of multiple base algorithms is suitable for detecting experienced significant shifts.

A New Balanced Replay Scheduling. As mentioned earlier in Subsection 3.1, previous adaptive works on contextual bandits fail to attain the optimal regret in this setting due to an inappropriate frequency of scheduling re-exploration. We introduce a novel scheduling (Line 6 of Algorithm 1) of replays which carefully balances exploration and fast detection of significant regret in the sense of ( ⋆ ). In particular, the determined probability (1 /m ) 1 2+ d (1 /t ) 1+ d 2+ d of scheduling a new Base-Alg ( t, m ) comes from the following intuitive calculation: a single replay of duration m will, if scheduled, incur an additional regret of about m 1+ d 2+ d . Then, summing over all possible replays, the total extra regret incurred due to scheduled replays is roughly upper bounded by

In other words, the cost of replays only incurs extra constants in the regret. Surprisingly, we find this scheduling rate is also sufficient for detecting significant regret in any experienced subregion B of the context space X , i.e. there is no need to do additional exploration on a localized per-bin basis.

Key in this observation is the fact that, to detect significant regret over interval I in any bin B , it suffices to check it at the critical level r I ∝ ( K/ | I | ) 1 2+ d , where | I | is the length of I . In particular, a well-timed Base-Alg running on the interval I will use this level r I (Line 2 of Algorithm 2) and is, thus, equipped to detect significant regret at all experienced bins over I .

Suffices to Only Check ( ⋆ ) at Critical Levels r I . At first glance, detecting experienced significant shifts (Definition 6) appears difficult as an arm a may incur significant regret over a different bin B ′ from the bin B that is currently being used by the algorithm.

We in fact show that it suffices to only estimate the R.H.S. of ( ⋆ ) in bins B ′ at the critical level r I (Lemma 9 and Proposition 15). We give a rough argument for why this is the case: first, note that ( ⋆ ) may be rewritten as

We next relate the two sides of the above display across different levels r ( B ) .

By concentration and the strong density assumption (Assumption 2), the R.H.S. of (7) is in fact of order ∝ 1 / √ | I | · r ( B ) d + r ( B ) , which is minimized at the critical level r ( B ) ∝ r I .

The L.H.S. of (7) is an estimate of the average gap 1 | I | ∑ s ∈ I δ a s ( x ) at any particular x ∈ B using nearby contexts. In fact, by concentration, the two can be conflated up to error terms of order the R.H.S. of (7).

̸

Combining the above two points, we see that if (7) holds for some bin B , then it will also hold for the 'critical bin' B ′ , with B ′ ∩ B = ∅ , at the critical level r I ∝ ( K/ | I | ) 1 2+ d . In other words, significant regret at any experienced bin implies significant regret at the critical level, thus allowing us to constrain attention to these critical levels.

On the other hand, we observe that the calculations in (6) would hold if we only checked ( ⋆ ) for bins B at level r I . Thus, it also suffices to only use the levels r I for regret minimization over intervals I with no experienced significant shift.

Yet, even still, the analysis is challenging as there may be 'missing data problems': arms a ∈ A ( B ) in contention at B may have been evicted from sibling bins inside the parent B ′ ⊃ B at the critical level. In other words, it is not a priori obvious how to do reliable estimation of arms a ∈ A ( B ) across a larger bin B ′ which may contain sub-regions where a has already been evicted. We show it is in fact possible to identify a subclass of intervals of rounds (Definition 12) and an associated class of replays (Definition 13) which can quickly evict arm a in the critical bin B ′ before there are missing data problems for bin B ⊆ B ′ . The details of this can be found in Proposition 15 of Appendix D.2.

6 Conclusion

We have shown that it is possible to adapt optimally to an unknown number of experienced significant shifts - a new notion introduced here - which captures severe changes in best-arm, only at observed contexts. An interesting future direction is to explore other notions of experienced shifts which may yield even more optimistic rates. For example, suppose changes in best arm occur at every round, but are localized to a sub-region Ξ of the context space X . Then, a procedure which discretizes X into bins at level T -1 2+ d and runs local instantiations of a suitable non-stationary MAB algorithm (e.g., META of Suk and Kpotufe [2022]) can attain faster rates than those of Theorem 3 for some choices of Ξ . At the same time, such a strategy cannot always attain the ˜ L ( X T ) 1 2+ d · T 1+ d 2+ d rate in general as using the level T -1 2+ d is insufficient to detect experienced significant shifts occurring at short intervals I of length | I |≪ T (which require larger levels). This prompts the questions of whether there exists a unified notion of experienced shift which captures the most optimistic rates in these scenarios and whether such a notion can be adapted to.

References

Yasin Abbasi-Yadkori, András György, and Nevena Lazic. A new look at dynamic regret for non-stationary stochastic bandits. arXiv preprint arXiv:2201.06532 , 2022.

Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. volume 32 of Proceedings of Machine Learning Research , pages 1638-1646. PMLR, 22-24 Jun 2014.

Robin Allesiardo, Raphaël Féraud, and Odalric-Ambrym Maillard. The non-stationary stochastic multi-armed bandit problem. International Journal of Data Science and Analytics , 3(4):267-283, 2017.

Sakshi Arya and Yuhong Yang. Randomized allocation with nonparametric estimation for contextual multi-armed bandits with delayed rewards. Statistics & Probability Letters , 164:108818, 2020. ISSN 0167-7152.

Jean-Yves Audibert and Alexander B Tsybakov. Fast learning rates for plug-in classifiers. The Annals of Statistics , 35(2):608-633, 2007.

Peter Auer, Pratik Gajane, and Ronald Ortner. Adaptively tracking the best bandit arm with an unknown number of distribution changes. Conference on Learning Theory , pages 138-158, 2019.

Omar Besbes, Yonatan Gur, and Assaf Zeevi. Optimal exploration-exploitation in a multi-armedbandit problem with non-stationary rewards. Stochastic Systems , 9(4):319-337, 2019.

Lilian Besson, Emilie Kaufmann, Odalric-Ambrym Maillard, and Julien Seznec. Efficient changepoint detection for tackling piecewise-stationary bandits. Journal of Machine Learning Research , 23(77):1-40, 2022.

Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert E. Schapire. Contextual bandit algorithms with supervised learning guarantees. AISTATS , 2011.

Moise Blanchard, Steve Hanneke, and Patrick Jaillet. Non-stationary contextual bandits and universal learning. arXiv preprint arXiv:2302.07186 , 2023.

Changxiao Cai, T. Tony Cai, and Hongzhe Li. Transfer learning for contextual multi-armed bandits. arxiv preprint arXiv:2211.12612 , 2022.

Yang Cao, Zheng Wen, Branislav Kveton, and Yao Xie. Nearly optimal adaptive procedure with change detection for piecewise-stationary bandit. Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS) , 2019.

Liyu Chen and Haipeng Luo. Near-optimal goal-oriented reinforcement learning in non-stationary environments. Advances in Neural Information Processing Systems , 2022.

Yifang Chen, Chung-Wei Lee, Haipeng Luo, and Chen-Yu Wei. A new algorithm for non-stationary contextual bandits: efficient, optimal, and parameter-free. In 32nd Annual Conference on Learning Theory , 2019.

Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Reinforcement learning for non-stationary Markov decision processes: The blessing of (more) optimism. In International Conference on Machine Learning , pages 1843-1854. PMLR, 2020.

Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Hedging the drift: learning to optimize under non-stationarity. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics , 2019.

Yuhao Ding and Javad Lavaei. Provably efficient primal-dual reinforcement learning for CMDPs with non-stationary objectives and constraints. AAAI Conference on Artificial Intelligence (AAAI) , 2023.

Omar Darwiche Domingues, Pierre Ménard, Matteo Pirotta, Emilie Kaufmann, and Michal Valko. A kernel-based approach to non-stationary reinforcement learning in metric spaces. In International Conference on Artificial Intelligence and Statistics , pages 3538-3546. PMLR, 2021.

Miroslav Dudik, Daniel Hsu, Satyen Kale, Nikos Karampatziakis, John Langford, Lev Reyzin, and Tong Zhang. Efficient optimal learning for contextual bandits. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence , page 169-178. AUAI Press, 2011.

Yingjie Fei, Zhuoran Yang, Zhaoran Wang, and Qiaomin Xie. Dynamic regret of policy optimization in non-stationary environments. Advances in Neural Information Processing Systems , 33:67436754, 2020.

Dylan Foster and Alexander Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. Proceedings of the 37th International Conference on Machine Learning , 119: 3199-3210, 2020.

Dylan Foster, Alekh Agarwal, Miroslav Dudik, Haipeng Luo, and Robert Schapire. Practical contextual bandits with regression oracles. In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1539-1548. PMLR, 10-15 Jul 2018.

Pratik Gajane, Ronald Ortner, and Peter Auer. A sliding-window algorithm for Markov decision processes with arbitrarily changing rewards and transitions. arXiv preprint arXiv:1805.10066 , 2018.

Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for switching bandit problems. In Proceedings of the 22nd International Conference on Algorithmic Learning Theory , pages 174-188. ALT 2011, Springer, 2011.

Melody Y Guan and Heinrich Jiang. Nonparametric stochastic contextual bandits. AAAI , 2018.

Yonatan Gur, Ahmadreza Momeni, and Stefan Wager. Smoothness-adaptive contextual bandits. Operations Research , 70(6):3198-3216, 2022.

Yichun Hu, Nathan Kallus, and Xiaojie Mao. Smooth contextual bandits: Bridging the parametric and non-differentiable regret regimes. Conference on Learning Theory , 2020.

Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research , 11:1563-1600, 2010.

Zohar S Karnin and Oren Anava. Multi-armed bandits: Competing with optimal sequences. In Advances in Neural Information Processing Systems , pages 199-207, 2016.

Akshay Krishnamurthy, John Langford, Aleksandrs Slivkins, and Chicheng Zhang. Contextual bandits with continuous actions: Smoothing, zooming, and adapting. In Proceedings of the ThirtySecond Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 2025-2027. PMLR, 25-28 Jun 2019.

John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In Advances in neural information processing systems , pages 817-824, 2008.

Fang Liu, Joohyun Lee, and Ness Shroff. A change-detection based framework for piecewisestationary multi-armed bandit problem. Proceedings of the AAAI Conference on Artificial Intelligence , 2018.

Tyler Lu, Dávid Pál, and Martin Pál. Showing relevant ads via context multi-armed bandits. In Proceedings of AISTATS , 2009.

Haipeng Luo, Chen-Yu Wei, Alekh Agarwal, and John Langford. Efficient contextual bandits in non-stationary worlds. In Conference On Learning Theory , pages 1739-1776. PMLR, 2018.

Thodoris Lykouris, Max Simchowitz, Alex Slivkins, and Wen Sun. Corruption-robust exploration in episodic reinforcement learning. In Conference on Learning Theory , pages 3242-3245. PMLR, 2021.

Weichao Mao, Kaiqing Zhang, Ruihao Zhu, David Simchi-Levi, and Tamer Basar. Near-optimal model-free reinforcement learning in non-stationary episodic mdps. In International Conference on Machine Learning , pages 7447-7458. PMLR, 2021.

Subhojyoti Mukherjee and Odalric-Ambrym Maillard. Distribution-dependent and time-uniform bounds for piecewise i.i.d bandits. Reinforcement Learning for Real Life (RL4RealLife) Workshop in the 36th International Conference on Mearning Learning , 2019.

Ronald Ortner, Pratik Gajane, and Peter Auer. Variational regret bounds for reinforcement learning. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference , volume 115 of Proceedings of Machine Learning Research , pages 81-90. PMLR, 22-25 Jul 2020.

Vianney Perchet and Philippe Rigollet. The multi-armed bandit problem with covariates. The Annals of Statistics , 41(2):693-721, 2013.

Yury Polyanskiy and Yihong Wu. Information Theory: From Coding to Learning . Cambridge University Press, 2022.

Wei Qian and Yuhong Yang. Kernel estimation and model combination in a bandit problem with covariates. Journal of Machine Learning Research , 17(149):1-37, 2016a.

Wei Qian and Yuhong Yang. Randomized allocation with arm elimination in a bandit problem with covariates. Electronic Journal of Statistics , 10(1):242 - 270, 2016b.

Henry Reeve, Joe Mellor, and Gavin Brown. The k-nearest neighbour ucb algorithm for multi-armed bandits with covariates. In Proceedings of Algorithmic Learning Theory , volume 83 of Proceedings of Machine Learning Research , pages 725-752. PMLR, 07-09 Apr 2018.

Phillipe Rigollet and Assaf Zeevi. Nonparametric bandits with covariates. COLT , 2010.

Jyotirmoy Sarkar. One-armed bandit problems with covariates. The Annals of Statistics , pages 1978-2002, 1991.

David Simchi-Levi and Yunzong Xu. Bypassing the monster: a faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research , 47(3):1904-1931, 2021.

Aleksandrs Slivkins. Contextual bandits with similarity information. The Journal of Machine Learning Research , 15(1):2533-2568, 2014.

Joe Suk and Samory Kpotufe. Tracking most significant arm switches in bandits. In Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 2160-2182. PMLR, 02-05 Jul 2022.

Joseph Suk and Samory Kpotufe. Self-tuning bandits over unknown covariate-shifts. International Conference on Algorithmic Learning Theory , 2021.

Ahmed Touati and Pascal Vincent. Efficient learning in non-stationary linear Markov decision processes. arXiv preprint arXiv:2010.12870 , 2020.

Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. Proceedings of the 32nd International Conference on Learning Theory , 2021.

Chen-Yu Wei, Christoph Dann, and Julian Zimmert. A model selection approach for corruption robust reinforcement learning. In International Conference on Algorithmic Learning Theory , pages 1043-1096. PMLR, 2022.

Lai Wei and Vaihbav Srivatsva. On abruptly-changing and slowly-varying multiarmed bandit problems. Annual American Control Conference (ACC) , 2018.

Michael Woodroofe. A one-armed bandit problem with a concomitant variable. Journal of the American Statistical Association , 74(368):799-806, 1979.

Qingyun Wu, Naveen Iyer, and Hongning Wang. Learning contextual bandits in a non-stationary environment. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval , pages 495-504, 2018.

Yuhong Yang, Dan Zhu, et al. Randomized allocation with nonparametric estimation for a multiarmed bandit problem with covariates. The Annals of Statistics , 30(1):100-121, 2002.

Huozhi Zhou, Jinglin Chen, Lav R. Varshney, and Ashish Jagmohan. Nonstationary reinforcement learning with linear function approximation. Transactions on Machine Learning Research , 2022. ISSN 2835-8856.

A Details for Specializing Previous Contextual Bandit Results to Lipschitz Contextual Bandits

A.1 Finite Policy Class Contextual Bandits

In the finite policy class setting 2 , one is given access to a known finite class Π of policies π : X → [ K ] , and in the non-stationary variant, seeks to minimize regret to the time-varying benchmark of best policies π ∗ t . = argmax π ∈ Π E ( X,Y ) ∈D t [ Y ( π ( X ))] . In other words, the 'dynamic regret' in this setting is defined by (for chosen policies { ˆ π t } t )

We can in fact recover the Lipschitz contextual bandit setting and relate the above to our notion of dynamic regret (Definition 2). To do so, we let Π be the class of policies which uses a level r ∈ R and discretizes decision-making across individual bins B ∈ T r . Then, we claim there is an oracle sequence of policies { π oracle t } t which attains the minimax regret rate of Theorem 1. So, it remains to bound the regret to the sequence { π oracle t } t in the sense above.

Parametrizing in Terms of Global Number L of Shifts. Suppose there are L +1 stationary phases of length T/ ( L +1) . Then, we first claim there is an oracle sequence of policies π oracle t which attains reget ( L +1) 1 2+ d · T 1+ d 2+ d .

First, recall from Subsection 2.3 the oracle choice of level r n for a stationary period of n rounds, or the level r n ∝ ( K/n ) 1 2+ d . Now, define { π oracle t } t as follows: at each round t , π oracle t uses the oracle level r . = r T/ ( L +1) ∝ ( K ( L +1) T ) 1 2+ d and plays in each bin B ∈ T r , the arm maximizing the average reward in that bin E [ f a t ( X t ) | X t ∈ B ] . As this is a biased version of the actual bandit problem { f a t ( X t ) } a ∈ [ K ] at context X t , it will follow that π oracle t incurs regret of order the bias of estimation in B which is r .

Concretely, suppose X t falls in bin B at level r , and let π oracle t ( B ) be the arm selected at round t by π oracle t in bin B . Then, as mean rewards are Lipschitz, each policy π oracle t suffers regret:

Thus, the sequence of policies { π oracle t } t achieves dynamic regret (in the sense of Definition 2)

Thus, it suffices to minimize dynamic regret in the sense of (8) to this oracle policy π oracle t . The state-of-the-art adaptive guarantee in this setting is that of the ADA-ILTCB algorithm of Chen et al. [2019], which achieves a dynamic regret of √ KLT log( | Π | ) . Thus, it remains to compute | Π | .

We first observe that we need only consider levels in R of size at least ( K/T ) 1 2+ d , which is the oracle choice of level for one stationary phase of length T . Thus, the size of the policy class Π is

Plugging this into √ KLT log( | Π | ) gives a regret rate of K 1 2+ d · L 1 / 2 T 1+ d 2+ d , which has a worse dependence on the global number of shifts L than the minimax optimal rate of L 1 2+ d · T 1+ d 2+ d (see Theorem 1).

· Parametrizing in Terms of Total-Variation V T . Fix any positive real number V ∈ [ T -3+ d 2+ d , T ] . Then, the lower bound construction of Theorem 1 reveals that there exists an environment with

Then, the earlier defined oracle sequence of policies { π oracle t } t attains the optimal dynamic regret rate in terms of V T (see Theorem 1) since

Meanwhile, the state-of-the-art adaptive regret guarantee in this parametrization is Theorem 2 of Chen et al. [2019], which shows ADA-ILTCB's regret bound is:

We claim this rate is no better than our rate in Corollary 5, in all parameters V, K, T . For K ≥ T , both rates imply linear regret. Assume K < T . Then, note by elementary calculations that for all d ∈ N ∪ { 0 } :

Then, it follows that rate of Corollary 5 is smaller using the fact that K < T :

A.2 Realizable Contextual Bandits

Lipschitz contextual bandits is also a special case of contextual bandits with realizability . In this broader setting, the learner is given a function class Φ which contains the true regression function ϕ ∗ t : X × [ K ] → [0 , 1] describing mean rewards of context-arm pairs at round t . The goal is to compete with the time-varying benchmark of policies π ϕ ∗ t ( x ) . = argmax a ∈ [ K ] ϕ ∗ t ( x, a ) , using calls to a regression oracle over Φ .

While the natural choice for Φ is the infinite class of all Lipschitz functions from X × [ K ] → [0 , 1] , the state-of-the-art non-stationary algorithm only provides guarantees for finite Φ [Wei and Luo, 2021, Appendix I.7].

However, it is still possible to recover the Lipschitz contextual bandit setting, by defining Φ similarly to how we defined the finite class of policies Π above. Let Φ be the class of all piecewise constant functions which depends on a level r ∈ R , and are constant on bins B ∈ T r at level r , taking values which are multiples of T -1 2+ d (there are O ( T ) many such values in [0 , 1] ). Here, Φ is essentially the class of different discretization-based regression estimates for the true mean rewards, as appears in prior works [Rigollet and Zeevi, 2010, Perchet and Rigollet, 2013].

For this specification of Φ , the realizability assumption is false. Rather, this is a mildly misspecified regression class which is allowed by the stationary guarantees of FALCON [Simchi-Levi and Xu, 2021, Section 3.2]. In particular, by smoothness, at each round t ∈ [ T ] there is a function ϕ ∗ t ∈ Φ such that

Specifically, we can let ϕ ∗ t ( x, a ) ∝ E X [ f a t ( X ) | X ∈ B ] be the smoothed version of f a t ( x ) in the bin B at level T -1 2+ d containing x . Then, the above misspecification introduces an additive term in the regret bound of FALCON of order T 1+ d 2+ d which is of the right order in our setting.

In this setting, the current state-of-the-art MASTER black-box algorithm using FALCON Simchi-Levi and Xu [2021] as a base algorithm can obtain dynamic regret upper bounded by [see Wei and Luo, 2021, Theorem 2]:

As Φ is essentially the same size as the policy class Π defined in the previous section, the above regret bound specializes to similar rates as those of ADA-ILTCB derived above, and are ultimately suboptimal in light of Theorem 1.

B Useful Lemmas

Throughout the appendix, c 1 , c 2 , . . . will denote universal positive constants not depending on T, K or any of the significant shifts { τ i ( X T ) } i .

B.1 Concentration of Aggregate Gap over an Interval within a Bin

We'll first establish some concentration bounds for the local gap estimators ˆ δ B s ( a ′ , a ) defined in (4). For this purpose, we recall Freedman's inequality.

Lemma 6 (Theorem 1 of Beygelzimer et al. [2011]) . Let X 1 , . . . , X n ∈ R be a martingale difference sequence with respect to some filtration F 0 , F 1 , . . . . Assume for all t that X t ≤ R a.s.. Then for any δ ∈ (0 , 1) , with probability at least 1 -δ , we have:

Recall from Section 4 that for round t , the local gap estimate ˆ δ B t ( a ′ , a ) in bin B at round t between arms a ′ , a is:

We next apply Lemma 6 to our aggregate estimator from Section 4.

Proposition 7. With probability at least 1 -1 /T 2 w.r.t. the randomness of Y T , { π t } t | X T , we have for all bins B ∈ T and rounds s 1 < s 2 and all arms a ∈ [ K ] that for large enough c 1 > 0 :

where F . = {F t } T t =1 is the filtration with F t generated by { π s , Y π s s } t s =1 .

Proof. The martingale difference ˆ δ B s ( a ′ , a ) -E [ ˆ δ B s ( a ′ , a ) | F s -1 ] is clearly bounded above by 2 K for all bins B , rounds s , and all arms a, a ′ . We also have a cumulative variance bound:

Then, the result follows from (9), and taking union bounds over bins B (note there are at most T levels and at most T bins per level), arms a, a ′ , and rounds s 1 , s 2 .

Since the error probability of Proposition 7 is negligible with respect to regret, we assume going forward in the analysis that (10) holds for all arms a, a ′ ∈ [ K ] and rounds s 1 , s 2 . Specifically, let E 1 be the good event over which the bounds of Proposition 7 hold for all all arms and intervals [ s 1 , s 2 ] .

B.2 Concentration of Context Counts

We'll next establish concentration w.r.t. the distribution of contexts X T . This will ensure that all bins B ∈ T have sufficient observed data.

Notation. To ease notation throughout the analysis, we'll henceforth use µ ( · ) to refer to the context marginal distribution µ X ( · ) .

Lemma 8. Let { i t } T t =1 be a random sequence of arms whose distribution depends on X T . With probability at least 1 -1 /T 2 w.r.t. the randomness of X T , we have for all bins B ∈ T , all arms a ′ , a ∈ [ K ] , and rounds s 1 < s 2 , for some large enough c 2 > 0 the following inequalities hold:

Proof. The first inequality (11) follow from Lemma 6 since ∑ s 2 s = s 1 1 { X s ∈ B } -µ ( B ) is a martingale, which has predictable quadratic variation is at most ( s 2 -s 1 +1) · µ ( B ) .

The other two inequalities are trickier since the corresponding sums are not necessarily martingales. Indeed, note δ s ( a ) depends on X s while δ s ( i s , a ) may not even be adapted to the canonical filtration generated by X T (i.e., i s may depend on X t for t > s ). Nevertheless, we observe that for any random variable W s = W s ( X T ) ∈ [ -1 , 1] :

The upper and lower bounds above are both martingale differences with respect to the canonical filtration of X T and thus, summing the above over s we have via Lemma 6:

Then, taking union bounds over rounds s 1 , s 2 , bins B ∈ T , and arms a ∈ [ K ] gives the result.

Notation 2 (good event) . Recall from earlier that E 1 is the good event over which the bounds of Proposition 7 hold for all rounds s 1 , s 2 ∈ [ T ] and arms a ′ , a ∈ [ K ] . Thus, on E 1 , our estimated gaps in each bin will concentrate around their conditional means.

Let E 2 be the good event on which bounds of Lemma 8 holds for all bins B , arms a ∈ [ K ] , rounds s 1 , s 2 ∈ [ T ] . Thus, on E 2 , our covariate counts n B ([ s 1 , s 2 ]) will concentrate and we will be able to relate the empirical quantities ∑ s 2 s = s 1 δ s ( a ) · 1 { X s ∈ B } with their expectations.

Next, we establish a lemma which allows us to relate significant regret ( ⋆ ) and thus our eviction criterion (5) between different bins and levels.

Lemma 9 (Relating Aggregate Gaps Between Levels) . On event E 2 , if for rounds s 1 < s 2 , bin B ′ at level r s 2 -s 1 and arm a , for some c 3 > 0 :

then for any bin B ⊆ B ′ and some c 4 > 0 :

The same applies for δ s ( a ) replaced with δ s ( a ′ , a ) for any fixed arm a ′ ∈ [ K ] .

Proof. We have using (13) and the strong density assumption (Assumption 2):

Again using (13)

Next, applying (11) to n B ′ ([ s 1 , s 2 ]) and using the strong density assumption (Assumption 2) to bound the mass µ ( B ′ ) above by C d · r ( B ′ ) d , the above R.H.S. is further upper bounded by

Finally, plugging (15) into (14) and using the fact that ( r ( B ′ ) / 2) d ≥ ( K/ ( s 2 -s 1 )) d 2+ d , we have that (14) is of the desired order. The proof of the same inequalities with δ s ( a ′ , a ) is analogous.

The following lemma relating the bias and variance terms in the notion of significant regret ( ⋆ ) will serve useful many places in the analysis. They all follow from concentration and similar calculations via the strong density assumption (Assumption 2) as done previously.

Lemma 10 (Relating Bias and Variance Error Terms via Strong Density Assumption) . Let r . = r s 2 -s 1 for some s 2 > s 1 . Then, on event E 2 , for any bin B ∈ T r :

B.3 Useful Facts about Levels r ∈ R and their Durations in Play

The following basic facts about the level selection procedure on Line 2 of Algorithm 2 will be useful as we will decompose the analysis into the blocks, or different periods of rounds, where different levels are used.

Definition 8. Namely, for r ∈ R , let s ℓ ( r ) and e ℓ ( r ) denote the first and last rounds when level r is used by the master Base-Alg in episode [ t ℓ , t ℓ +1 ) , i.e. rounds t ∈ [ t ℓ , t ℓ +1 ) such that r t -t ℓ = r . Then, we call [ s ℓ ( r ) , e ℓ ( r )] a block .

The proofs of the following facts all follow from the definition of the level r n (see Notation 1) and basic calculations.

Fact 1 (Relating Level to Interval Length) . The level r s 2 -s 1 = 2 -m satisfies for s 2 -s 1 ≥ K :

and hence

Fact 2 (First Block) . The first block [ s ℓ (1) , e ℓ (1)] consists of rounds [ t ℓ , t ℓ + K ] .

Fact 3 (Start and End Times of a Block) . For r < 1 , the start time or first round s ℓ ( r ) of the block corresponding to level r in episode [ t ℓ , t ℓ +1 ) is s ℓ ( r ) = t ℓ + ⌈ K · (2 r ) -(2+ d ) ⌉ and the anticipated end time , or last round of the block if no new episode is triggered in said block, is e ℓ ( r ) = t ℓ + ⌈ K · r -(2+ d ) ⌉ -1 .

Fact 4 (Length of a Block) . Each block [ s ℓ ( r ) , e ℓ ( r )] is at least K rounds long. For the first block [ s ℓ (1) , e ℓ (1)] , this is already clear. Otherwise, suppose r < 1 in which case:

In particular, since 2 2+ d (1 -2 -(2+ d ) ) ≥ 2 for all d ≥ 0 , we have the above is at least K .

We also have the above implies

Rearranging, this becomes for some constant c 9 > 0 depending only on d :

Note we can make c 9 large enough so that the above also holds for level r = 1 .

The above along with the definition of r t -t ℓ (see Notation 1) implies that the block length e ℓ ( r ) -s ℓ ( r ) and the episode length e ℓ ( r ) -t ℓ ( r ) up to the end of block [ s ℓ ( r ) , e ℓ ( r )] can be conflated up to constants

C Proof of Oracle Regret Bound (Proposition 2)

Recall that E 2 is the good event on which our covariate counts concentrate by Lemma 8. It suffices to show our desired regret bound for any fixed context sequence X T on this event.

Fix a phase [ τ i , τ i +1 ) and let r . = r τ i +1 -τ i . Fix a bin B ∈ T r and let τ a i be the last round t ∈ [ τ i , τ i +1 ) such that X t ∈ B and arm a is included in G t . If a is never excluded from G t for all such t , let τ a i . = τ i +1 -1 . WLOG suppose τ 1 i ≤ τ 2 i ≤ · · · ≤ τ i K . Then, letting B ′ be the bin at level r τ a i -τ i containing covariate X τ a i , we have by ( ⋆ ) that:

From Lemma 9, we conclude ∑ τ a i t = τ i δ t ( a )1 { X t ∈ B } |G t | is at most

where we use the fact that |G t | ≥ K +1 -a for t ≤ τ a i such that X t ∈ B . Summing over arms a ∈ [ K ] with ∑ a ∈ [ K ] 1 K +1 -a ≤ log( K ) , we obtain from summing the above over K :

Next, we claim that each significant phase [ τ i , τ i +1 ) is at least K rounds long or K ≤ τ i +1 -τ i . This follows from the definition of significant regret ( ⋆ ) since for [ s 1 , s 2 ] ⊆ [ τ i , τ i +1 ) :

Then K ≤ τ i +1 -τ i implies (via Fact 1 about the level r τ i +1 -τ i )

Additionally, we have by Lemma 10:

Then, plugging the above into (16) and summing over bins B at level r , we have the regret in episode [ τ i , τ i +1 ) is with probability at least 1 -1 /T 2 w.r.t. the distribution of X T :

where we use the strong density assumption to bound ∑ B ∈T r r d ≤ ∑ B ∈T r c -1 d · µ ( B ) ≤ c -1 d in the last inequality. Summing the regret over all experienced significant phases [ τ i , τ i +1 ) gives the desired result.

D Proof of CMETA Regret Upper Bound (Theorem 3)

Recall from Line 3 of Algorithm 1 that t ℓ is the first round of the ℓ -th episode. WLOG, there are T total episodes and, by convention, we let t ℓ . = T +1 if only ℓ -1 episodes occurred by round T . We first quickly handle the simple case of T < K . In this case, the regret bound of Theorem 3 is vacuous since by the sub-additivity of x ↦→ x 1+ d 2+ d :

Thus, it remains to show Theorem 3 for T ≥ K .

We first transform the expected regret into a more suitable form, which will allow us to analyze regret in a similar fashion to the proof of the oracle regret bound (Appendix C).

D.1 Decomposing the Regret

We first transform the regret into a more convenient form. Let F . = {F t } T t =1 be the filtration with F t generated by { π s , Y π s s } t s =1 conditional on a fixed X T . Then,

Now, it suffices to bound the above R.H.S. on the good event E 1 ∩E 2 where the bounds of Lemmas 8 and 9 hold. Going forward in the rest of the analysis, we will assume said bounds hold wherever convenient.

Next, as alluded to in defining the oracle procedure (Definition 7), until the end of a significant phase [ τ i , τ i +1 ) , there is a safe arm in each bin B at level r τ i +1 -τ i which is experienced.

Definition 9 (local last safe arm in each phase a ♯ t ) . For a round t ∈ [ τ i , τ i +1 ) , let B be the bin at level r τ i +1 -τ i which contains X t and let t i ( B ) be the last round in [ τ i , τ i +1 ) such that X t i ( B ) ∈ B . Then, by Definition 6, there is a (local) last safe arm a ♯ t which does not yet incur significant regret in bin B in the following sense: for all [ s 1 , s 2 ] ⊆ [ τ i , t i ( B )] letting r = r s 2 -s 1 and B ′ ∈ T r such that B ′ ⊇ B we have:

Remark 6. The local last safe arms { a ♯ t } t only depend on the distribution of X T and not on the realized rewards Y T . In particular, the sequence { a ♯ t } t is fixed conditional on X T .

We first decompose the regret at round t as (a) the regret of the local last safe arm a ♯ t and (b) the regret of arm a to a ♯ t . In other words, it suffices to bound:

Note that the expectation on the first sum disappears since a ♯ t is only a function of X T and the mean reward functions { f a t ( · ) } t,a .

D.2 Bounding the Regret of the Local Last Safe Arm

Bounding ∑ T t =1 δ t ( a ♯ t ) · 1 {E 1 ∩ E 2 } will be similar to the proof of Proposition 2. We show that the oracle procedure could have essentially just played arm a ♯ t every round.

Fix a phase [ τ i , τ i +1 ) and let r = r τ i +1 -τ i . Fix a bin B ∈ T r and let a i ( B ) be the local last safe arm a ♯ t of the last round t ∈ [ τ i , τ i +1 ) such that X t ∈ B . Then, a ♯ t = a i ( B ) for every round t ∈ [ τ i , τ i +1 ) such that X t ∈ B . Then, we have by Definition 6 that for bin B ′ ⊇ B at level r t -τ i :

Then, by Lemma 9, we have:

Then, summing the above over bins in the same fashion as the proof of Proposition 2 gives:

Finally, summing over phases [ τ i , τ i +1 ) we have ∑ T t =1 δ t ( a ♯ t ) is of the right order.

D.3 Relating Episodes to Significant Phases

We next show that w.h.p. a restart occurs (i.e., a new episode begins) only if a significant shift has occurred sometime within the episode. Recall from Definition 6 that τ 1 , τ 2 , . . . , τ ˜ L are the times of the significant shifts and that t 1 , . . . , t T are the episode start times.

Lemma 11 ( Restart Implies Significant Shift). On event E 1 , for each episode [ t ℓ , t ℓ +1 ) with t ℓ +1 ≤ T (i.e., an episode which concludes with a restart), there exists a significant shift τ i ∈ [ t ℓ , t ℓ +1 ] .

Proof. Fix an episode [ t ℓ , t ℓ +1 ) . Then, by Line 11 of Algorithm 1, there is a bin B such that every arm a ∈ [ K ] was evicted from B at some round in the episode, i.e. (5) is true for each arm a on some interval [ s 1 , s 2 ] ⊆ [ t ℓ , t ℓ +1 ) . It suffices to show that this implies a significnat shift has occurred between rounds t ℓ and t ℓ +1 .

̸

Suppose (5) first triggers the eviction of arm a at time t in B ′ ⊇ B over interval [ s 1 , s 2 ] where r ( B ′ ) = r s 2 -s 1 . By concentration (10) and our eviction criteria (5), we have that there is an arm a ′ = a such that (using the notation of Proposition 7) for large enough C 0 > 0 and some c 14 > 0 :

Next, if arm a is evicted from A ( B ′ ) at round t , then we have by the definition of ˆ δ B ′ s ( a ′ , a ) (4):

In any case, the above L.H.S. conditional expectation is bounded above by δ s ( a ) · 1 { X s ∈ B ′ } . Thus, (18) implies arm a incurs significant regret ( ⋆ ) in B ′ on [ s 1 , s 2 ] :

Then, since every arm a is evicted in bin B by round t , a significant shift must have occurred in episode [ t ℓ , t ℓ +1 ] .

D.4 Regret of CMETA to the Last Safe Arm

It remains to bound E [ ∑ T t =1 ∑ a ∈A t δ t ( a ♯ t , a ) / |A t | | X t ] . We further decompose this sum over t into episodes and then the blocks (see Definition 8) where a particular choice of level is used within the episode. The following notation will be useful.

̸

Definition 10. Let PHASES ( ℓ, r ) . = { i ∈ [ ˜ L ] : [ τ i , τ i +1 ) ∩ [ s ℓ ( r ) , e ℓ ( r )] = ∅} be the phases which intersect block [ s ℓ ( r ) , e ℓ ( r )) , let T ( i, r, ℓ ) . = | [ τ i , τ i +1 ) ∩ [ s ℓ ( r ) , e ℓ ( r )] | be the effective length of the phase as observed in block [ s ℓ ( r ) , e ℓ ( r )] .

̸

Similarly, define PHASES ( ℓ ) . = { i ∈ [ ˜ L ] : [ τ i , τ i +1 ) ∩ [ t ℓ , t ℓ +1 ) = ∅} as the phases which intersect episode [ t ℓ , t ℓ +1 ) .

It will in fact suffice to show w.h.p. w.r.t. the distribution of X T , for each episode [ t ℓ , t ℓ +1 ) , each block [ s ℓ ( r ) , e ℓ ( r )] in [ t ℓ , t ℓ +1 ) , and each bin B ∈ T r :

D.5 Summing the Per-(Bin, Block, Episode) Regret over Bins, Blocks, and Episodes.

Admitting (19), we show that the total dynamic regret over T rounds is of the desired order.

Recall from earlier that there are WLOG T total episodes with the convention that t ℓ . = T +1 if only ℓ episodes occur by round T . Then, summing our per-bin regret bound (19) over all the bins B ∈ T r

at level r gives (using strong density to bound ∑ B ∈ r r d ≤ C d c d ):

Next, summing over the different levels r (of which there are at most log( T ) used in any episode), we obtain by Jensen's inequality on the concave function z ↦→ z 1+ d 2+ d :

Now, we have

We also have (via Fact 1 about level r t ℓ +1 -t ℓ which is the smallest level used in episode [ t ℓ , t ℓ +1 ) ).

Thus, combining the above inequalities with (20), we obtain overall bound:

Recall now that E 1 is the good event over which the concentration bounds of Proposition 7 hold. Then, using the fact that, on event E 1 , each phase [ τ i , τ i +1 ) intersects at most two episodes (Lemma 11), summing the above R.H.S over episodes ℓ ∈ [ T ] gives us (since at most log( T ) blocks per episode) order

It then remains to show the per-(bin, block, episode) regret bound (19).

D.6 Bounding the Per-(Bin, Block, Episode) Regret to the Last Safe Arm

To show (19), we first fix a block [ s ℓ ( r ) , e ℓ ( r )] and a bin B ∈ T r . We then further decompose δ t ( a ♯ t , a ) in two parts:

(a) The regret of a to the local last master arm , denoted by a r ( B ) , to be evicted from A master ( B ) in block [ s ℓ ( r ) , e ℓ ( r )] (ties are broken arbitrarily).

(b) The regret of the local last master arm a r ( B ) to the last safe arm a ♯ t .

In other words, the L.H.S. of (19) is decomposed as:

We will show both (a) and (b) are of order (19).

Bounding the Regret of Other Arms to the Local Last Master Arm a r ( B ) . We start by partitioning the rounds t such that X t ∈ B and a ∈ A t in (a) according to before or after they are evicted from A master ( B ) . Suppose arm a is evicted from A master ( B ) at round t a r ∈ [ s ℓ ( r ) , e ℓ ( r )] (formally, we let t a r . = e ℓ ( r ) if a is not evicted in block [ s ℓ ( r ) , e ℓ ( r )] ). Then, it suffices to bound:

Suppose WLOG that t 1 r ≤ t 2 r ≤ · · · ≤ t K r . Then, for each round t < t a r all arms a ′ ≥ a are retained in A master ( B ) and thus retained in the candidate arm set A t for all rounds t where X t ∈ B . Importantly, at each round t a level of at least r is used since a child Base-Alg can only use a higher level than the master Base-Alg . Thus, |A t | ≥ K +1 -a for all t ≤ t a r .

Next, we bound the first double sum in (21), i.e. the regret of playing a to a r ( B ) from s ℓ ( r ) to t a r -1 . Applying our concentration bounds (Proposition 7), since arm a is not evicted from A ( B ) till round t a r , on event E 1 we have for some c 19 > 0 and any other arm a ′ ∈ A ( B ) through round t a r -1 (i.e., a ′ ∈ A t for all t ∈ [ t ℓ , t a r ) such that X t ∈ B since we always use level at least r at such a round t ): for bin B ′ ⊇ B at level r t a r -1 -s ℓ ( r ) : on event E 1 (note that we necessarily always have A ( B ′ ) ⊇ A ( B ) for B ′ ⊇ B by Line 14 of Algorithm 2):

Next, since a, a ′ ∈ A t for each t ∈ [ s ℓ ( r ) , t a r -1) such that X t ∈ B , we have:

Thus, we conclude by (5):

Thus, by Lemma 9, and since B ′ ⊇ B , we conclude for any such a ′ on event E 1 : ∑ t a r -1 t = s ℓ ( r ) δ t ( a ′ ,a ) |A t | · 1 { X t ∈ B } is at most

where we use the fact that |A t | ≥ K + 1 -a for all t ∈ [ s ℓ ( r ) , t a r ) . Since this last bound holds uniformly for all a ′ ∈ A ( B ) through round t a r -1 , it must hold for a ′ = a r ( B ) , the local last master arm.

Then, summing over all arms a , we have on event E 1 :

Next note that by Lemma 10:

Additionally, since K ≤ e ℓ ( r ) -s ℓ ( r ) (Fact 4), we have:

Thus, combining the above two displays with (23) gives us

We next show the second double sum in (21) has an upper bound similar to the above. For this, we first observe that if arm a is played in bin B after round t a r , then it must be due to an active replay. The difficulty here is that replays may interrupt each other and so care must be taken in managing the contribution of ∑ t δ t ( a r ( B ) , a ) (which may be negative) by different overlapping replays.

Our strategy, similar to that of Section B.1 in Suk and Kpotufe [2022], is to partition the rounds when a is played by a replay after round t a r according to which replay is active and not accounted for by another replay. This involves carefully identifying a subclass of replays whose durations while playing a in B span all the rounds where a is played in B after t a r . Then, we cover the times when a is played by a collection of intervals corresponding to the schedules of this subclass of replays, on each of which we can employ the eviction criterion (5) and concentration bound as done earlier.

For this purpose, we first define the following terminology (which is all w.r.t. a fixed arm a , along with fixed block [ e ℓ ( r ) , s ℓ ( r )] and bin B ∈ T r ):

Definition 11.

(i) For each scheduled and activated Base-Alg ( s, m ) , let the round M ( s, m ) be the minimum of two quantities: (a) the last round in [ s, s + m ] when arm a is retained in A ( B ) by Base-Alg ( s, m ) and all of its children, and (b) the last round that Base-Alg ( s, m ) is active and not permanently interrupted by another replay. Call the interval [ s, M ( s, m )] the active interval of Base-Alg ( s, m ) .

(ii) Call a replay Base-Alg ( s, m ) proper if there is no other scheduled replay Base-Alg ( s ′ , m ′ ) such that [ s, s + m ] ⊂ ( s ′ , s ′ + m ′ ) where Base-Alg ( s ′ , m ′ ) will become active again after round s + m . In other words, a proper replay is not scheduled inside the scheduled range of rounds of another replay. Let PROPER ( s ℓ ( r ) , e ℓ ( r )) be the set of proper replays scheduled to start in the block [ s ℓ ( r ) , e ℓ ( r )] .

(iii) Call a scheduled replay Base-Alg ( s, m ) a sub-replay if it is non-proper and if each of its ancestor replays (i.e., previously scheduled replays whose durations have not concluded) Base-Alg ( s ′ , m ′ ) satisfies M ( s ′ , m ′ ) < s . In other words, a sub-replay either permanently interrupts its parent or does not, but is scheduled after its parent (and all its ancestors) stops playing arm a in B . Let SUBPROPER ( s ℓ ( r ) , s ℓ ( r )) be the set of all sub-replays scheduled before round t ℓ +1 .

Equipped with this language, we now show some basic claims which essentially reduce analyzing the complicated hierarchy of replays to analyzing the active intervals of replays in PROPER ( s ℓ ( r ) , e ℓ ( r )) ∪ SUBPROPER ( s ℓ ( r ) , s ℓ ( r )) .

Proposition 12. The active intervals

Proof. Clearly, the classes of replays PROPER ( t ℓ , t ℓ +1 ) and SUBPROPER ( s ℓ ( r ) , s ℓ ( r )) are disjoint. Next, we show the respective active intervals [ s, M ( s, m )] and [ s ′ , M ( s ′ , m ′ )] of any two Base-Alg ( s, m ) and Base-Alg ( s ′ , m ′ ) ∈ PROPER ( s ℓ ( r ) , e ℓ ( r )) ∪ SUBPROPER ( s ℓ ( r ) , s ℓ ( r )) are disjoint.

proper1

replays

arm a evicted

sub-replays reintroduce arm a to At

Proper replay vs. sub-replay: a sub-replay can only be scheduled after the round M ( s, m ) of the most recent proper replay Base-Alg ( s, m ) (which is necessarily an ancestor). Thus, the active intervals of proper replays and sub-replays are disjoint.

Two distinct proper replays: two such replays can only intersect by one permanently interrupting the other, and since M ( s, m ) always occurs before the permanent interruption of Base-Alg ( s, m ) , we have the active intervals of two such replays are disjoint.

Two distinct sub-replays: consider two non-proper replays Base-Alg ( s, m ) , Base-Alg ( s ′ , m ′ ) ∈ SUBPROPER ( s ℓ ( r ) , s ℓ ( r )) with s ′ > s . Suppose their active intervals intersect and that Base-Alg ( s, m ) is an ancestor of Base-Alg ( s ′ , m ′ ) . Then, if Base-Alg ( s ′ , m ′ ) is a sub-replay, we must have s ′ > M ( s, m ) , which means that [ s ′ , M ( s ′ , m ′ )] and [ s, M ( s, m )] are disjoint.

Next, we claim that the active intervals [ s, M ( s, m )] for Base-Alg ( s, m ) ∈ PROPER ( t ℓ , t ℓ +1 ) ∪ SUBPROPER ( s ℓ ( r ) , s ℓ ( r )) contain all the rounds where a is played in B after being evicted from A master ( B ) . To show this, we first observe that for each round t when a replay is active, there is a unique proper replay associated to t , namely the proper replay scheduled most recently. Next, note that any round t > t a r where X t ∈ B and where arm a ∈ A t must belong to the active interval [ s, M ( s, m )] of this unique proper replay Base-Alg ( s, m ) associated to round t , or else satisfies t > M ( s, m ) in which case a unique sub-replay Base-Alg ( s ′ , m ′ ) ∈ SUBPROPER ( s ℓ ( r ) , s ℓ ( r )) is active at round t and not yet permanently interrupted by round t . Thus, it must be the case that t ∈ [ s ′ , M ( s ′ , m ′ )] .

Overloading notation, we'll let A t ( B ) be the value of A ( B ) for the Base-Alg active at round t . Next, note that every round t ∈ [ s, M ( s, m )] for a proper or subproper Base-Alg ( s, m ) is clearly a round where a ∈ A t ( B ) and no such round is accounted for twice by Proposition 12. Thus,

Then, we can rewrite the second double sum in (21) as:

Recall in the above that the Bernoulli R.V. Z m,s (see Line 6 of Algorithm 1) decides whether Base-Alg ( s, m ) is scheduled.

Further bounding the sum over t above by its positive part, we can expand the middle sum above over Base-Alg ( s, m ) ∈ PROPER ( t ℓ , t ℓ +1 ) ∪ SUBPROPER ( s ℓ ( r ) , s ℓ ( r )) to instead be over all Base-Alg ( s, m ) , or obtain:

where the sum is over all replays Base-Alg ( s, m ) , i.e. s ∈ { t ℓ + 1 , . . . , t ℓ +1 -1 } and m ∈ { 2 , 4 , . . . , 2 ⌈ log( T ) ⌉ } . It then remains to bound the contributed relative regret of each Base-Alg ( s, m ) in the interval [ s ∨ t a r , M ( s, m )] , which will follow similarly to the previous steps in bounding the first double sum of (21).

We first have (now overloading the notation M ( s, m ) as M ( s, m, a ) for clarity), i.e. combining our concentration bound (10) with the eviction criterion (5) and applying Lemma 9:

Thus, it remains to bound

Swapping the outer two sums and, similar to before, recognizing that ∑ K a =1 1 min t ∈ [ s,M ( s,m,a )] |A t | ≤ log( K ) by summing over arms in the order they are evicted by Base-Alg ( s, m ) , we have that it remains to bound:

where ˜ m . = m ∧ ( e ℓ ( r ) -s ℓ ( r )) (note we may freely restrict all active intervals to the current block [ s ℓ ( r ) , e ℓ ( r )] ). Let

Then, in light of the previous calculations, R ( m,B ) is an upper bound on the within-bin B regret contributed by a replay of total duration m (note we can always coarsely upper bound this regret by n B ([ s, s + m ]) .

Then, plugging R ( m,B ) into (24) gives via tower law:

Next, we observe that Z m,s and 1 { s ≤ e ℓ ( r ) } are independent conditional on s ℓ ( r ) since 1 { s ≤ e ℓ ( r ) } only depends on the scheduling and observations of base algorithms scheduled before round s . Additionally, conditional on s ℓ ( r ) , the episode start time t ℓ is also fixed since the two are deterministically related (see Fact 3). Then, we have that:

Thus,

Plugging this into (25) and unconditioning, we obtain:

We first evaluate the inner sum over n . Note that

Next, we plug in the above displays into (26). In particular, multiplying the above displays by ( s -t ℓ ) -1+ d 2+ d and taking a further sum over s ∈ [ s ℓ ( r ) , e ℓ ( r )] gives an upper bound of:

First, we note the first term inside the parentheses above inside dominates the second term for all values of K,e ℓ ( r ) , s ℓ ( r ) , T .

Next, note from Fact 4 that e ℓ ( r ) -t ℓ ≤ c 10 ( e ℓ ( r ) -s ℓ ( r )) and so the above is at most:

We next recall from Fact 4 that each block [ s ℓ ( r ) , e ℓ ( r )] is at least K rounds long. Thus,

Thus, the second term of (27) is at most the order of the first term.

Showing (a) is order (19) then follows from writing e ℓ ( r ) -s ℓ ( r ) as the sum of effective phase lengths T ( i, r, ℓ ) (see Definition 10) of the phases [ τ i , τ i +1 ) intersecting block [ s ℓ ( r ) , e ℓ ( r )] , and using the sub-additivity of x ↦→ x 1+ d 2+ d .

Bounding the Regret of the Last Master Arm a r ( B ) to the Last Safe Arm a ♯ t . Before we proceed, we first convert ∑ e ℓ ( r ) t = s ℓ ( r ) δ t ( a ♯ t , a r ( B )) · 1 { X t ∈ B } into a more convenient form in terms of the masses µ ( B ) . By concentration (12) of Proposition 7, we have

We first show the two concentration error terms on the R.H.S. above are negligible with respect to the desired bound (19). The log( T ) term is clearly of the right order, whereas the other term is handled by Lemma 10, by which

By similar arguments to before, where we write e ℓ ( r ) -s ℓ ( r ) = ∑ i ∈ PHASES ( ℓ,r ) T ( i, r, ℓ ) and use the sub-additivity of the function x ↦→ x 1+ d 2+ d , the above is of the right order w.r.t. (19).

Thus, going forward, by the strong density assumption (Assumption 2) and in light of (19), it suffices to show for any fixed arm a ∈ [ K ] (which we will take to be a r ( B ) in the end):

where E ( B,a ) is the last round in block [ s ℓ ( r ) , e ℓ ( r )] for which a ∈ A master ( B ) .

This aggregate gap is the most difficult quantity to bound since arm a ♯ t may have been evicted from A master ( B ) before round t and, thus, we rely on our replay scheduling (Line 6 of Algorithm 2) to bound the regret incurred while waiting to detect a large aggregate value of δ t ( a ♯ t , a ) .

In an abuse of notation, we'll conflate e ℓ ( r ) with the anticipated block end time based on s ℓ ( r ) ; that is, the end block time if no episode restart occurs within the block. Now, for each phase [ τ i , τ i +1 ) which intersects the block [ s ℓ ( r ) , e ℓ ( r )] , our strategy will be to map out in time the local bad segments or subintervals of [ τ i , τ i +1 ) where a fixed arm a incurs significant regret to arm a ♯ t in bin B , roughly in the sense of ( ⋆ ). The argument will conclude by arguing that a well-timed replay is scheduled w.h.p. to detect some local bad segment in B , before too many elapse.

In particular, conditional on just the block start time s ℓ ( r ) , we define the bad segments for a fixed arm a and then argue that if too many bad segments w.r.t. a elapse in the block's anticipated set of rounds [ s ℓ ( r ) , e ℓ ( r )] , then arm a will be evicted in bin B . Crucially, this will hold uniformly over all arms a and, in particular, for arm a . = a r ( B ) . This will then bound the regret of a r ( B ) in block [ s ℓ ( r ) , e ℓ ( r )] in the sense of (28).

Notation. Going forward, we will drop the dependence on the level r , block [ s ℓ ( r ) , e ℓ ( r )] , and episode [ t ℓ , t ℓ +1 ) in certain definitions as they are fixed momentarily. Recall from Appendix D.2 that a ♯ t is the local last safe arm of the last round t i ( B ) ∈ [ τ i , τ i +1 ) such that X t i ( B ) ∈ B where B is the bin at level r τ i +1 -τ i containing X t (see Definition 9).

We first introduce the notion of a bad segment of rounds which is a minimal period where large regret in the sense of ( ⋆ ) within bin B is detectable by a well-timed replay.

Definition 12. Fix an arm a and s ℓ ( r ) , and let [ τ i , τ i +1 ) be any phase intersecting [ s ℓ ( r ) , e ℓ ( r )] . Define rounds s i, 0 ( a ) , s i, 1 ( a ) , s i, 2 ( a ) . . . ∈ [ t ℓ ∨ τ i , τ i +1 ) recursively as follows: let s i, 0 ( a ) . = t ℓ ∨ τ i and define s i,j ( a ) as the smallest round in ( s i,j -1 ( a ) , τ i +1 ∧ e ℓ ( r )) such that arm a satisfies for some fixed c 21 > 0 :

Otherwise, we let the s i,j ( a ) . = τ i +1 -1 . We refer to the interval [ s i,j -1 ( a ) , s i,j ( a )) as a bad segment . We call [ s i,j -1 ( a ) , s i,j ( a )) a proper bad segment if (29) above holds.

It will in fact suffice to constrain our attention to proper bad segments, since non-proper bad segments [ s i,j -1 ( a ) , s i,j ( a )) (where s i,j ( a ) = τ i +1 -1 and (29) is reversed) will be negligible in the regret analysis since there is at most one non-proper bad segment per phase [ τ i , τ i +1 ) (i.e., the regret of each non-proper bad segment is at most the R.H.S. of (28)).

We first establish some elementary facts about proper bad segments which will later serve useful in analyzing the detectability of ( ⋆ ) along such segments of time.

Lemma 13. Let [ s i,j ( a ) , s i,j +1 ( a )) be a proper bad segment defined w.r.t. arm a . Let m ∈ N ∪ { 0 } be such that r s i,j +1 ( a ) -s i,j ( a ) = 2 -m . Then, for some c 25 = c 25 ( d ) > 0 depending on the dimension d :

Proof. First, we may assume s i,j +1 ( a ) -s i,j ( a ) ≥ 4 · K by choosing c 24 in (29) large enough (this will make m -1 ≥ 0 ).

First, observe by the definition of r s i,j +1 ( a ) -s i,j ( a ) (Notation 1) that

Now, let ˜ s . = s i,j +1 ( a ) -K 2 ( m -2)(2+ d ) -1 . Then, we have by (29) in the construction of the s i,j ( a ) 's (Definition 12) that:

Let m i,j ( a ) . = s i,j +1 ( a ) -s i,j ( a ) . Then, we have by (31) that:

Plugging this into our earlier bound the constants in our updated lower bound scale like:

But, this last term is positive for all d ∈ N ∪ { 0 } and only depends on d .

Lemma 14 (Aggregate Gap Dominates Concentration Error) . Fix a bin B at level r . Let [ s i,j ( a ) , s i,j +1 ( a )) be a proper bad segment and let B ′ ⊇ B be the bin at level r s i,j +1 ( a ) -˜ s where ˜ s . = s i,j +1 ( a ) -K 2 ( m -2)(2+1) -1 is as in Lemma 13. Then, for some c 26 > 0 :

Proof. We have via concentration ((12) of Lemma 8), Lemma 13, and the strong density assumption (Assumption 2):

Now, the first term on the final R.H.S. above dominates the other two terms for large enough c 25 and via strong density assumption (Assumption 2).

Thus, it suffices to show

Wefirst upper bound the 'variance' term, or the first term on the R.H.S. above. Let W . = s i,j +1 ( a ) -˜ s . By Lemma 10, we have

Now, we also have

Next, we note that by the definition of a proper bad segment ((29) in Definition 12) that

This implies ( s i,j +1 ( a ) -s i,j ( a )) 1 2+ d ≥ c 24 2 log( T ) . By similar reasoning, we have s i,j +1 ( a ) -s i,j ( a ) ≥ c 28 K . From this, we conclude for c 24 > 0 large enough:

Thus, (32) is shown.

Now, we define a well-timed or perfect replay which, if scheduled, will detect the badness of arm a (in the sense of (5)) in bin B over a proper bad segment [ s i,j ( a ) , s i,j +1 ( a )) . The simplest such perfect replay is one which is scheduled directly from rounds s i,j ( a ) to s i,j +1 ( a ) . We in fact show there is a spectrum of replays (of size the length of the segment Ω( s i,j +1 ( a ) -s i,j ( a )) ) each of which can detect arm a is bad in bin B , possibly by using a larger ancestor bin B ′ ⊇ B .

Definition 13 (Perfect Replay) . For a fixed proper bad segment [ s i,j ( a ) , s i,j +1 ( a )) , define a perfect replay as a Base-Alg ( t start , M ) with t start ∈ [ s i,j +1 ( a ) -K 2 ( m -2)(2+ d ) + 1 , s i,j +1 ( a ) -K 2 ( m -2)(2+ d ) -1 ] (where m ∈ N ∪ { 0 } is as in Lemma 13) and t start + M ≥ s i,j +1 ( a ) .

The following proposition analyzes the behavior of a perfect replay and shows, if scheduled, it will in fact evict arm a from A ( B ) within a proper bad segment [ s i,j ( a ) , s i,j +1 ( a )) .

Proposition 15 (Perfect Replay Evicts Bad Arm in Proper Bad Segment) . Suppose event E 1 ∩ E 2 holds (see Notation 2). Fix a bin B at level r . Let [ s i,j ( a ) , s i,j +1 ( a )) be a proper bad segment defined with respect to arm a . Let Base-Alg ( t start , M ) be a perfect replay as defined above which becomes active at t start (i.e., Z t start ,M = 1 ) for a fixed integer M ≥ s i,j +1 ( a ) -s i,j ( a ) . Then:

(i) Let B ′ be the bin at level r ˜ s -s i,j ( a ) where ˜ s . = s i,j +1 ( a ) -K 2 ( m -2)(2+ d ) -1 ( m ∈ N ∪{ 0 } is as in Lemma 13), as in Lemma 14. Then, there is a 'safe arm' a ♯ ( B ′ ) which will not be evicted from A ( B ′ ) by Base-Alg ( t start , M ) (or any of its children) before round s i,j +1 ( a )+1 .

(ii) If a ∈ A t for all rounds t ∈ [˜ s, s i,j +1 ( a )) where X t ∈ B , w then arm a will be excluded from A ( B ) by round s i,j +1 ( a ) .

Proof. For (i), we can define the 'safe arm' a ♯ ( B ′ ) in a similar fashion to how a ♯ t was defined. Let t i ( B ′ ) be the last round in [˜ s, s i,j +1 ( a )] such that X t i ( B ′ ) ∈ B ′ . Then, since s i,j +1 ( a ) < τ i +1 , we have that at round t i ( B ′ ) , there is a safe arm a ♯ ( B ′ ) which does not satisfy ( ⋆ ) for any bin B ′′ intersecting B ′ and interval of rounds I ⊆ [˜ s, s i,j +1 ( a )] . Once Base-Alg ( t start , M ) is scheduled, it (or any of its children) cannot evict arm a ♯ ( B ′ ) from B ′ as doing so would imply it has significant regret in some bin intersecting B ′ (following the same calculations as in Lemma 11).

We next turn to (ii). We first suppose that arms a is active in bin B ′ from rounds ˜ s to s i,j +1 ( a ) (we'll carefully argue later this is indeed the case). We first observe E [ ˆ δ B t ( a ♯ ( B ) , a ) | F t -1 ] = δ t ( a ♯ i ( B ) , a ) for any round t ∈ [˜ s, s i,j +1 ( a )] such that X t ∈ B ′ . We next observe that:

By Lemma 14, the first term on the R.H.S. is at least

Meanwhile, the second term on the R.H.S. of (33) is at most the same order by the definition of a ♯ ( B ) . Thus, choosing c 26 large enough gives us that

Then, combining the above with our eviction criterion (5) and concentration (10), we have that arm a will be evicted in the bin B ′ ⊇ B at level r s i,j +1 ( a ) -˜ s by round s i,j +1 ( a ) .

Finally, it remains to show that, within Base-Alg ( t start , M ) 's play, arm a will not be evicted in any child of B ′ before round s i,j +1 ( a ) . This will follow from the fact that any perfect replay must use a level in R of size at least r W . In particular, by Definition 13, the starting round t start is 'close enough' to the critical round s i,j +1 ( a ) -K 2 ( m -2)(2+ d ) -1 so that it will not use a different level than the perfect replay which starts exactly at this critical round.

Formally, we have that the smallest level a perfect replay can use is r ˜ W where ˜ W . = K · 2 ( m -2)(2+ d ) -1 .

Next, note that s i,j +1 ( a ) -t start ≤ K · 2 ( m -2)(2+ d ) -1 and so

Thus, r ˜ W ≥ 2 -( m -2) . On the other hand,

Thus, 2 -( m -2) = r s i,j +1 ( a ) -˜ s is also the level used to detect that arm a is bad in bin B ′ . Thus, we conclude that r ˜ W is no smaller than the level r s i,j +1 ( a ) -˜ s used to evict arm a in bin B ′ . This means arm a cannot be evicted in a child of B ′ before round s i,j +1 ( a ) , if a is not already evicted in B .

Next, we show for any arm a (in particular, a = a r ( B ) ), a perfect replay characterized by Definition 13 is scheduled with high probability if too many bad segments w.r.t. a elapse, thus bounding the regret of a to a ♯ i ( B ) over the phases [ τ i , τ i +1 ) intersecting block [ s ℓ ( r ) , e ℓ ( r )] .

D.7 Bounding the Regret of the Last Master Arm a r ( B ) to the Last Safe Arm a ♯ t

Next, we bound the the regret of a fixed arm a to a ♯ t over the bad segments w.r.t. a in B . Recall from earlier (28) that our remaining goal is to establish the following bound for every bin B ∈ T r at level r :

where E ( B,a ) is the last round in block [ s ℓ ( r ) , e ℓ ( r )] for which a ∈ A master ( B ) .

Note that the bad segments (Definition 12) are defined for a level r , block start time s ℓ ( r ) , and phase [ τ i , τ i +1 ) . In particular, they are defined independent of any choice of bin B at level r . We'll similarly define a bad round s ( a ) which will indicate when 'too many' bad segments w.r.t. a have elapsed, irrespective of a choice of bin B . Then, we'll argue that for any bin B at level r , E ( B,a ) ≤ s ( a ) .

It should be understood that in what follows, we condition on the block start time s ℓ ( r ) . First, fix an arm a and define the bad round s ( a ) > s ℓ ( r ) as the smallest round which satisfies, for some fixed c 30 > 0 :

where the above sum is over all pairs of indices ( i, j ) ∈ N × N such that [ s i,j ( a ) , s i,j +1 ( a )) is a proper bad segment with s i,j +1 ( a ) < s ( a ) . We will show that, for any bin B at level r , arm a is evicted from A ( B ) episode ℓ with high probability by the time the bad round s ( a ) occurs.

For each proper bad segment [ s i,j ( a ) , s i,j +1 ( a )) , let ˜ s i,j ( a ) . = s i,j +1 ( a ) -K 2 ( m -2)(2+ d ) -1 denote the 'critical point' of the bad segment as in Lemma 13 and also let m i,j . = 2 n where n ∈ N satisfies:

Next, recall that the Bernoulli Z M,t decides whether Base-Alg ( t, M ) activates at round t (see Line 6 of Algorithm 1). If for some t ∈ [ˆ s i,j ( a ) , ˜ s i,j ( a )] where ˆ s i,j ( a ) . = s i,j +1 ( a ) -K 2 ( m -2)(2+ d ) +1 , Z m i,j .t = 1 , i.e. a perfect replay is scheduled, then a will be evicted from A ( B ) by round s i,j +1 ( a ) (Proposition 15).

We will show this happens with high probability via concentration on the sum ∑ ( i,j ) ∑ t Z m i,j ,t where j, i, t run through all t ∈ [ˆ s i,j ( a ) , ˜ s i,j ( a )) and all proper bad segments [ s i,j ( a ) , s i,j +1 ( a )) with s i,j +1 ( a ) < s ( a ) . Note that these random variables, conditional on X T , depend only on the fixed arm a , the block start time s ℓ ( r ) , and the randomness of scheduling replays on Line 6. In particular, the Z m i,j ,t are independent conditional on t ℓ .

Then, a Chernoff bound over the randomization of CMETA on Line 6 of Algorithm 1 conditional on t ℓ yields

We claim the error probability on the R.H.S. above is at most 1 /T 3 . To this end, we compute:

where the last inequality follows from (34). The R.H.S. above is larger than 24 log( T ) for c 30 large enough, showing that the error probability is small. Taking a further union bound over the choice of arm a ∈ [ K ] gives us that ∑ ( i,j ) ∑ t Z m i,j ,t > 1 for all choices of arm a (define this as the good event E 3 ( s ℓ ( r )) ) with probability at least 1 -K/T 3 .

Recall on the event E 1 ∩E 2 the concentration bounds of Proposition 7 and Lemma 8 hold. Then, on E 1 ∩E 2 ∩E 3 ( s ℓ ( r )) , we must have for each bin B ∈ T r at level r , E ( B,a ) ≤ s ( a ) since otherwise a would have been evicted in A ( B ) by some perfect replay before the end of the block e ℓ ( r ) by virtue of ∑ ( i,j ) ∑ t Z m i,j ,t > 1 for arm a . Thus, by the definition of the bad round s ( a ) (34), we must have:

Thus, by (29) in Definition 12, over the proper bad segments [ s i,j ( a ) , s i,j +1 ( a )) which elapse before round e ℓ ( r ) ∧ E ( B,a ) in phase [ τ i , τ i +1 ) : the regret is at most

Over each non-proper bad segment [ s i,j ( a ) , s i,j -1 ( a )) and the last segment [ s i,j ( a ) , e ℓ ( r ) ∧ E ( B,a )] , the regret of playing arm a to a ♯ t is at most log( T ) · K 1 2+ d m 1+ d 2+ d i,j since there is at most one non-proper bad segment per phase [ τ i , τ i +1 ) (see (29) in Definition 12).

So, we conclude that on event E 1 ∩ E 2 ∩ E 3 ( s ℓ ( r )) : for any bin B ∈ T r

Let event G . = E 1 ∩E 2 .Then, taking expectation, we have by conditioning first on s ℓ ( r ) and then on event G ∩ E 3 ( s ℓ ( r )) :

We first handle the first double expectation on the R.H.S. above. We have this is at most

where in the last step we bound 1 {G ∩ E 3 ( s ℓ ( r )) } ≤ 1 {G} and apply tower law again. The above R.H.S. is of the right order w.r.t. (19). So, it remains to bound

We first observe that events G and E c 3 ( s ℓ ( r )) are independent conditional on X T and s ℓ ( r ) since the former event only depends on the distribution of Y s ℓ ( r ) , . . . , Y T while the latter event depends on the distribution of the Bernoulli's { Z M,t } t>s ℓ ( r ) ,M (which are independent). Then, writing 1 {G ∩E c 3 ( s ℓ ( r )) } =1 {G}· 1 {E c 3 ( s ℓ ( r )) } , the above becomes at most E [1 {G}· ( K/T 2 ) | X T ] which is also of the right order w.r.t. (19).

E Proof of Corollary 5

The proof of Corollary 5 will follow in a similar fashion to the proof of Corollary 2 in Suk and Kpotufe [2022], which relates the total-variation rates to significant shifts in the non-stationary MAB setting. A novel difficulty here is that our notion of significant shift τ i ( X T ) , ˜ L ( X T ) (Definition 6) depends on the full context sequence X T , and so it is not clear how the (random) significant phases [ τ i ( X T ) , τ i +1 ( X T )) relate to the total-variation V T , which is a deterministic quantity.

Our strategy will be to first convert the regret rate of Theorem 3 into one which depends on a weaker worst-case notion of significant shift which does not depend on the observed X T . Although this notion of shift is weaker, it will be easier to relate to the total-variation quantity V T .

Recall that δ a t ( x ) . = max a ′ ∈ [ K ] δ a ′ ,a t ( x ) and δ a ′ ,a t ( x ) . = f a ′ t ( x ) -f a t ( x ) are the gap functions in mean rewards.

Definition 14 (worst-case sig shift) . Let τ 0 = 1 . Then, recursively for i ≥ 0 , the ( i +1) -th worstcase significant shift is recorded at time ˜ τ i +1 , which denotes the earliest time ˜ τ ∈ (˜ τ i , T ] such that there exists x ∈ X such that for every arm a ∈ [ K ] , there exists round s ∈ [˜ τ i , ˜ τ ] , such that δ a s ( x ) ≥ ( K t -˜ τ i ) 1 2+ d .

We will refer to intervals [˜ τ i , ˜ τ i +1 ) , i ≥ 0 , as worst-case (significant) phases . The unknown number of such phases (by time T ) is denoted ˜ L pop +1 , whereby [˜ τ ˜ L pop , ˜ τ ˜ L pop +1 ) , for τ ˜ L pop +1 . = T +1 , denotes the last phase.

We next claim that

This follows since the experienced significant phases [ τ i ( X T ) , τ i +1 ( X T )) interleave the population analogues [˜ τ i , ˜ τ i +1 ) in the following sense: at each significant shift τ i +1 ( X T ) , for each arm a ∈ [ K ] , there is a round s ∈ [ τ i ( X T ) , τ i +1 ( X T )] such that for δ s ( X τ i +1 ) > ( K τ i +1 -τ i ) 1 2+ d . This means there must be a worst-case significant shift ˜ τ j in the interval [ τ i ( X T ) , τ i +1 ( X T )] since the criterion of Definition 14 is triggered at x = X τ i +1 . This in turn allows us to conclude that each worst-case significant phase [˜ τ i , ˜ τ i +1 ) can intersect at most two significant phases [ τ i ( X T ) , τ i +1 ( X T )) .

Thus, by the sub-additivity of the function x ↦→ x 1+ d 2+ d (dropping dependence on X T in ˜ L, τ i to ease notation):

̸

where we use Jensen's inequality for a p + b p ≤ 2 1 -p ( a + b ) p for p ∈ (0 , 1) and a, b ≥ 0 in the last step to re-combine the subintervals of each worst-case significant phase [˜ τ j , ˜ τ j +1 ) .

Then, it suffices to show

To start, fix a worst-case significant phase [˜ τ i , ˜ τ i +1 ) such that τ i +1 < T +1 . By Definition 14, there exists a context x i ∈ X such that for arm a i ∈ argmax a ∈ [ K ] f a ˜ τ i +1 ( x i ) we have there exists a round t i ∈ [ τ i , τ i +1 ] such that:

On the other hand, δ a i ˜ τ i +1 ( x i ) = 0 by the definition of arm a i being the best at x i at round ˜ τ i +1 . Thus, letting ˜ a i ∈ argmax a ∈ [ K ] f a t i ( x i ) be an optimal arm at x i at round t i , we have:

For each round t = 2 , . . . , T , define the function H i : X × [0 , 1] K → [ -1 , 1] by H i ( X,Y ) . = Y ˜ a i -Y a i which is the realized difference in rewards between arms ˜ a i and a i within the reward vector Y . Then, using this notation and summing the above display over worst-case phases i ∈ [ ˜ L pop ] , we get:

We next recall from the variational representation of the total variation distance [Polyanskiy and Wu, 2022, Theorem 7.24] that for any measurable function H : X × [0 , 1] K → [ -1 , 1] ,

Thus, plugging (38) into (37), we get

Remark 7. Note that it is crucial in this argument that the functions H i do not depend on the realized rewards Y t or contexts X t at any particular round t . Rather, H i only depends on the arms ˜ a i , a i which in turn only depend on the mean reward sequence { f t } t ∈ [ T ] . In other words, the above step does not follow if a i , ˜ a i were defined in terms of the experienced significant shifts τ i ( X T ) .

Now, by Hölder's inequality for p ∈ (0 , 1) and q ∈ ( 0 , 1+ d 2+ d ) :

In particular, letting p = 1 3+ d and q = 1 (2+ d )(3+ d ) and plugging in our earlier bound (39) makes the above R.H.S.

F Proof of Theorem 1

We first note that it suffices to show (3) for integer L ∈ [0 , T ] ∩ N as lower bounds for all other L follow via approximation and modifying the constant c > 0 in (3). Thus, going forward, fix V ∈ [0 , T ] and L ∈ Z ∩ [0 , T ] .

At a high level, our construction will repeat L +1 times a hard environment for stationary contextual bandits. In particular, within each stationary phase of length T/ ( L + 1) one is forced to pay a regret of ( T/ ( L +1)) 1+ d 2+ d , summing to a total regret lower bound of ( L +1) · ( T/ ( L +1)) 1+ d 2+ d ≈ ( L +1) 1 2+ d · T 1+ d 2+ d .

To obtain the lower bound expressed in terms of total variation budget V in (3), we will choose L ∝ V 2+ d 3+ d · T 1 3+ d and argue that the actual total variation V T (Definition 5) is at most V in the constructed environments for this choice of L . This is similar to the arguments of the analogous dynamic regret lower bound [Besbes et al., 2019, Theorem 1] for the non-contextual bandit problem.

We start by establishing a lower bound for stationary Lipschitz contextual bandits. The construction is identical to that of Rigollet and Zeevi [2010, Theorem 4.1]. We provide the details here to (1) highlight a minor novelty in circumventing the reliance of the cited result on a positive 'margin parameter' α > 0 and (2) to assist in later calculating the total variation V T .

Remark 8. There are other stationary lower bound results for Lipschitz contextual bandits using similar constructions and arguments, but with context marginal measures µ X of finite support [Foster and Rakhlin, 2020, Theorem 2; Slivkins, 2014, Theorem 7]. To contrast, our construction involves setting µ X to be uniform on [0 , 1] d , thus satisfying the strong density assumption (Assumption 2).

Proposition 16. Suppose there are K = 2 arms. Then, there exists a finite family of stationary Lipschitz contextual bandit environments E ( n ) over n rounds such that for any algorithm π taking as input random variable U , we have for some constant c > 0 and environment E generated uniformly and independently (of π, U ) at random from E ( n ) :

Proof. Let the covariates X t be uniformly distributed on [0 , 1] d at each round t ∈ [ n ] , so that µ X ≡ Unif { [0 , 1] d } . For ease of presentation, let us reparametrize the two arms as +1 and -1 .

At each round t ∈ [ n ] , let arm -1 have reward Y -1 t ∼ Ber(1 / 2) and let arm +1 have reward Y +1 t ∼ Ber( f ( X t )) where f : X → [0 , 1] is some mean reward function to be defined. Let

We next partition X = [0 , 1] d into a regular grid of bins with centers Q = { q 1 , . . . , q M d } , where q k denotes the center of bin B k , k = 1 , . . . , M d . Concretly, re-indexing the bins, for each index k . = ( k 1 , . . . , k d ) ∈ { 1 , . . . , M } d , we define the bin B k coordinate-wise as:

Define C ϕ . = 1 / 4 . Then, let ϕ : R d → R + be the smooth function defined by:

It's straightforward to verify ϕ is 1 -Lipschitz over R d .

Next, to ease notation, define the integer m . = M d . Also, define Σ m . = {-1 , 1 } m and for any ω ∈ Ω m , define the function f ω on [0 , 1] d via

where ϕ j ( x ) . = M -1 · C ϕ · ϕ ( M · ( x -q j )) · 1 { x ∈ B j } . Then, the optimal arm at context x ∈ X in this environment is given by π ∗ f ( x ) . = sgn( f ( x ) -1 / 2) (where we use the convention sgn(0) . = 1 ). If x ∈ B j , then π ∗ f ω ( x ) = ω j .

Then, define the family C of environments induced by f ω for ω ∈ Ω m . Note that f ω is also 1 -Lipschitz for all ω . Next, let Int( B j ) be the ℓ ∞ ball centered at q k of radius 1 2 M (i.e., Int( B j ) is a ball of half the ℓ ∞ radius contained in B j ). Then, by the definition of ϕ ( x ) above, we have for any x ∈ Int( B j ) and any j ∈ [ m ] :

Next, we bound the average regret w.r.t. a uniform prior over the family C of environments as:

̸

̸

where the outer expectation is over an f ∈ C chosen uniformly at random from C .

As recall M ∝ n 1 2+ d , it will suffice to show the expectation on the above R.H.S. is of order Ω( n ) . This will follow essentially the same steps as the reduction to hypothesis testing in the proof of Theorem 4.1 in Rigollet and Zeevi [2010]. In particular, we note the KL divergence calculations for the induced distributions over observed data and decisions allows for the additional randomness U without consequence, as it's ignorable by use of KL chain rule. The only slight modification we make in their argument is to account for the fact that we only count rounds when X t ∈ Int( B j ) rather than when X t ∈ B j . This will, in fact, only affect constants in the lower bound as B j and Int( B j ) have similar masses.

Going into details, the following notation will be useful.

Notation 3. Let ω [ -j ] be ω with the j -th entry removed, and let ω i [ -j ] = ( ω 1 , . . . , ω j -1 , i, ω j +1 , . . . , ω m ) for i ∈ {± 1 } . Also, let P π,f , E π,f denote the joint measure and expectation, respectively, over all the randomness of π and observations within an environment induced by mean reward function f .

Then, the aforementioned reduction to hypothesis testing in the proof of Theorem 4.1 of Rigollet and Zeevi [2010] yields the following lower bound on (40):

where, letting X ∼ µ X be an independently drawn context,

We next claim ˜ N j,π ( ω ) = N j,π / 2 d , which will follow from swapping the order of expectations and summation in the above formulas. In particular, we have

Thus, (41) becomes lower bounded by

The above R.H.S. is optimized at z ∗ . = 3 M 2 4 log ( n 3 M 2+ d ) . Plugging this into the above along with our choice of M defined earlier gives us a lower bound of Ω( n 1+ d 2+ d ) .

Given Proposition 16, the ( L +1) · ( T L +1 ) 1+ d 2+ d lower bound immediately follows by lower bounding the total regret over a random environment formed by concatenating L +1 i.i.d. sampled environments from Unif( E ( T/ ( L +1))) . Any such resultant environment clearly has at most L global shifts. Note that the average regret (w.r.t. the random environment) over any stationary phase of length T L +1

is lower bounded by ( T L +1 ) 1+ d 2+ d regardless of the information learned prior to that phase, as such information can be formalized as exogeneous randomness U in Proposition 16.

Next, we tackle the lower bound V 1 3+ d · T 2+ d 3+ d in terms of total-variation budget V . First, if V < O ( T -1 2+ d ) , then we're already done as the desired rate

is minimized by the first term which is of order O ( T 1+ d 2+ d ) . Thus, using Proposition 16 with a single stationary phase E ( T ) gives lower bound of the right order in this regime. Such an environment clearly has total-variation V T = 0 ≤ V .

Suppose then that V ≥ 2 · T -1 2+ d . Let ∆ . = ⌈ ( T V ) 2+ d 3+ d ⌉ ≤ T and consider L + 1 = ρ · T/ ∆ stationary phases of length ∆ , for some fixed constant ρ > 0 . Then, by the previous arguments we have the regret is lower bounded by

Additionally, T 2+ d 3+ d · V 1 3+ d dominates T 1+ d 2+ d since V ≥ T -1 2+ d . Thus, the regret lower bound is proven in terms of V .

It remains to verify that the total-variation V T is at most V in the above constructed environments so that any such environment lies in the family P ( V, L, T ) .

Clearly, the 'instantaneous total-variation' ∥D t -D t -1 ∥ TV = 0 for all rounds t not being the start of a new stationary phase. On the other hand, for a round t marking the beginning of a new phase, we have that since conditioning increases the total-variation [Polyanskiy and Wu, 2022, Theorem 7.5(c)], the instantaneous total-variation is at most:

Since Y a t | X t = x ∼ Ber( f a t ( x )) , we have the R.H.S.'s inner TV quantity is just the total variation between Bernoulli's or max a ∈{± 1 } | f a t ( x ) -f a t -1 ( x ) | . Carefully analyzing the variations in the constructed Lipschitz reward functions in the proof of Proposition 16 reveals this TV between Bernoulli's is at most (3 e ) 1 2+ d 2 · ( L +1 T ) 1 2+ d . Then, summing the instantaneous total-variation over phases, we have

where the third inequality follows by letting ρ be a small enough constant.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies nonparametric contextual bandit problems with distributional shifts. This paper proposes a new notion of distributional changes called the experienced significant shifts. Based on this notion, the authors develop new algorithms that achieve the minimax rates without knowing some problem-dependent parameters.\n\n====After rebuttal====\nI have read the rebuttal. I'd like to keep my scores. I also encourage the authors to add more discussions regarding the adaptivity issue.\n\nSTRENGTHS:\nThe authors developed a new notion called the experience significant shifts that better capture the distribution shifts in contextual bandits. Based on this new notion, the authors show that the minimax optimal regret in contextual bandits with distributional shifts can be achieved, even without knowing problem-dependent parameters. In my opinion, this result is significant to the community. Also, along the way, the authors develop several new techniques to achieve this result (e.g., those have been highlighted in Section 5), which can be of independent interest.\n\nWEAKNESSES:\nIn the paper, the authors mention several generalizations of the existing setting, e.g., (i) generalization to H\\\"older continuity, and (ii) the one mentioned in Remark 1. However, no formal results are provided for these generalizations. It will be great if the authors can provide some formal statements.\n\nAlso, for the oracle procedure described in Definition 7, it's better formally state the power of the oracle, e.g., what is known to the oracle and what is unknown.\n\nQUESTIONS:\nIn bandit learning, when the goal is to minimize the cumulative regret, researchers have previously shown that adaptivity to the usual minimax rate is usually impossible and the best one can hope for is the Pareto optimality, e.g., for the model selection problem [1, 2]. However, in this paper, the authors show the opposite in this paper for learning with distributional shifts. Can authors elaborate more on this? Is this because of the common assumptions made in this setting, e.g., Assumption 2?\n\nCitations:\n\n[1] Teodor Marinov and Julian Zimmert. The Pareto frontier of model selection for general contextual bandits. \n[2] Yinglun Zhu and Robert Nowak. Pareto optimal model selection in linear bandits.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper studies the contextual bandits problem with changing Lipschitz reward function and proves a *minimax optimal* regret bounds for this problem, which includes both upper and lower bounds. For the upper bound, this paper comes up with an algorithm that achieves it. The algorithm is based on carefully maintaining a hierarchical partition tree that discretizes the context space.

STRENGTHS:
1. This paper achieves minimax optimality for the problem, which closes the existing gap and solves the problem.
2. The novel idea of significant shifts and the algorithm design of maintaining the partition tree, could be of independent interest. 
3. This paper provides excellent plain word explanation to highlight the key ideas and steps in their proof.

WEAKNESSES:
1. The algorithm is recursive and complicated, so it makes the audience hard to understand the algorithm, even with the algorithm explanation in Sec. 4. Maybe it is better to replace Line 9 in Algorithm 2 with some while-loop to improve readability.

QUESTIONS:
Typos:
1. Line 178, "Miminimax".
2. Below Line 259, Algorithm 1, "tree $T$" -> tree $\mathcal T$.
3. Line 305, "ut". 
4. Line 306, "(i.e., the bin at level $r_{s_2-s_1}$ containing $X_t$)" duplicates. 
4. Line 13 in Algorithm 2, trailing ; after :
The authors may spend some time to polish the paper before finalizing.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies nonstationary contextual bandits. In particular, a new notion of \"significant shift\" is introduced (Definition 6) which accounts for shifts in the distribution localized at the possible actions and with significant magnitude. First, the authors derive a lower bound for existing definitions of shifts (Theorem 1) and provide an oracle algorithm to achieve it. Then, after introducing the new notion of shifts, they derive an algorithm which adapts to it (i.e., which does not require the time indexes of the shifts), see Theorem 3.\n\nSTRENGTHS:\n- The topic is of interest to the NeurIPS community\n- The paper is globally clear and well written\n- The intuition and analysis are sound\n- Authors constantly compares their results with existing ones to position their contribution\n\nWEAKNESSES:\n- The paper lacks experiments. Since the authors provided a detailed algorithm, I find it disappointing that no experiment is carried out. It would be of particular interest to study the behavior of the different algorithm depending on the types of switches\n- I feel previous paper on significant switches could be discussed a bit more, to highlight the changes and challenges raised by the contextual setting\n- No lower bound for the proposed definition of switches is provided\n\n\nMinor:\n- Abstract: MAB is not defined yet\n- Section 1.1: linebreaks do not seem necessary to me and make the reading less fluid\n- When citing several works, the chronological order is preferable\n- Equation ($\\star$): what is $r(B)$?\n- Lines 242, 252: are the log factors omitted?\n- l. 305: ut\n- l. 316: $\\approx$ could be avoided\n\nQUESTIONS:\n- In this paper, the available actions are the same at every time steps. Could the authors think about a generalization when the action set changes over time? In particular, would it be possible to restrict the significant changes only to the playable arms?\n- Could the Lipschitz assumption be removed? \n- In Definition 7: does $\\cal{G}_t$ always exist?\n- Do authors have in mind simple examples of $f_a^t$ where both switches characterization drastically differ?\n- If I'm not missing anything, it seems to me that no definition of switch implies the other. Then, I find it a bit misleading to compare results all along. It should be made clear that the two are two different parameterizations incomparable in general (I agree on the identity $\\tilde{L} \\le L$)\n- Assume that changes have small magnitude (e.g., small drift at each time step) or do not apply to every arms (but only to the best let say), is the regret of CMETA not impacted by those switches?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies nonparametric contextual bandits where the mean reward functions can change over time. A key assumption is that the rewards are Lipchitz in context. The authors then adopt a typical approach to discretize the context space into bins. The notions “significant regret”, “unsafe at context”, and “experienced significant shift,”; in particular, an experienced significant shift implies a change in the optimal arm in a particular bin. The authors then propose an algorithm Contextual Meta-Elimination while Tracking (CMETA) and establish regret bounds in terms of the total number of “experienced significant shifts.”\n\nSTRENGTHS:\nThis paper is well-organized and introduces both a notion of “experienced significant shift” and an algorithm, CMETA, accompanied with theoretical guarantees.\n\nWEAKNESSES:\nMy major concerns centers around the comparison of CMETA and its analysis to the algorithm and regret analysis introduced by Suk and  Kpotufe (2022), and the presentation of the theoretical results. \n1. Besides discretization of the context space into bins, how does CMETA and its analysis differ from that introduced by Suk and  Kpotufe (2022)?\n2. As for the presentation of the results, we take Theorem 3 as an example. First, the notation is inconsistent: the notation $\\tilde{L}$ is introduced in line 214, and in Theorem 3 it is appeared as $\\tilde{L}$ in line 248 but $\\tilde{L}(\\mathbb{X}_T)$ in line 250. In addition, $\\tilde{L}$ is dependent on the discretization of context space into bins, which is determined by level $r$ in Algorithm 1, yet $r$ does not appear in Theorem 3.1? Moreover, could the authors elaborate on “choice of level” in Section 4 on how the level is adaptively chosen?\n\nQUESTIONS:\nMy main concerns were raised in the “Weakness” section. To reiterate:\n1. Besides discretization of the context space into bins, how does CMETA and its analysis differ from that introduced by Suk and  Kpotufe (2022)?\n2. It would be very helpful if the authors could elaborate on the “choice of level” in Algorithm 1 (CMETA) and how the regret bound established in Theorem 3.1 depends on the level.\n\nA remaining question is:\n-  In Corollary 4, it seems that if $\\tilde{L}$ grows linearly in $T$, the bound on cumulative regret becomes linear in $\\log^3(T) T$, in bandits where rewards are bounded in $[0, 1]$ (and cumulative regret is at most $T$)?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}