Title: Non-Stationary Bandits with Auto-Regressive Temporal Dependency
Venue: NeurIPS 2023
Accepted: yes
Keywords: non-stationary bandits; autoregressive model; low-regret policy; online learning algorithms
Abstract: Traditional multi-armed bandit (MAB) frameworks, predominantly examined under stochastic or adversarial settings, often overlook the temporal dynamics inherent in many real-world applications such as recommendation systems and online advertising. This paper introduces a novel non-stationary MAB framework that captures the temporal structure of these real-world dynamics through an auto-regressive (AR) reward structure. We propose an algorithm that integrates two key mechanisms: (i) an alternation mechanism adept at leveraging temporal dependencies to dynamically balance exploration and exploitation, and (ii) a restarting mechanism designed to discard out-of-date information. Our algorithm achieves a regret upper bound that nearly matches the lower bound, with regret measured against a robust dynamic benchmark. Finally, via a real-world case study on tourism demand prediction, we demonstrate both the efficacy of our algorithm and the broader applicability of our techniques to more complex, rapidly evolving time series.

Non-Stationary Bandits with Auto-Regressive Temporal Dependency

Abstract

Traditional multi-armed bandit (MAB) frameworks, predominantly examined under stochastic or adversarial settings, often overlook the temporal dynamics inherent in many real-world applications such as recommendation systems and online advertising. This paper introduces a novel non-stationary MAB framework that captures the temporal structure of these real-world dynamics through an auto-regressive (AR) reward structure. We propose an algorithm that integrates two key mechanisms: (i) an alternation mechanism adept at leveraging temporal dependencies to dynamically balance exploration and exploitation, and (ii) a restarting mechanism designed to discard out-of-date information. Our algorithm achieves a regret upper bound that nearly matches the lower bound, with regret measured against a robust dynamic benchmark. Finally, via a real-world case study on tourism demand prediction, we demonstrate both the efficacy of our algorithm and the broader applicability of our techniques to more complex, rapidly evolving time series.

1 Introduction

The multi-armed bandit (MAB) framework [51, 46] is commonly used to study online decisionmaking under uncertainty. It is primarily examined under either the stochastic setting [51, 6], where arms have fixed unknown reward distributions, or the adversarial setting [7, 5, 31, 16], where an adversary determines the reward distributions that can change arbitrarily over time. However, neither setting accurately represents real-world decision-making problems (see examples in dynamic pricing [30], online advertising [42], and online auctions [33]), where reward distributions change over time following intrinsic temporal structures. These structures often exhibit frequent variations and temporal dependencies, making them challenging to approximate using stationary distributions.

This motivates us to consider a non-stationary bandits setting involving certain temporal structure, which captures the real-world characteristics. Specifically, we are interested in temporal structures with linear amount of changes, which are distinct from the infrequent, limited changes typically handled by change-point detection [25] or seasonality analysis [15]. Such frequent changes are commonly observed in applications such as financial data [14] and click-through rates in online advertising [2], where the data experience lots of volatility within a short time. Traditional bandits algorithms, such as Thompson sampling or UCB, may explore too aggressively in response to these drastic changes, leading to suboptimal performance [39]. That being said, the observed time series in these scenarios typically exhibit certain temporal dependencies between past and present, which can be leveraged to improve decision-making.

To encapsulate the main characteristics of real-world time series, we relax the previously restrictive assumptions on the reward distributions and study a non-stationary MAB problem, where the reward

of each arm evolves over time following an auto-regressive (AR) temporal structure. The AR model is a popular time series model for predicting outcomes in applications such as sales [47], advertising [1] and marketing [18]. It simultaneously captures frequent changes (i.e., linear amount of changes over the time horizon) and temporal dependencies, both of which impact the quality of online decisionmaking. Although real-world data cannot be perfectly represented by an AR model, it has been shown that the AR parameters accurately capture temporal dependencies and can serve as a useful proxy [47]. The non-stationary bandits framework with an AR reward structure can effectively model a wide range of real-world applications, including:

(1) recommendation systems , where MAB can determine the best product/contents to display to users, while the AR model captures the evolution of user preferences and demand over time [21] (see Section 7 for a related case study on tourism demand prediction);

(2) online advertising , where MAB adjusts budget allocation for ad campaigns, and the AR model can capture the evolution of click-through rates of ads. The amount of changes are usually linear in time, making the AR structure a potential modeling choice [2];

(3) financial portfolio management , where MAB can determine the investment allocation, and the AR model can capture the evolution of returns for each investment/asset [4, 19].

When making online decisions in the presence of frequent changes, traditional methods that are designed for stationary or adversarial settings can be ineffective, due to two main challenges.

Balancing exploration and exploitation becomes more challenging, as complete exploration is often unattainable. Continuous exploration is crucial, but there is also a risk of over-exploration. Therefore, finding the right balance becomes essential in navigating this tradeoff effectively.

Frequent changes in the environment can significantly diminish the value of past learnings. Failing to promptly leverage the knowledge we have gained can lead to a swift deterioration in our confidence levels regarding the accuracy of our reward estimates.

The goal of this work is to show, via our non-stationary bandits framework with an AR reward structure, that effectively leveraging knowledge of temporal structure can address these challenges . In our approach, we would use our knowledge or estimates of the temporal dependency, measured by the AR parameter. Since non-stationary bandits with temporal structures exhibiting such real-world characteristics have been scarcely explored in the literature, we consider the AR structure as a suitable starting point for showcasing key ideas and techniques. We anticipate that this work will inspire future research on bandits with alternative temporal structures with similar features.

1.1 Main Contributions

We present an algorithm for the non-stationary AR bandits, called AR2 , which stands for 'Alternating and Restarting' algorithm for non-stationary 'AR' bandits. AR2 features two mechanisms: (i) an alternation mechanism that handles the first challenge above by enforcing exploitation at least every other round, and switch to exploration only when we discover an arm with potential, instead of simply adopting the principle of 'optimism in the fact of uncertainty' which can lead to over-exploration. (ii) a restarting mechanism that handles the second challenge by discarding unnecessary information when our confidence level deteriorates significantly, which balances a second tradeoff known as 'remembering'-and-'forgetting'; this tradeoff is crucial even for time series with limited changes [10]. Overall, our work addresses an important gap in the literature by studying non-stationary bandits with temporal structure that exhibit frequent changes and temporal dependency. While the AR structure does not capture all temporal structures in the real world, we believe that our high-level messages and techniques will guide future research on bandits with other temporal structures.

In evaluating the performance of our algorithm, we employ the concept of per-round steady state dynamic regret (Definition 2.1). This metric serves as a robust dynamic benchmark that competes with the best arm in each round. It surpasses the static benchmark used in stochastic and adversarial MAB, making it a considerably stronger measure of performance. We provide an upper bound on the regret of AR2 (Theorem 5.2), for which the analysis is rather intricate (Section 6), and show that it almost matches the regret lower bound (Theorem 3.1). Our lower bound result also characterizes the challenges embedded in AR temporal processes, as it shows the per-round dynamic regret does not go to zero as T increases, implying that any algorithm needs to keep exploring over time.

Finally, we conduct a real-world case study on tourism demand prediction [36] in Section 7, confirming the superiority of AR2 compared to benchmark algorithms. There, we also show that the techniques and high-level ideas of our algorithm can be readily extended to handle more complicated, rapidly changing temporal structure (e.g., general ARp processes) while still achieving good performance. Our case study is complemented by synthetic experiments in Appendix A which show the strength of AR2 against a number of benchmarks designed for stationary and non-stationary settings.

1.2 Related Work

Our work, which focuses on MAB with an AR temporal structure, contributes to the non-stationary MABliterature that draws attention in recent years. Most related works on non-stationary bandits focused on either the rested bandits [26] or restless bandits [53]. In rested bandits, only the expected reward of the arm that is being pulled will change, while in restless bandits, the expected reward of each arm changes at each round according to a known yet arbitrary stochastic process, regardless of whether the arm is pulled. Recently, there has also been a third stream of literature that studies non-stationary bandits where the changes in reward distributions depend on both which arm is pulled and the time period that passes by since its last pull (see, e.g., [32, 13]). The non-stationary bandit problem that we study belongs to the restless setting, since the reward distributions change at each round based on AR processes, independent of our actions. Restless bandits, however, is known to be difficult and intractable [43], and many work thus focus on studying its approximation [28] or relaxation [9].

One line of closely related work on restless bandits studies non-stationary MAB problems with limited amount of changes. These works either assume piecewise stationary rewards (see [25, 37, 12]), or impose a variation budget on the total amount of changes (see [10]). They differ from our setting in that they rely on the amount of changes in the environment to be sublinear in T , while in AR processes, changes are more rapid and linear in T . Another line of relevant research on restless bandits assumes different types of temporal structures for the reward distributions. See, e.g., [38, 41, 50, 40] for finite-state Markovian processes, [27] for stationary ϕ -mixing processes and [48] for Brownian processes, which is a special case of an AR model but does not experience the exponential decay in the correlation between the past and the future. Recently, [39] proposed a predictive sampling algorithm that can also be applied to AR-1 bandits; their main focus, however, is to show the algorithm's superiority over the traditional Thompson sampling algorithm in non-stationary environments.

Similar to the second line of works above, our work also aims to exploit the temporal structure of the reward distributions to devise well-performing learning algorithms. However, the AR process is much more unpredictable than the temporal structures studied previously, due to its infinite state space and the fast decay in the values of the past observations. We are thus prompted to take a different approach that is designed specifically for adapting to the rapid changes attributed to the AR process.

2 Preliminaries

Expected rewards. Consider a non-stationary MAB problem with k arms over T rounds. The state (expected reward) of arm i ∈ [ k ] at any round t ∈ [ T ] is denoted by r i ( t ) , where r i ( t ) ∈ [ -, + ] . Conditioned on the state r i ( t ) , the realized reward of arm i at round t , denoted by R i ( t ) , is given by R i ( t ) = r i ( t ) + ϵ i ( t ) , where ϵ i ( t ) ∼ N (0 , σ ) is independent across arms and rounds and we call σ ∈ (0 , 1) the stochastic rate of change. More generally, if each arm i ∈ [ k ] has a different, unknown stochastic rate of change σ i , as long as we have an upper bound σ ≥ max i σ i , all of our theoretical results naturally carries over with the same dependency on σ .

Evolution of expected rewards. The expected reward of each arm i ∈ [ k ] , r i ( t ) , evolves over time. In our model, we assume that r i ( t ) undergoes an independent AR-1 process:

or equivalently, R i ( t + 1) = αR i ( t ) + ϵ i ( t + 1) , where α ∈ (0 , 1) is the parameter of the AR-1 model and can be used as a proxy to measure temporal correlation over time. As α increases, the temporal correlation increases. As commonly seen in the bandit literature (e.g., [35]), we make the rewards bounded by truncating r i ( t +1) when its absolute value exceeds some finite boundary value > 0 . That is, r i ( t +1) = B ( α ( r i ( t ) + ϵ i ( t )) ) , where B ( y ) = min { max { y, - } , } .

Here, we assume that the AR parameter α is known to the decision maker, a common assumption in literature [10, 27, 48], and often justifiable in real-world scenarios where sufficient historical data enables accurate fitting of AR models (see Section 7). This assumption is made to simplify the analysis. However, in Section 8, we would show that this assumption can be relaxed, and the AR parameter can be effectively estimated via maximum likelihood estimation.

We also assume here that all arms share the same AR parameter α . As our numerical studies in Appendix A and Section 8 suggest, this assumption can be relaxed too. There, we generalize our model by considering heterogeneous AR parameters α i for each arm i ∈ [ k ] , and our algorithm maintains its performance. Nonetheless, it is worth noting that the homogeneity of AR parameter can be justified in some applications. One example can be seen in our case study in Section 7, where a travel agency offers vacation packages to travelers originating from a particular origin and heading

to a specific destination. In this scenario, the demand is modeled as a general AR4 process, with the AR parameters being consistent across all arms. This is because the temporal correlation here is primarily determined by exogenous factors related to the travel route, such as popular attractions and the availability of flights between the two locations, hence impacting all arms in the same way.

Feedback structure and goal. Our goal is to design an algorithm that obtains high reward, against a strong dynamic benchmark that in every round t , pulls an arm that has the highest expected reward. This benchmark is much more demanding than the time-invariant benchmark used in traditional MAB literature. Obtaining good performance against the dynamic benchmark is particularly challenging as in any round t ∈ [ T ] , only the reward of the pulled arms can be observed. Not being able to observe the reward of unpulled arms in dynamic and time-varying environments introduces missing values in the AR process associated with arms. This, in turn, deteriorates the prediction quality of the future expected rewards and the performance of any algorithm that relies on such predictions (see [3] for a work that studies the impact of missing values on the prediction quality in AR processes).

We now formally define how we measure the performance of any non-anticipating algorithm.

Definition 2.1 (Dynamic steady state regret) . Let A be a non-anticipating algorithm that, at each round t ∈ [ T ] , pulls arm I t based on the history { I 1 , R 1 , . . . , I t -1 , R t -1 } , where R t ′ = R I t ′ ( t ′ ) is the observed reward at round t ′ < t . The dynamic regret of A at round t is defined as

where r ⋆ ( t ) = max i ∈ [ k ] { r i ( t ) } is the maximum expected reward at round t . Furthermore, the per-round steady state regret of algorithm A is defined as

We remark that in Definition 2.1, our per-round regret is defined asymptotically mainly because we would like to focus on evaluating the steady state performance of our algorithm (i.e., under the steady state distribution). 1 All of our theoretical analyses for regret lower bound (Section 3) and upper bounds (Section 5) are also performed under the steady state distribution.

In this work, we would like to design an algorithm with a small per-round steady state regret, where the per-round regret is a function of the stochastic rate of change ( σ ) and the temporal correlation ( α ).

3 Regret Lower Bound

We now provide a lower bound on the best achievable dynamic per-round steady state regret.

Theorem 3.1 (Regret Lower Bound) . Consider a non-stationary MAB problem with k arms, where the expected reward of each arm evolves as an independent AR-1 process with parameter α , stochastic rate of change σ and truncating boundaries [ -, ] . Then the per-round steady state regret of any algorithm A is at least Ω( g ( k, α, σ ) ασ ) , where g ( k, α, σ ) is the probability that two best arms are within ασ distance of each other at any given round, under the steady state distribution. See the expression of this probability in Equation (16) in Appendix F.

The proof of Theorem 3.1, provided in Appendix F, builds on the following idea: even if algorithm A has access to all past information at round t , due to stochastic noise ϵ i ( t ) , A will pull a sub-optimal arm with constant probability at round t +1 . We show that the probability of the two best arms' expected rewards being within ασ of each other can be expressed as g ( k, α, σ ) (see (16)). 2 If the two best arms are ασ close, A will incur Ω( ασ ) regret with constant probability at round t +1 .

In the extreme case when α is close to one, the steady state distribution of r i ( t ) can be approximated with a uniform distribution within the boundaries and two probability masses at the boundaries (see Appendix D). Using the uniform approximation, one can show that g ( k, α, σ ) = Ω( kασ ) (see discussion in Appendix F). This then yields a regret lower bound of order Ω( kα 2 σ 2 ) .

Theorem 3.1 implies that under our setup, the best achievable per-round regret with respect to a strong dynamic benchmark does not converge to zero, which differs from stationary or adversarial

Algorithm 1 Alternating and Restarting algorithm for non-stationary AR bandits ( AR2 )

Input: AR Parameter α , stochastic rate of change σ , epoch size ∆ ep, parameter c 0 .

Set the epoch index s = 1 , parameter c 1 = 24 c 0 .

Repeat while s ≤ ⌈ T/ ∆ ep ⌉ :

(a) Initialization: Set t 0 = ( s -1)∆ ep. Set the initial triggered set T = ∅ . Pull each arm i ∈ [ k ] at round t 0 + i and set τ i = t 0 + i . Set estimates ̂ r i ( t 0 + k +1) = α k -i · B ( αR i ( t 0 + i )) .

(b) Repeat for t = t 0 +( k +1) , . . . , min { t 0 +∆ ep , T }

Update the identity of the superior arm and its estimated reward

Trigger arms with potential: For i / ∈ T ∪ { i sup ( t ) } , trigger arm i if

If triggered, add i to T and set τ trig i = t .

Alternate between exploration and exploitation:

-If t is odd and T ̸ = ∅ , pull a triggered arm with the earliest triggering time: I t = arg min j ∈T τ trig j .

-Otherwise, pull the superior arm I t = i sup ( t ) .

Receive a reward R I t ( t ) , and set τ I t = t .

Maintain Estimates: Set ̂ r I t ( t +1) = B ( αR I t ( t )) and set ̂ r i ( t +1) = α ̂ r i ( t ) for i = I t .

(c) Set s = s +1 .

bandits with a time-invariant benchmark where algorithms like UCB for stationary bandits and Exp3 for adversarial bandits can achieve zero per-round regret as T approaches infinity. In our setting, each arm undergoes linear amount of changes, making it challenging for any algorithm to adapt to the changing environment in time. Our result aligns with the lower bound from [10], which states that when the total variation of expected rewards is O ( T ) , the regret also grows linearly.

4 Algorithm AR2

In this section, we present our algorithm, called AR2 (Alternating and Restarting algorithm for nonstationary "AR" bandits). Algorithm 1 outlines the workings of AR2 , which operates in epochs of a fixed length. Within each epoch, AR2 maintains and updates estimates of the expected rewards for each arm i ∈ [ k ] . It alternates between exploitation and exploration steps based on these estimates. During exploitation, AR2 plays a "superior arm" expected to yield high rewards, while in exploration, it selects a "triggered arm" that hasn't been pulled recently but has high potential. At the end of an epoch, the algorithm restarts. As alluded earlier, AR2 effectively balances two inherent tradeoffs in non-stationary MAB problems with AR reward structure. Firstly, it addresses the explorationexploitation tradeoff by alternating between exploiting the superior arm and exploring the triggered arm within each epoch. Secondly, it handles the tradeoff between 'remembering'-and-'forgetting', a commonly considered tradeoff in non-stationary environments [10, 11], via restarting.

Maintaining estimates of expected reward of arms. For any round t and arm i ∈ [ k ] , let τ i ( t ) be the last round before t (including t ) at which arm i is pulled, and let τ next i ( t ) be the next round after t (excluding t ) at which arm i is pulled. Let ∆ τ i ( t ) = τ next i ( t ) -τ i ( t ) be the gap between two consecutive pulls of arm i around t . Define ̂ r i ( t ) as the estimate of the reward of arm i at t based on the most recent observed reward of arm i (i.e., R i ( τ i ( t )) ). Via recursive updates (see Step (b)), AR2 maintains the following estimate of expected reward for each arm:

Superior arms. The superior arm at round t , denoted by i sup ( t ) , is one of the two most recently pulled arms, i.e., I t -1 or I t -2 , that has the higher estimated reward (see (1)). We further define ̂ r sup ( t ) = ̂ r i sup ( t ) ( t ) as the estimated reward of the superior arm at round t . We remark that here, for simplicity of analysis, our definition of the superior arm only considers the two most recently pulled arms. We can in fact set the superior arm to be the one with the highest estimated reward among the m most recently pulled arms for any constant m ≥ 2 . A similar theoretical analysis will then yield the same theoretical guarantee, as shown in Theorem 5.2. We further verify the robustness of AR2 to the choice of m in our numerical studies, where we consider all arms as potential candidates for the superior arm, and AR2 maintains its competitive performance (see Appendix A).

̸

̸

Triggered arms. In order to adapt to changes, AR2 identifies and keeps track of a set of arms with potential that are not pulled recently. We refer to these arms as triggered arms and denote their associated set by T . In a round t , an arm i = i sup ( t ) gets triggered if the triggering criteria in (2) is satisfied. We call the earliest such time t > τ i ( t ) as the triggering time of arm i and denote it as τ trig i ( t ) . If arm i is triggered, it is added to the triggered set T . When arm i gets pulled or is chosen as the superior arm, it is removed from T . The right-hand side of the triggering criteria is a confidence bound constructed based on Hoeffding's Inequality (see details in Section 6 and Appendix H).

Note that at the exploration step, AR2 would only pull a triggered arm if the triggered set T is non-empty. This inherently adjusts the rate of exploration in our alternation mechanism. In a slowlychanging environment (e.g., with small σ ), the triggered set may not always include arms needing exploration, thus allowing focused exploitation of the superior arm; in a fast-changing environment, the rate of exploration can be as high as the rate of exploitation.

5 Regret of Algorithm AR2

Our algorithm AR2 works for AR-1 model with any choice of parameter α ∈ (0 , 1) . However, it turns out that the problem is less challenging when the future is weakly correlated with the past (i.e., when α is small). Theorem 5.1 states that any algorithm-including both a naive approach that continuously pulls the same arm throughout the horizon and our algorithm AR2 -can achieve nearoptimal performance when α is too small. This is because with small α , (i) for any arm i ∈ [ k ] , the steady state distribution of its expected reward r i ( t ) concentrates around zero with high probability, and (ii) our observations of past rewards quickly deteriorate in their value of providing insights on the evolution of expected rewards in the future.

Theorem 5.1. Any non-anticipating algorithm A incurs per-round steady state regret of at most O ( min( √ log(1 /ασ ) + log k · ασ √ 1 -α 2 , 2 ) ) .

In Figure 1, we compare the order of the regret upper bound in Theorem 5.1 with the order of the regret lower bound in Theorem 3.1. 3 Observe that ¯ α ≜ 0 . 5 serves as a rough threshold value such that when α ∈ (0 , ¯ α ) , the orders of lower and upper bounds almost match each other, which suggests limited room for improvement. (That being said, our numerical studies in Appendix A in fact show that AR2 still outperforms other benchmarks even when α is small.) On the other hand, as α → 1 , the gap between lower and upper bounds start to expand, implying that the problem becomes more difficult and simplistic approaches such as the naive algorithm would no longer produce satisfying performance.

In the rest of the discussion, we thereby focus on the more challenging regime (i.e., α ∈ [ ¯ α, 1) ), and establish a regret upper bound for AR2 when α ∈ [ ¯ α, 1) .

0.2

0.4

0.6

0.8

10

2

10

1

value

C

log(1/

) + log

k

1

2

g

(

k

,

,

)

Theorem 5.2. Let K ( α ) ≜ ⌊ 1 2 (log( 1 8 ) / log α + 1) ⌋ , and let A be the AR2 algorithm that gets restarted in each epoch of length ∆ ep = ⌈ kα -3 σ -3 ⌉ . If α ∈ [ ¯ α, 1) and k ≤ K ( α ) , the per-round steady state regret of A satisfies REG A ≤ O ( c 2 0 α 2 σ 2 k 3 log( c 0 ασ √ k )) , where c 0 = √ 4 log(1 /ασ ) + 4 log ∆ ep +2log(4 k ) .

Recall that if α is close to one, the regret lower bound can be characterized as Ω( kα 2 σ 2 ) . This shows that our algorithm is optimal in terms of AR parameter α and stochastic rate of change σ (up to logarithmic factors). In Appendix E.3, we further highlight the significance of Theorem 5.2 by illustrating the evolution of upper and lower bounds, as well as the per-round regret attained by AR2 , at different values of α and σ . We show that Theorem 5.2 well characterizes the performance of AR2 under different settings.

We remark that in terms of the dependency on k , Theorem 5.2 requires k ≤ K ( α ) mainly for the rigor of theoretical analysis, and this bound loosens as α approaches one (for example, when α = 0 . 95 ,

K ( α ) = 20 .). 4 Our numerical studies in Appendix A also reveal that (i) AR2 maintains its competitive performance even when the number of arms k exceeds K ( α ) ; and (ii) its per-round regret grows modestly with k . These suggest that the assumption/dependency on k for our upper bound are artifacts of our analysis, rather than an intrinsic property of our algorithm.

6 Proof of Theorem 5.2

The proof of Theorem 5.2 proceeds in four steps. Step 1 introduces distributed regret , a novel notion that distributes instantaneous regret over subsequent rounds, enabling us to rephrase per-round steady state regret. Step 2 defines a high-probability good event crucial for analyzing distributed regret. In Step 3, we leverage the good event to establish an upper bound for each round's distributed regret. Finally, Step 4 aggregates these regrets across rounds and epochs, establishing the per-round steady state regret of AR2 against the dynamic benchmark.

Step 1: Distributed regrets. Let ∆ r i ( t ) = r ⋆ ( t ) -r i ( t ) be the instantaneous regret from pulling arm i at round t , where r ⋆ ( t ) = max i ∈ [ k ] r i ( t ) is the maximum expected reward at round t . Consider the expected per-round regret: REG A = E [ 1 T ∑ T t =1 REG A ( t ) ] = 1 T ∑ T t =1 E [ ∆ r I t ( t ) ] , where ∆ r I t ( t ) = r ⋆ ( t ) -r I t ( t ) is the instantaneous regret incurred at round t . In the following, we show that we can distribute this instantaneous regret over subsequent rounds.

Note that for any i ∈ [ k ] , let ℓ ( j ) i denote the round at which arm i gets pulled for the j th time within an epoch. Then during the period between two consecutive pulls ℓ ( j ) i ≤ t < ℓ ( j +1) i , the only regret incurred from pulling arm i is the instantaneous regret ∆ r i ( ℓ ( j ) i ) . By previous definition, we also have τ i ( t ) = ℓ ( j ) i , τ next i ( t ) = ℓ ( j +1) i and ∆ τ i ( t ) = ℓ ( j +1) i -ℓ ( j ) i for all ℓ ( j ) i ≤ t < ℓ ( j +1) i . Observe that we can decompose ∆ r i ( ℓ ( j ) i ) as follows, which then leads to the notion of distributed regret :

Definition 6.1 (Distributed regret) . The distributed regret of arm i at round t is defined as

Note that D i ( t ) = 0 if arm i is the best arm at round τ i ( t ) . Now, let T i be the total number of rounds we pull arm i in the horizon, we can rewrite the expected per-round regret as

where the second equality follows from the Definition 6.1 and the third equality follows from that AR2 proceeds in epochs of length ∆ ep, and S = T/ ∆ ep is the total number of epochs.

Step 2: Good event and its implications. Before proceeding, we first define a good event G ( t ) at round t , which would help simplify our analysis. In principle, we say that a good event G ( t ) happens at round t if the noises within the epoch including t are not too large in magnitude. Recall from Section 2 that r i ( t ) follows an AR-1 process with truncating boundaries: r i ( t + 1) = B ( α ( r i ( t ) + ϵ i ( t ))) , where B ( y ) = min { max { y, - } , } . Hence, we need to first define a new noise term that shows the influence of the truncating boundary.

In particular, the new noise ˜ ϵ i ( t ) satisfies the following recursive relationship: r i ( t +1) = αr i ( t ) + α ˜ ϵ i ( t ) . We now formally define the good event G ( t ) , which states that for any sub-interval [ t 0 , t 1 ] within the epoch including t , the weighted sum of ˜ ϵ i ( t ) satisfies a concentration inequality.

Definition 6.2 (Good event at round t ) . We say that the good event G ( t ) occurs at round t if for every i ∈ [ k ] , and for every sub-interval [ t 0 , t 1 ] , where t -∆ ep ≤ t 0 < t 1 ≤ t +∆ ep, the weighted sum of the noises ˜ ϵ i ( t ) satisfies ∣ ∣ ∣ ∑ t 1 -1 t = t 0 α t 1 -t ˜ ϵ i ( t ) ∣ ∣ ∣ < c 0 σ √ ( α 2 -α 2( t 1 -t 0 +1) ) / (1 -α 2 ) , where c 0 = √ 4 log(1 /ασ ) + 4 log ∆ ep +2log(4 k ) .

By building a connection between ˜ ϵ i ( t ) and ϵ i ( t ) and Hoeffding's inequality, we show Lemma 6.3, which confirms that the good event G ( t ) happens with high probability .

Lemma 6.3. For any t ∈ [ T ] , we have P [ G c ( t )] ≤ ( ασ ) 2 .

Step 3: Distributed regret analysis. For a given round t in the s -th epoch, we can bound its expected distributed regret by first decomposing it into two terms: E [ D i ( t )] = E [ D i ( t ) ✶ G c ( t ) ]+ E [ D i ( t ) ✶ G ( t ) ] , where ✶ E is the indicator function of event E . We can bound the first term by applying Lemma 6.3

To bound the second term, we rely on the implications of the good event G ( t ) (presented in Appendix H.1) to show the following.

Lemma 6.4. Suppose that we apply AR2 to the non-stationary MAB problem with α ∈ [ ¯ α, 1) and and k ≤ K ( α ) , for every arm i ∈ [ k ] and some round τ i ( t ) ≤ t < τ next i ( t ) , where τ i ( t ) and τ next i ( t ) are two consecutive rounds at which i gets pulled, we have E [ D i ( t ) ✶ G ( t ) ] ≤ O ( c 2 0 α 2 σ 2 k 2 log( c 0 σα √ k )) .

The proof of Lemma 6.4 is deferred to Appendix H.2. There, we provide a bound for the nominator and denominator of D i ( t ) respectively, conditioning on the good event G ( t ) and the value of ∆ r i ( t ) . In the proof, we critically use the implications of the good event (see Appendix H.1). Now, combining (6) and Lemma 6.4, for any round t within the s th epoch such that τ i ( t ) ≤ t < τ next i ( t ) , we have

Step 4: Summing the distributed regrets. Given Equations (4), (7) and ∆ ep = ⌈ kα -3 σ -3 ⌉ , summing the expected distributed regrets first within an epoch, and then along the horizon yields

where the additional 2 term comes from the initialization round and the last round we pull arm i within an epoch. This concludes the proof if we plug in ˜ C = c 0 ασ √ k . ■

7 A Real-World Case Study on Tourism Demand Prediction

In this section, we numerically demonstrate the efficacy of AR2 via a real-world case study based on a international Tourism Demand dataset for Australia [36]. In this case study, we act as a travel agency that needs to determine which vacation package to offer, where the demand for each vacation package is highly dependent on the tourism demand during each quarter. Our case study is further complemented by a number of synthetic experiments in Appendix A, where we compare AR2 against various benchmarks designed for both stationary and non-stationary settings and again show the superior performance of AR2 in adapting to the rapidly changing environment.

Dataset and setup. The international tourism demand dataset [36], obtained from the Australian Bureau of Statistics, records the number of individual tourist arrivals to Australia from Hong Kong during each quarter between the years 1975-1989. The authors of [36] have fitted an AR model to the logarithms of quarterly tourist arrivals, which results in an AR4 model with a trend component 5 : r i ( t ) = -0 . 01 + 0 . 32 R i ( t -2) + 0 . 6 R i ( t -4) . Note that here, the number of tourist arrivals is strongly correlated with the number of arrivals four quarters before, which is likely due to seasonal patterns. The time series analysis in [36] also supports our assumption in Section 2 that the AR

parameters are known. In this dataset, as well as many others from the real world, decision-makers are likely to have access to historical data, which enables them to fit AR processes with good precision.

Given the AR model, we consider a recommendation setting where the travel agencies have k = 5 vacation packages (arms) to offer to the tourists from Hong Kong, and wish to dynamically feature the package with the highest demand (reward) at each round. For each arm, we simulate its reward sequence by randomly drawing the initial reward from a uniform distribution in [0 , 1] and simulate r i ( t ) for a total of T = 200 rounds, based on the AR-4 model above. We take the stochastic rate of change to be σ = 0 . 1 for all arms.

Extension of AR2 to ARp processes. We extend our algorithm AR2 to handle MAB problems with rewards modeled using general ARp processes with trends; see Algorithm 2 in Appendix B.2. At a high level, the extension of AR2 uses similar techniques as described in Section 4. The key difference is that, for each arm i ∈ [ k ] , it maintains not only an estimate ̂ r i ( t ) of the arm's reward at round t , but also an estimate ̂ E i ( t ) that captures the associated estimation error. These estimates are dynamically updated based on the structure of the ARp process with trends. The extended algorithm, akin to Algorithm 1, (i) selects the arm with the highest estimated reward ̂ r i ( t ) and (ii) triggers arms with potential, where the confidence bound in the triggering criteria now depends on the estimate of the error ̂ E i ( t ) . For a comprehensive discussion of our extension, please refer to Appendix B.2.

Results. We compare the performance of AR2 against the two most competitive benchmarks ( ϵ -greedy and a modified UCB algorithm) that we identfied in synthetic experiments; see Appendix A for comparisons with the other benchmarks. The comparison is shown in Table 1. It can be seen that AR2 stands out in terms of both the regret 6 and the number of times it pulls the optimal arm. In this case study, while the ϵ -greedy algorithm frequently selects the optimal arm, it still accumulates high regret due to its purely random exploration, which can lead to the selection of arms with low rewards. Similarly, the modified UCB algorithm, which leverages the knowledge of the temporal structure in its confidence bound (see description in Appendix C), doesn't perform well. Due to the rapidly changing nature of the ARp process, modified UCB over-explores. AR2 , on the other hand, prove to be effective even amidst the rapid changes introduced by more complex time series.

8 Extension on Learning the AR Parameter

In the previous sections, we have assumed full knowledge of the AR parameter, which as discussed in Sections 2 and 7, when we have access to past data. This section proposes an algorithm extension for when the AR parameter needs estimation. We introduce a maximum likelihood method and numerically evaluate the performance of AR2 , which show that AR2 still remains competitive despite noise in the estimated AR parameter.

A Maximum Likelihood Estimator. If the decision maker does not have prior knowledge of the AR parameter α , one can learn the AR parameter via Maximum Likelihood Estimation (MLE). In the first T est rounds of the time horizon, where T est = O ( T β ) for some β ∈ (0 , 1) , we pull one arm i ∈ [ k ] consecutively for a fixed arm i and observe rewards R i (1) , . . . , R i ( T est ) . We then define the maximum likelihood estimator ̂ α = arg min α ∈ (0 , 1) L ( α ) , where L ( α ) is the negative of the log-likelihood function defined in (60) of Appendix I. Observe that here we cannot directly apply linear regression for estimating α because the existence of truncating boundaries would lead to a biased estimator. MLE, on the other hand, remains robust even when the expected reward of our arm hits the boundary. Since the number of rounds used for estimation T est scales sublinearly with T , the regret incurred within the first T est rounds would not impact our per-round regret upper bound. If we have heterogeneous AR parameters α i for arms i ∈ [ k ] (as in Appendix A), one can simply pull each arm i ∈ [ k ] consecutively for T est rounds and perform MLE for each α i .

The following proposition quantifies the amount of noise in the estimated AR parameter.

Proposition 8.1. Let ̂ α = arg min α ∈ (0 , 1) L ( α ) be the estimated AR parameter. Then, for γ > 0 , with probability at least 1 -2 /T γ est -2 exp( -T est V 2 / 2 2 ) , there exists constants L 1 , L 2 such that

| ̂ α -α | ≤ (4 σ L 1 ) / ( V L 2 ) · √ 2 γ log T est /T est , where V is the variance of the observed reward in the steady state distribution, which is independent of our algorithm or the time horizon T .

We plot the normalized regret incurred by AR2 , ϵ -greedy and mod-UCB using the estimated AR parameters, obtained through MLE with T est = { 25 , 50 , 100 } . We observe that whenever the AR parameters α i are small or large, the performance of AR2 outcompetes the other two benchmarks. In particular, AR2 appears to be robust to the noises in the estimated AR parameter, with performance close to what we would otherwise obtain with the accurate AR parameters (see Table 2 in Appendix A).

AR2

-greedy

0.55

0.65

0.75

0.85

small

i

Test

=25

AR2

-greedy

0.55

0.65

0.75

0.85

Test

=50

AR2

-greedy

0.55

0.65

0.75

0.85

Test

=100

AR2

-greedy

0.2

0.4

0.6

large

i

AR2

-greedy

0.2

0.4

0.6

AR2

-greedy

0.2

0.4

0.6

9 Conclusion and Future Directions

In this paper, we studied a non-stationary MAB problem with an AR structure, which captures the rapid changes commonly observed in real-world dynamics. Our proposed algorithm, AR2 , leverages our knowledge or estimate of the temporal dependency to effectively handle the challenges associated with the fast-changing environment, and our techniques can be potentially adapted to more complex temporal series. As the realm of non-stationary bandits with rewards governed by temporal structures remains largely unexplored, there are several exciting avenues for future research. One intriguing direction is to incorporate seasonality into our framework, exploring models such as seasonal ARIMA with both long-term and short-term changes. Additionally, building on the promising numerical results in Section 8, it would be interesting to theoretically characterize the performance of our algorithm when estimation of the AR parameter and online-decision making are performed simultaneously.

Acknowledgments

We are grateful to the MIT-IBM Watson AI Lab for their generous support.

References

D. A. Aaker, J. M. Carman, and R. Jacobson. Modeling advertising-sales relationships involving feedback: a time series analysis of six cereal brands. Journal of Marketing Research , 19(1):116125, 1982.

D. Agarwal, B.-C. Chen, and P. Elango. Spatio-temporal models for estimating click-through rate. In Proceedings of the 18th International Conference on World Wide Web , page 21-30, 2009.

O. Anava, E. Hazan, and A. Zeevi. Online time series prediction with missing data. In Proceedings of the 32nd International Conference on Machine Learning , volume 37, pages 2191-2199, 2015.

A. A. Ariyo, A. O. Adewumi, and C. K. Ayo. Stock price prediction using the arima model. In 2014 UKSim-AMSS 16th international conference on computer modelling and simulation , pages 106-112. IEEE, 2014.

P. Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3:397-422, 2003.

P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning , 47:235-256, 2002.

P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The non-stochastic multi-armed bandit problem. SIAM journal of computing , 32:48-77, 2002.

K. Azuma. Weighted sums of certain dependent random variables. Tohoku Mathematical Journal, Second Series , 19(3):357-367, 1967.

D. Bertsimas and J. Niño-Mora. Restless bandits, linear programming relaxations, and a primal-dual index heuristic. Operations Research , 48(1):80-90, 2000.

O. Besbes, Y. Gur, and A. Zeevi. Stochastic multi-armed-bandit problem with non-stationary rewards. In Advances in Neural Information Processing Systems , volume 27, 2014.

I. Bogunovic, J. Scarlett, and V. Cevher. Time-varying gaussian process bandit optimization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics , volume 51, pages 314-323. PMLR, 2016.

Y. Cao, Z. Wen, B. Kveton, and Y. Xie. Nearly optimal adaptive procedure with change detection for piecewise-stationary bandit. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics , volume 89 of PMLR , pages 418-427, 2019.

L. Cella and N. Cesa-Bianchi. Stochastic bandits with delay-dependent payoffs. In International Conference on Artificial Intelligence and Statistics , pages 1168-1177. PMLR, 2020.

V. Chavez-Demoulin and J. McGill. High-frequency financial data modeling using hawkes processes. Journal of Banking & Finance , 36(12):3415-3426, 2012.

N. Chen, C. Wang, and L. Wang. Learning and optimization with seasonal patterns. arXiv preprint arXiv:2005.08088 , 2020.

V. Dani and T. P. Hayes. Robbing the bandit: Less regret in online geometric optimization against an adaptive adversary. In Proceedings of the 17th Annual ACM-SIAM Symposium on Discrete Algorithm (SODA) , page 937-943, 2006.

H. A. David and H. N. Nagaraja. Order Statistics . Wiley, 2003.

M. G. Dekimpe and D. M. Hanssens. The persistence of marketing effects on sales. Marketing science , 14(1):1-21, 1995.

B. U. Devi, D. Sundar, and P. Alli. An effective time series analysis for stock trend prediction using arima model for nifty midcap-50. International Journal of Data Mining & Knowledge Management Process , 3(1):65, 2013.

R. Durrett. Probability: Theory and Examples . Cambridge University Press, 5th edition, 2019.

J. Fattah, L. Ezzine, Z. Aman, H. E. Moussami, and A. Lachhab. Forecasting of demand using arima model. International Journal of Engineering Business Management , 10, 2018.

T. C. Gard. Introduction to Stochastic Differential Equations . Marcel Dekker, 1988.

C. W. Gardiner. Handbook of stochastic methods: for the natural and social sciences . Springer, 2009.

A. Garivier, T. Lattimore, and E. Kaufmann. On explore-then-commit strategies. In Advances in Neural Information Processing Systems , volume 29, 2016.

A. Garivier and E. Moulines. On upper-confidence bound policies for switching bandit problems. In Algorithmic Learning Theory , pages 174-188. Springer Berlin Heidelberg, 2011.

J. Gittins and D. Jones. A dynamic allocation index for the sequential design of experiments. In Progress in Statistics , pages 241-266. North-Holland, 1974.

S. Grünewälder and A. Khaleghi. Approximations of the restless bandit problem. Journal of Machine Learning Research , 20(14):1-37, 2019.

S. Guha and K. Munagala. Approximation algorithms for partial-information based stochastic control with markovian rewards. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07) , pages 483-493, 2007.

W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American statistical association , 58(301):13-30, 1963.

N. B. Keskin and A. Zeevi. Chasing demand: Learning and earning in a changing environment. Mathematics of Operations Research , 42(2):277-307, 2017.

R. Kleinberg. Nearly tight bounds for the continuum-armed bandit problem. In Proceedings of the 17th International Conference on Neural Information Processing Systems , page 697-704, 2004.

R. Kleinberg and N. Immorlica. Recharging bandits. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS) , pages 309-319. IEEE, 2018.

R. Kleinberg and T. Leighton. The value of knowing a demand curve: Bounds on regret for online posted-price auctions. In Proceedings of the 44th Annual IEEE Symposium on Foundations of Computer Science (FOCS) , pages 594-605, 2003.

P. Kloeden and E. Platen. Numerical Solution of Stochastic Differential Equations . Springer, 1992.

T. L. Lai, H. Robbins, et al. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22, 1985.

C. Lim and M. MacAleer. Time series forecasts of international tourism demand for australia. Technical report, ISER Discussion Paper, 2001.

F. Liu, J. Lee, and N. B. Shroff. A change-detection based framework for piecewise-stationary multi-armed bandit problem. In AAAI , pages 3651-3658, 2018.

K. Liu and Q. Zhao. Indexability of restless bandit problems and optimality of whittle index for dynamic multichannel access. IEEE Trans. Inf. Theor. , 56(11):5547-5567, nov 2010.

Y. Liu, B. Van Roy, and K. Xu. Nonstationary bandit learning via predictive sampling. In International Conference on Artificial Intelligence and Statistics , pages 6215-6244. PMLR, 2023.

A. Mate, J. Killian, H. Xu, A. Perrault, and M. Tambe. Collapsing bandits and their application to public health intervention. In Advances in Neural Information Processing Systems , volume 33, pages 15639-15650. Curran Associates, Inc., 2020.

R. Ortner, D. Ryabko, P. Auer, and R. Munos. Regret bounds for restless markov bandits. In Proceedings of the 23rd International Conference on Algorithmic Learning Theory , page 214-228, 2012.

S. Pandey, D. Agarwal, D. Chakrabarti, and V. Josifovski. Bandits for taxonomies: A modelbased approach. In SIAM International Conference on Data Mining , pages 216-227, 2007.

C. H. Papadimitriou and J. N. Tsitsiklis. The complexity of optimal queuing network control. Mathematics of Operations Research , 24(2):293-305, 1999.

G. A. Pavliotis. Stochastic processes and applications . Springer-Verlag New York, 2016.

D. Pollard. Convergence of stochastic processes . Berlin: Springer, 1984.

H. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society , 58(5):527 - 535, 1952.

R. H. Shumway and D. S. Stoffer. Time Series Analysis and Its Applications: With R Examples . Springer, 4th edition, 2017.

A. Slivkins and E. Upfal. Adapting to a changing environment: the brownian restless bandits. In 21st Conference on Learning Theory (COLT) , pages 343-354, 2008.

R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction . The MIT Press, second edition, 2018.

C. Tekin and M. Liu. Online learning of rested and restless bandits. IEEE Transactions on Information Theory , 58(8):5588-5611, 2012.

W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25:285 - 294, 1933.

F. Trovo, S. Paladino, M. Restelli, and N. Gatti. Sliding-window thompson sampling for non-stationary settings. Journal of Artificial Intelligence Research , 68:311-364, 2020.

P. Whittle. Restless bandits: activity allocation in a changing world. Journal of Applied Probability , 25A:287-298, 1988.

Appendices for

Non-Stationary Bandits with Auto-Regressive Temporal Dependency

A Numerical Studies on Synthetic Data

Set-up. We consider a non-stationary bandit problem with k arms and T = 10 , 000 rounds. To make the setting more general, we let the expected reward r i ( t ) of each arm i ∈ [ k ] follow an independent AR-1 process with heterogeneous AR parameter α i , stochastic rate of change σ i , and truncating boundary [ -1 , 1] . We consider two different regimes of α i 's: (i) Small α i 's : the AR-1 parameters { α i } k i =1 are drawn from a dirichlet distribution with rescaling such that for all i ∈ [ k ] , E [ α i ] = 0 . 4 , which is less than the threshold value ¯ α = 0 . 5 that we defined in Section 5. (ii) Large α i 's : the AR-1 parameters { α i } k i =1 are chosen from a dirichlet distribution with rescaling such that E [ α i ] = 0 . 9 for all i ∈ [ k ] . The concentration parameters of the dirichlet distributions are all set to be 5 . For each regime of α i 's (small v.s. large), we consider 100 instances, where each instance corresponds to AR parameters ( α 1 , . . . , α k ) generated as described above; and stochastic rates of change ( σ 1 , . . . , σ k ) generated from i.i.d. uniform distributions in (0 , 0 . 5) . The initial reward r i (0) is drawn independently from the steady state distribution discussed in Appendix D.

Performance comparison. We implement AR2 and compare it against a number of benchmarks designed for stationary and non-stationary bandits respectively. (See Appendix B.1 for the implementation details, where we generalize AR2 to the setting with heterogeneous α i 's and σ i 's.) All parameters used in the following algorithms are tuned.

The benchmark algorithms designed for stationary settings are: (i) 'explore-then-commit' (ETC) policy [24] that first selects each arm for m rounds, and commits to the arm with the highest average pay-off in the remaining T -mk rounds; (ii) UCB algorithm [6], which achieves sublinear regret under static settings; (iii) ϵ -greedy algorithm [49] with probability of exploration ϵ = 0 . 1 , which acts greedily with probability (1 -ϵ ) and explores a random arm otherwise. (iv) Exp3 algorithm [7], which is designed for the adversarial setting and achieves sublinear regret when evaluated against the weak static benchmark.

Further, we also consider several benchmark algorithms designed for non-stationary settings: (i) Rexp3 algorithm [10], which modifies the Exp3 algorithm for the non-stationary setting by introducing a restarting mechanism, and assumes that the total variation of expected rewards is bounded by a variation budget V T ; (ii) a modified UCB algorithm (mod-UCB), which modifies the upper confidence bound based on the structure of the AR-1 process; see Appendix C for a description and discussion of the algorithm; (iii) the sliding-window UCB (SW-UCB) algorithm [25] and the sliding-window Thompson Sampling (SW-TS) algorithm [52], both of which adopt sliding-window approaches; (iv) the predictive sampling (PS) algorithm for AR-1 bandits recently introduced by [39], which is shown to outperform the Thompson sampling algorithm in non-stationary environments.

Results. In Table 2, we evaluate and compare the performance of AR2 against the benchmark algorithms in instances with small α i 's ( E [ α i ] = 0 . 4) as well as large α i 's ( E [ α i ] = 0 . 9) . We also vary the number of arms k ∈ { 2 , 10 , 20 } . To keep our results consistent across instances, we evaluate each algorithm A using the normalized regret 7 : ∑ T t =1 REG A ( t ) / ∑ T t =1 r ⋆ ( t ) , which normalizes the total regret using the reward of optimal in hindsight dynamic policy.

From Table 2, we see that AR2 clearly outperforms the benchmarks in almost all choices of k and regimes of α i 's. The algorithms designed for stationary settings (ETC, UCB, Exp3) perform poorly, as the AR setting includes non-stationarity that differs substantially from the stochastic or adversarial settings they were intended for. Similarly, the algorithms designed for non-stationary setting with limited amount of changes (RExp3, SW-UCB, SW-TS) do not perform well since they are designed to tackle environments either with abrupt changes, or with sublinear amount of changes, and are not suited for the fast changes in the AR environment. Finally, when compared against benchmarks that take into account the AR structure of rewards ( ϵ -greedy, modified UCB, PS), AR2 also displays superior performance. Here, the PS algorithm indeed dominates TS-based approaches

in non-stationary environments, as suggested by [39], but AR2 remains competent in all kinds of AR-based setups; further, [39] also suggests that deriving the conditional probability distribution in the predictive sampling procedure and implementing it can be complicated in general, and hence it is unclear how one can adapt the PS algorithm for more complicated ARp processes as we did in our case study (see Section 7 and Appendix B.2). Among all stationary benchmarks, ϵ -greedy is the most competitive; while mod-UCB stands out as the most competitive within the non-stationary benchmarks. However, there are instances where AR2 can contribute to a decrease in regret as high as 50% when compared to ϵ -greedy, and a 15 . 5% decrease in regret when compared to modified UCB. All of the aforementioned results attest to the efficacy of AR2 in rapidly changing environments governed by the AR process.

B Implementation Details and Extensions of Algorithm AR2

B.1 Implementation of AR2

To apply AR2 to the more general setting described in Appendix A, where the expected reward of each arm i undergoes an AR-1 process with heterogeneous AR parameter α i and stochastic rate of change σ i , we make a few straightforward updates to AR2 : (i) when maintaining estimated reward ̂ r i ( t ) of arm i , we use its AR parameter α i ; (ii) we assume that an arm i = i sup ( t ) gets triggered if at round t ,

̸

Empirically, we make two more minor changes to AR2 in its numerical implementation. (i) The definition of the superior arm: instead of considering one of the two most recently pulled arms, we consider all arms and choose the one with the highest estimated reward at round t to be the superior arm at round t , i.e., i sup ( t ) = arg max i ∈ [ k ] ̂ r i ( t ) . (ii) The triggered arm to be played: in the exploration round t (when t is odd and T ̸ = ∅ ), we consider the upper confidence bound of the expected reward of each arm i , defined as:

and pull the triggered arm with the highest upper confidence bound, i.e. I t = arg max j ∈T UCB j ( t ) . Note that the arms in the triggered set T differ in terms of both triggering time and estimated rewards. Previously, our criteria for selecting the triggered arm to play only takes into account one aspect of such heterogeneity, i.e., the triggering time. The new criteria allows us to also consider the estimated reward of each arm, which is another indicator of the arm's potential.

B.2 Extension of AR2 to ARp Process with Trend

In this section, we provide an extension of AR2 (Algorithm 1), which would apply to non-stationary bandits where the rewards follow general ARp processes with trends for any p ≥ 1 . This algorithm is applied when we perform our case study on tourism demand prediction in Section 7, and is shown to adapt well to the more complex, rapidly evolving environment.

Let us assume that for each arm i ∈ [ k ] , the reward distribution following an ARp process, potentially with a trend component. That is, conditioning on past history during rounds t -1 , . . . , 1 , the expected reward of arm i at round t is

where α i, 0 is the trend component and ( α i, 1 , . . . , α i,p ) is the set of AR parameters associated with arm i .

The details of the extension of AR2 is presented in Algorithm 2, which we called AR2-p . At a high level, the main techniques used in AR2-p are very similar to those used in AR2 . As AR2-p is specifically designed to address practical scenarios where real-world dynamics are represented by more complex time series, we have incorporated the implementation changes discussed in Appendix B.1 to enhance its practical applicability.

At each round t of Algorithm 2, we first determine the superior arm i sup ( t ) , which is the arm with the highest expected reward. We then decide if there are any arms that we wish to trigger, depending on whether some untriggered arm has an upper confidence bound that exceeds the expected reward of the superior arm ̂ r sup ( t ) . We then alternate between exploration and exploitation: (i) during exploitation rounds, we pull the superior arm; (ii) during exploration rounds, we pull the arm with the highest upper confidence bound out of the set of triggered arms , denoted as T . Finally, after we pull arm I t at each round, we update the estimate for the state of each arm i ∈ [ k ] .

The main difference between AR2-p and AR2 for AR-1 process lies in how we maintain estimates for each arm. To account for the more complicated structure of an ARp process with trends, we need to maintain two estimates for each arm-an estimate for the expected reward ̂ r i ( t ) , and an estimate for the error bound ̂ E i ( t ) , defined as follows:

Here, the estimate for error bound ̂ E i ( t ) essentially captures how much error is contained in our estimate for arm i at round t , i.e., ̂ r i ( t ) . Since we estimate ̂ r i ( t ) based on p past estimates ̂ r i ( t -1) , . . . , ̂ r i ( t -p ) , the error associated with each of these terms, i.e., ̂ E i ( t -1) , . . . , ̂ E i ( t -p ) , would contribute the total error for arm i at round t , i.e., ̂ E i ( t ) .

We would then use the error estimate ̂ E i ( t ) in determining our triggering condition. We trigger an arm i / ∈ T ∪ { i sup ( t ) } at round t if

The error bound estimates would also be useful when we determine which arm to pull out of all the triggering arms. To do that, we form the upper confidence bound for each arm

and pick the one with the highest upper confidence bound: I t = arg max i ∈T UCB i ( t ) .

C Description of Modified UCB Algorithm

In our numerical studies (Appendix A), we have considered the modified UCB algorithm as one of the non-stationary benchmarks. We now describe this algorithm, which borrows idea from the UCB algorithm [6] for stochastic MAB, and modifies the expression of the upper confidence bound based on the AR temporal structure.

Algorithm 2 Alternating and Restarting algorithm for non-stationary bandits with ARp reward structure ( AR2-p )

Input: AR parameters { α i,j } i ∈ [ k ] j ∈ [ p ] , trend components { α i, 0 } i ∈ [ k ] , stochastic rate of change { σ i } i ∈ [ k ] , epoch size ∆ ep, parameter c .

Set the epoch index s = 1 .

Repeat while s ≤ ⌈ T/ ∆ ep ⌉ :

(a) Initialization:

Set t 0 = ( s -1)∆ ep. Set the initial triggered set T = ∅ .

For each arm i ∈ [ k ] , set estimates for expected rewards ̂ r i ( t ) = 0 and error bound estimates ̂ E i ( t ) = ∞ for t ≤ 0 .

(b) For t = t 0 +1 , . . . , min { t 0 +∆ ep , T }

If t -t 0 ≤ p · k , pull arm i = ⌊ ( t -t 0 -1) /p ⌋ + 1.

Else,

-Update the identity of the superior arm and its estimated reward

-Trigger arms with potential: For i / ∈ T ∪ { i sup ( t ) } , trigger arm i if

If triggered, add i to T .

-Alternate between exploration and exploitation:

̸

If t is odd and T = ∅ , pull a triggered arm with the highest upper confidence bound estimate:

I

t

i

∈T

√

i

̂

r

(

t

) +

cσ

i

̂

E

i

(

t

)

.

Otherwise, pull the superior arm I t = i sup ( t ) .

Receive a reward R I t ( t ) . Update ̂ r I t ( t ) = R I t ( t ) and ̂ E I t ( t ) = 0 .

Maintain Estimates: For each arm i ∈ [ k ] , set estimates for expected rewards and error bounds

(c) Set s = s +1 .

Algorithm 3 A modified UCB algorithm for non-stationary AR bandits (mod-UCB)

Input: AR Parameters ( α 1 , . . . , α k ) , stochastic rates of change ( σ 1 , . . . , σ k ) , parameter δ .

Initialization: pull each arm i ∈ [ k ] at round i . Set estimates ̂ r i ( k +1) = α k -i · B ( αR i ( i )) and τ i = i .

While t ≤ T :

(a) For each arm i ∈ [ k ] , compute its upper confidence bound (UCB):

(b) Pull the arm with the highest UCB: I t = arg max i ∈ [ k ] UCB i ( t ) . Receive a reward R I t ( t ) , and set τ I t = t .

̸

(c) Maintain Estimates: Set ̂ r I t ( t +1) = B ( αR I t ( t )) and set ̂ r i ( t +1) = α ̂ r i ( t ) for i = I t .

The details of modified UCB is presented in Algorithm 3. Similar to the original UCB algorithm, we consider an upper confidence bound of the expected reward of each arm i at round t , denoted as UCB i ( t ) , which consists of two terms. The first term ̂ r i ( t ) is the estimated reward of arm i based on past observations; the second term is the size of the one-sided confidence interval for the expected reward r i ( t ) . The confidence interval is constructed based on Hoeffding's Inequality (see Lemma J.3), and is designed such that the true expected reward r i ( t ) will fall into the confidence interval with probability at least 1 -δ , where δ is an input parameter. To derive the expression of the upper confidence bound in the AR setting, consider a non-stationary MAB problem, in which the expected reward of arm i follows an AR-1 process with parameter α i ∈ (0 , 1) , stochastic rate of change σ i ∈ (0 , 1) , and truncating boundary [ -, ] . (See Section 2 for details about the set-up.) The estimated reward of arm i at round t , ̂ r i ( t ) , can be maintained in the same way as done in Algorithm 1.

// Pull each arm consecutively for p rounds.

The modified UCB algorithm always selects the arm with the highest upper confidence bound. To construct the confidence bound, we first recall from Section 6 that we define a new noise term ˜ ϵ i ( t ) (see (5)) that incorporates the influence of the truncating boundary, and the new noise term satisfies the recursive relationship: r i ( t +1) = αr i ( t ) + α ˜ ϵ i ( t ) . Suppose that at round t , we last observed the reward of arm i at round τ i ( t ) . Using the recursive relationship, we have:

where the last equality follows from the way we maintain estimates. By Lemma H.1, we have that for any s > 0 ,

where ϵ i ( t ′ ) ∼ N (0 , σ i ) is the independent Gaussian noise applied to arm i at round t ′ . By Hoeffding's Inequality (Lemma J.3), for fixed δ > 0 , we also have the following concentration result:

Equations (8), (9) and (10) together establish a confidence bound centered around ̂ r i ( t ) , such that the true expected reward r i ( t ) falls into the confidence bound with probability at least 1 -δ . In Algorithm 3, we thus define the upper confidence bound of arm i at round t as:

and we have P [ r i ( t ) > UCB i ( t )] ≤ 1 -δ when implementing the modified UCB algorithm for numerical experiments in Appendix A.

Finally, we remark that our algorithm AR2 and the modified UCB algorithm share some similarities. Both algorithms incorporate our knowledge of the AR temporal structure into their design, which makes them more competitive against the rest of the benchmarks. The triggering criteria of AR2 essentially constructs a confidence interval around the estimated reward ̂ r i ( t ) of arm i at round t , and triggers arm i if the upper confidence bound of arm i exceeds the estimated reward of the superior arm. On the other hand, AR2 also differs from the modified UCB in two main aspects. (i) By alternating between exploration and exploitation rounds, AR2 avoids potential over-exploration. This is especially common when α i 's are close to one, and any arm that has not been explored for a few rounds would have a very high confidence bound. In that case, modified UCB is likely to shift in between arms very often, which leads to too little exploitation. (ii) As discussed in Section 4, unlike the modified UCB algorithm, AR2 also deals with the tradeoff of 'remembering' and 'forgetting' via the restarting mechanism, which is especially important in a fast changing environment such as the AR setting.

D Steady State Distribution of Rewards

In this section, we characterize the steady state distribution of the expected reward r i ( t ) . Lemma D.1 shows that the steady state distribution depends on two probability masses at the boundaries and a probability density function between the boundaries.

Lemma D.1 (Probability density of steady state distribution) . Consider the steady state distribution of r i ( t ) , which follows the AR process with truncating boundaries defined in Section 2. Suppose that at the steady state distribution, we have two probability masses at the boundaries, i.e., P [ r i ( t ) = ] = P [ r i ( t ) = -] = p , where p ∈ (0 , 1 / 2) depends on α and σ . Then, the steady state distribution between the boundaries can be characterized by the following probability density function (PDF), denoted by f p : ( -, ) → ❘ + .

where C ( α, σ ) = √ 1 -α √ πασ erf ( √ 1 -α ασ ) is a normalization factor and erf( z ) = 2 √ π ∫ z 0 e -s 2 ds is the error function.

Given Lemma D.1, we are now ready to define the steady state distribution of r i ( t ) :

Definition D.2 (Steady state distribution of rewards) . Let r be a random variable with range [ -, ] whose probability distribution characterizes the steady state distribution of r i ( t ) . It has probability masses on the boundaries, i.e., P [ r = ] = P [ r = -] = p , and probability density function f p on ( -, ) as defined in (11) , that satisfies the following conditions:

Note that the probability masses on the boundaries result from the truncating boundary condition, and by symmetry, the masses on upper and lower boundaries are the same. If one adopts a reflecting boundary condition, we would have p = 0 ; if one adopts an absorbing boundary condition, we would have p = 1 / 2 (see [44] for definitions of the aforementioned boundary conditions).

We remark that for any fixed σ ∈ (0 , 1) , when α → 1 , the probability density function f p ( x ) in the steady state distribution of r i ( t ) can be well approximated by a constant function. In Figure 3, we plot the PDF of the steady state distributions for α ∈ { 0 . 3 , 0 . 6 , 0 . 9 } (with σ = 0 . 8 and = 1 ), and numerically compute the value of p (see legends in Figure 3) 8 . It can be seen that for larger α , the distribution within the boundaries becomes almost 'uniform'. In the other extreme case, when α → 0 , it can be seen both from Figure 3 and from the definition of f p ( x ) in Lemma D.1 that the distribution can be approximated by a normal distribution.

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

x

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

value

=0.3,

p

=3×10 5

=0.6,

p

=0.04

=0.9,

p

=0.16

Proof of Lemma D.1: We first note that the continuous-time analogue of an AR-1 process is a Ornstein-Uhlenbeck (OU) process [22], defined as the following stochastic differential equation (SDE):

where κ, θ, ζ are constant parameters 9 and W t denotes the Wiener process 10 . If we perform EulerMaryuama discretization (see [34]) on the above OU process at ∆ t, 2∆ t, . . . , we get

where ϵ ′ t is a standard normal noise. Now, recall from Section 2 that the evolution of expected rewards evolve as an independent AR-1 process:

where ϵ i ( t ) ∼ N (0 , σ ) . Let r t ≜ r i ( t ) . By coefficients in (13) and (14), the two equations are equivalent if we let κ = 1 -α, θ = 0 , ζ = ασ, ∆ t = 1 . By plugging the values of coefficients into (12), the continuous-time analogue of (14) can be described by the following SDE:

Let f p ( x ) be the PDF for the steady state distribution of r i ( t ) on ( , ) , given that the probability mass on either side of the boundary is p . We can now solve for f p ( x ) by solving the following

ordinary differential equation (see [23, 44] for a discussion of forward and backward equations for SDEs):

for ˜ C that depends on α, σ . Since f p ( x ) is a probability density, we can solve for ˜ C ( α, σ ) by requiring ∫ -f p ( x ) dx = 1 -2 p , given that the total probability within the boundaries is 1 -2 p . This then gives √

where C ( α, σ ) is defined in the statement of Lemma D.1. ■

E Additional Illustrations

E.1 Illustration of g ( k, α, σ ) in Regret Lower Bound

Recall that in Section 3, we characterize our regret lower bound to be Ω( g ( k, α, σ ) ασ ) , where g ( k, α, σ ) is the probability that two best arms are within ασ distance of each other at any given round under the steady state distribution, with its closed form presented in Equation (16) in Appendix F.

To better understand how g ( k, α, σ ) evolves in terms of our parameters, in Figure 4 we plot the evolution of g ( k, α, σ ) in terms of σ in two different regimes: (i) a smallα regime when α = 0 . 4 , and (ii) a largeα regime when α = 0 . 9 . Note that here, we consider the two regimes separately because in Section 5, we will show that the largeα regime is inherently much more challenging than the smallα regime.

In Figure 4, we see that in the smallα regime ( α = 0 . 4 ), g ( k, α, σ ) is roughly constant as σ increases, while in the largeα regime ( α = 0 . 9 ), g ( k, α, σ ) is roughly linear in terms of σ . To understand why, recall from our discussion in Appendix D that when α → 0 , the steady state distribution approaches a normal distribution with standard deviation proportional to ασ , while when α → 1 , the steady state distribution resembles a uniform distribution. Hence, when α is small, we expect g ( k, α, σ ) to be roughly constant in terms of σ , while when α is close to one, we expect g ( k, α, σ ) to be roughly linearly in terms of σ . Here, our observations in Figure 4 concur with our discussion in Appendix D.

0.1

0.2

0.3

0.4

0.5

0.5

0.6

0.7

0.8

g

(

k

,

,

)

=0.4

=0.9

E.2 Illustration of Upper/Lower Bounds in the Smallα Regime

In Figure 5, we illustrate the evolution of our upper and lower bounds in the smallα regime (i.e., α ∈ (0 , ¯ α ) ). Here, we take α = 0 . 4 , and similar plots can be obtained for different values of α < ¯ α . Recall from Section 5 that this regime is considered less challenging due to our result in Theorem 5.1, which states that any algorithm can attain near-optimal performance when α ∈ (0 , ¯ α ) . Here, Figure 5 reveals that our upper/lower bounds indeed have the same trend of increase in terms of σ in the smallα regime, which complements our result in Figure 1 (that shows our upper/lower bounds match in terms of α in the smallα regime).

which gives

0.1

0.2

0.3

0.4

0.5

0.050

0.075

0.100

0.125

0.150

value

C

log(1/

) + log

k

1

2

g

(

k

,

,

)

E.3 Illustrations of Upper/Lower Bounds and Per-Round Regrets of AR2

To further highlight the significance of Theorem 5.2 and how well it characterizes the performance of AR2 , in Figure 6, we plot the evolution of the upper bound in Theorem 5.2, the lower bound O ( kα 2 σ 2 ) for α close to one, and the per-round regret attained by AR2 , at different values of parameters α and σ respectively. In Figure 6a, the per-round regret of AR2 increases at roughly the same rate as the upper and lower bounds, and stays close to the lower bound, as α increases. In Figure 6b, the upper and lower bounds follow the same trend of increase with respect to σ . We note that as σ increases, the upper bound loosens up as the per-round regret of AR2 experiences smaller growth and approaches the worst-case lower bound. This especially highlights the strength of AR2 in adapting to a rapidly changing environment.

0.86

0.88

0.90

0.92

0.94

10

1

10

0

value

upper bound

lower bound

per-round regret

0.1

0.2

0.3

0.4

0.5

10

2

10

1

10

0

10

1

value

upper bound

lower bound

per-round regret

(b) Evolution of the upper/lower bounds and per-round regret of AR2 with varying σ . We take σ ∈ [0 . 1 , 0 . 5] and α = 0 . 9 .

F Missing Proof of Section 3

Before we show the proof for Theorem 3.1, we first present Lemma F.1 [17] which characterizes the joint distribution for order statistics.

Lemma F.1 (The joint distribution of order statistics of a continuous distribution) . Let X (1) ≤ X (2) ≤ · · · ≤ X ( k ) be the order statistics of k i.i.d. samples drawn from some continuous distribution with probability density function (PDF) f ( x ) and cumulative density function (CDF) F ( x ) . The joint PDF of X ( j ) , X ( j ) for any 1 ≤ i < j ≤ k is

In particular, if we take i = k -1 and j = k , Lemma F.1 suggests that

which will be used in the following proof of Theorem 3.1.

Proof of Theorem 3.1: The proof proceeds in two steps. First, we compute the probability that the expected rewards of two best arms are close to each other at any given round t . Then, conditioning on this event, we show that at the next round t +1 , any algorithm incurs Ω( ασ ) regret with constant probability.

Step 1. We first lower bound the probability that the expected rewards of the two best arms are within ασ of each other at round t . Recall from Appendix D that r i ( t ) follows the steady state distribution with probability mass p on the boundary and PDF f p ( x ) defined in (11) within the boundaries. Let F ( x ) be the CDF of the the steady state distribution of r i ( t ) . We can compute the probability that the expected rewards of the two best arms are within ασ by breaking the event into the following cases:

Both arms are at the upper boundary. This will happen with probability ∑ k j =2 ( k j ) p j (1 -p ) k -j , where j denote the number of arms are at the upper boundary, i.e., have expected reward .

The best arm is at the upper boundary, and the second best arm is within the boundaries. This will happen with the following probability: k ( k -1) · p · ∫ -ασ F ( x ) k -2 f p ( x ) dx

Both the best arm and the second best arm are between the boundaries. This will happen with the following probability: k ( k -1) ∫ ασ 0 ∫ -F ( x ) k -2 f p ( x ) f p ( x + z ) dxdz . To obtain this probability, we used the joint PDF stated in Lemma F.1.

The best arm is within the boundaries, and the second best arm is at the lower boundary. This will happen with the following probability k · ∫ ασ --f p ( x ) dx · p k -1 .

Both arms are at the lower boundary. This will happen with probability p k .

We define g ( k, α, σ ) as the probability that the expected rewards of the two best arms are within ασ of each other at round t , which is the sum of the terms above:

We now make a brief remark about the order of g ( k, α, σ ) as α approaches one. First note that among the aforementioned cases, the first three cases would happen with probability at least 1 / 2 given that the steady state distribution of expected reward is symmetric and the expected rewards of the two best arms are skewed towards the upper boundary. Conditioning on the third case, i.e., both the best and the second best arms are within the boundaries, as α approaches one, the conditional PDF can be approximated as a uniform function (see Remark G.1 in Appendix G for a related discussion), and hence the conditional probability that two best arms are within ασ of each other are of order Ω( kασ ) . Further, conditioning on the first and the second cases, i.e., the best arm is at the upper boundary, the conditional probability that two best arms are within ασ of each other are lower bounded by Ω( kασ ) . We thus have g ( k, α, σ ) ≥ Ω( kασ ) as α approaches one.

Step 2. Let A be any algorithm that selects arm A ( t ) at round t . In this step, we show that conditioning on the event that the two best arms are within ασ of each other at round t , with constant probability, A will make a mistake at round t +1 and incur regret of order O ( ασ ) .

Without loss of generality, let us suppose r 1 ( t ) ≥ · · · ≥ r k ( t ) , and that algorithm A ( t ) selects arm 1 at round t (if A ( t ) selects a different arm at round t , the following proof works similarly). We first assume that r 1 ( t ) ∈ [ -+3 σ, -3 σ ] . Let Φ( x ) be the CDF of standard normal distribution. Since ϵ i ( t ) i.i.d. ∼ N (0 , σ ) , when A pulls arm 1 at round t , with constant probability 2Φ( 1 2 ) -1 , we have | ϵ 1 ( t ) | ≤ σ 2 . Let A ( t +1) be the arm that A chooses to pull at round t +1 . At round t +1 , one of the following scenarios will take place:

(i) If A ( t +1) = 1 , then if ϵ ( t ) ∈ [2 σ, 3 σ ] , it will incur regret

The event ϵ ( t ) ∈ [2 σ, 3 σ ] happens with constant probability Φ(3) -Φ(2) .

̸

2 (ii) If A ( t +1) = j for any j = 1 at t +1 , then if ϵ j ( t ) ∈ [ -2 σ, -σ ] , it will incur regret

The event ϵ j ( t ) ∈ [ -2 σ, -σ ] happens with constant probability Φ(2) -Φ(1) .

̸

In all of the above cases, we show that A will incur regret at least ασ 2 with constant probability, conditioning on the event that r (1) -r (2) ≤ ασ . If r 1 ( t ) is close to either side of the boundary, a similar analysis can establish the same result. To see that, if r 1 ∈ [ -3 σ, ] and A pulls arm 1 at round t , with constant probability we have ϵ 1 ( t ) ∈ [ -2 σ, -3 σ ] . Then, we have one of the following: (i) A ( t +1) = 1 . If | ϵ 2 ( t ) | ≤ σ 2 , arm 1 is sub-optimal and incurs regret by at least ασ 2 ; (ii) A ( t +1) = j for some j = 1 . If ϵ j ( t ) ∈ [ -3 σ, -4 σ ] , arm j is sub-optimal and incurs regret by at least ασ . This, again, shows that A incurs regret at least Ω( ασ ) at round t +1 with constant probability. The proof follows similarly when r 1 ∈ [ -, -+3 σ ] .

Combining step 1 and 2, we conclude that the per-round regret of any algorithm A is at least Ω( g ( k, α, σ ) ασ ) , which concludes the proof. ■

G Missing Proof of Section 5

Proof of Theorem 5.1: At any given round t , recall from Appendix D that r i ( t ) follows a steady state distribution as defined in Definition D.2. Let us define r ′ i ( t ) i.i.d. ∼ N (0 , ασ √ 1 -α 2 ) . The variable r ′ i ( t ) has the following PDF:

for all x ∈ R . Note that for all 0 < s ≤ , we have that P [ | r i ( t ) | > s ] ≤ P [ | r ′ i ( t ) | > s ] , which can be obtained by directly computing both probabilities. Intuitively, one can think of the normal distribution above as the steady state distribution of r i ( t ) that follows the same AR-1 process r i ( t +1) = α ( r i ( t )+ ϵ i ( t )) , without the truncating boundaries at [ -, ] . As a result of the truncating boundaries, the steady state distribution D (characterized in Appendix D) is more 'concentrated' than the normal distribution N (0 , ασ √ 1 -α 2 ) . Now, using the concentration bound for normal random variable (see Lemma J.2), we have

for every arm i ∈ [ k ] . Now let δ = 2exp( -c 2 2 ) . By the union bound, with probability at least 1 -kδ , we would have | r i ( t ) | ≤ c ασ √ 1 -α 2 for all i ∈ [ k ] . And hence ∆ r i ( t ) ≤ 2 c ασ √ 1 -α 2 for all i ∈ [ k ] . Let us denote the above event as E . If we set c = √ 2 log(1 /ασ ) + 2 log(2 k ) , we would have δ = ασ k , and hence P [ E ] ≥ 1 -kδ = 1 -ασ .

Let A be any non-anticipating algorithm that selects arm A ( t ) at round t . The expected regret A incurs at round t is

Given that the per-round regret is also bounded by 2 , this naturally gives

In Section 5, we choose ¯ α ≜ 0 . 5 as the threshold value such that when α ∈ [ ¯ α, 1) , the problem is more challenging and thereby of our interest. Given the threshold ¯ α , we further expand on the remark made in Appendix D, which will be useful in the proof of Lemma 6.4 in Appendix H.

Remark G.1 (Uniform Approximation) . For any fixed σ ∈ (0 , 1) , when α → 1 , the probability density function that characterizes steady state distribution of r i ( t ) within the boundaries can be well approximated by a constant function. This can be seen from

In particular, by Lemma D.1, for any α ∈ [ ¯ α, 1) and σ ≥ c ′ √ 1 -α , we can bound the PDF f p ( x ) that characterizes the steady state distribution of r i ( t ) for all x ∈ [ -, ] using constants:

where ρ -= (1 -2 p ) C (¯ α, c ′ √ 1 -¯ α ) exp ( -2 / ¯ α 2 c ′ 2 ) , ρ + = (1 -2 p ) C (¯ α, c ′ √ 1 -¯ α ) are both constants.

When α ∈ [ ¯ α, 1) and σ = o ( √ 1 -α ) , the steady state distribution of r i ( t ) can be well approximated by a normal distribution, and similar to what we show in Theorem 5.1, in this case even the naive algorithm performs well. We thus focus on the more challenging regime when α ∈ [ ¯ α, 1) and σ = Ω( √ 1 -α ) in the following analysis.

H Missing Proofs of Section 6

H.1 Step 2: Good event and its implications.

In Section 6, we define a new noise term in (5) that shows the influence of truncating boundary:

The new noise ˜ ϵ i ( t ) satisfies the recursive relationship: r i ( t +1) = αr i ( t ) + α ˜ ϵ i ( t ) . Recall that here, the boundary function is defined as B ( y ) = min { max { y, - } , } .

Before proceeding with the proof of Lemma 6.3, we first show an auxiliary lemma that establishes the relationship between ˜ ϵ i ( t ) and ϵ i ( t ) . In particular, Lemma H.1 shows that if the weighted sum of ϵ i ( t ) is small with high probability, the weighted sum of ˜ ϵ i ( t ) is also small with high probability.

Lemma H.1. For any s > 0 , i ∈ [ k ] , and rounds t 0 < t 1 , we have

Proof of Lemma H.1: For simplicity of notation, let us drop the notation for dependence on i . We first show that the following inequality holds for any r ( t 0 ) :

̸

To show (19), we first consider a fictional scenario in which r ( . ) only gets truncated at the first time it hits the boundary, and no longer gets truncated afterwards. Let t ′ denote the first round at which r ( . ) hits the boundary. With slight abuse of notation, we define corresponding fictional noises ϵ 1 ( t 0 ) , . . . , ϵ 1 ( t 1 -1) such that ϵ 1 ( t ) = ϵ ( t ) for all t = t ′ and ϵ 1 ( t ′ ) = ˜ ϵ ( t ′ ) .

Since the expected reward r ( . ) hits the boundary for the first time at t ′ , we have | αr ( t ′ )+ αϵ ( t ′ ) | > for some t 0 ≤ t ′ < t 1 . Recall that in the fictional scenario, ϵ 1 ( t ) = ϵ ( t ) for all t < t ′ . Let L = α t ′ -t 0 +1 r ( t 0 ) , E 1 = ∑ t ′ -1 t = t 0 α t ′ -t +1 ϵ ( t ) . Without loss of generality, let us assume that the expected reward hits the upper boundary , i.e.

Let d = L + E 1 + αϵ ( t ′ ) -> 0 be the distance by which the expected reward exceeds the boundary at t ′ . By (5) and the definition of the truncating boundary in Section 2, we have

From the discussion above, we have E 1 + αϵ ( t ′ ) = + d -L and E 1 + αϵ 1 ( t ′ ) = -L. Since in the fictional scenario, the expected reward only gets reflected at the first time of hitting the boundary, we also have ϵ 1 ( t ) = ϵ ( t ) for all t > t ′ . Let E 2 = ∑ t 1 -1 t = t ′ +1 α t 1 -t ϵ ( t ) denote the weighted sum of the noises after time of reflection t ′ . Now conditioning on the initial expected reward r ( t 0 ) , we have

Note that the inequality in (20) holds because E 2 is normally distributed (it is a weighted sum of normal noises). If we let q 1 = α t 1 -t ′ -1 ( -L ) and q 2 = α t 1 -t ′ -1 ( + d -L ) , we must have | q 1 | < | q 2 | . This, along with the normal distribution of E 2 , gives

Following the same procedure as above, we can define fictional scenarios in which r ( . ) only gets truncated when it hits the boundary for the first m times, for 1 ≤ m ≤ t 1 -t 0 . We can then define corresponding fictional noises ϵ m ( t 0 ) , . . . , ϵ m ( t 1 -1) and use the same proof ideas to show

for all 1 ≤ m ≤ t 1 -t 0 . Note that if m = t 1 -t 0 , the fictional rewards ϵ m = ˜ ϵ for all t ∈ [ t 0 , t 1 -1] . Hence,

Finally, since (19) holds for any initial expected reward r ( t 0 ) , taking expectation with respect to r ( t 0 ) gives (18), which concludes the proof. ■

Proof of Lemma 6.3: Let us first consider the sequence of noises ϵ i ( t ) (i.e., the noises before applying the truncating boundary). For any fixed i ∈ [ k ] , and for t ′ ∈ [ t -∆ ep , t +∆ ep ] , we have ϵ i ( t ′ ) i.i.d. ∼ N (0 , σ ) . Then, by Hoeffding's inequality (see Lemma J.3, where for 1 ≤ j ≤ t 1 -t 0 , we set X j = α j ϵ i ( t 1 -j ) , σ j = α j σ and s = c 0 σ t 1 -t 0 √ α 2 -α 2( t 1 -t 0 +1) 1 -α 2 ), for t -∆ ep ≤ t 0 < t 1 ≤ t +∆ ep , we have

where the last equality follows from c 0 = √ 4 log(1 /ασ ) + 4 log ∆ ep +2log(4 k ) . By Lemma H.1, we can use the above inequality to also obtain a concentration inequality for the weighted sum of the new noises ˜ ϵ i ( t ) , which incorporate the influence of truncating boundary:

Since we have k arms and the number of sub-intervals to consider in [ t -∆ ep , t +∆ ep ] is 2∆ ep (2∆ ep -1) 2 , we can apply union bounds to get

Having showed that good event G ( t ) happens with high probability, we now establish a number of important implications of the good event:

(i) In Lemma H.2, we show that the expected reward r i ( t ) satisfies a series of concentration inequalities. These inequalities guarantee that our estimated reward ̂ r i ( t ) does not deviate too much from the true expected reward r i ( t ) .

(ii) In Lemma H.3, we show that AR2 does not wait too long to play an arm with high expected reward.

(iii) In Lemma H.4, we show that the best expected reward r ⋆ ( t ) is not too far away from our estimate reward for the superior arm ̂ r sup ( t ) .

These implications will be critical for our proof of Lemma 6.4. Before we proceed, let us define the following:

̸

where the second equality follows from the geometric sum formula. The expression (21) will be used throughout the rest of the analysis. In particular, since c 1 = 24 c 0 , the triggering criteria becomes: at round t , an arm i = i sup ( t ) gets triggered if the following is satisfied:

Lemma H.2. Given G ( t ) , for any arm i ∈ [ k ] and any rounds t 0 , t 1 such that t -∆ ep ≤ t 0 < t 1 ≤ t +∆ ep, the expected rewards satisfy the following inequalities:

Proof of Lemma H.2 Given the AR-1 model and (5), we have the following recursive relationship

by repeating the above steps. Then by Lemma 6.3, we have that

which concludes the proof for (22).

As for (23), note that

and the other side of the inequality can be derived similarly.

The last inequality (24) easily follows from (22), (23), and the triangle inequality

This concludes the proof of Lemma H.2. ■

Lemma H.3. Suppose we apply AR2 to the non-stationary MAB problem with α ∈ [ ¯ α, 1) and k ≤ K ( α ) , given G ( t ) , the following holds for the best arm i at round t : t -τ i ( t ) ≤ 4 k -3 .

Proof of Lemma H.3 Suppose α ∈ [ ¯ α, 1) . Let arm i be the best arm at round t , i.e. i = arg max j r j ( t ) . We let τ = τ i ( t ) , and let τ trig ≥ τ be the triggering time of arm i after it gets pulled at round τ . First note that by design of AR2 , we must have t -τ trig ≤ 2( k -1) , since at any given round there are at most k -1 triggered arms. We would prove the statement of the lemma by showing the following inequality:

If (26) holds, we then have t -τ = ( τ trig -τ ) + ( t -τ trig ) ≤ 2( t -τ trig ) + 1 ≤ 4 k -3 .

Suppose by contradiction that (26) does not hold. By the triggering criteria, we have

Let j = i sup ( τ trig -1) as defined in (1). Hence we have ̂ r sup ( τ trig -1) = r j ( τ trig -1) or ̂ r sup ( τ trig -1) = αr j ( τ trig -2) . Let us assume ̂ r sup ( τ trig -1) = r j ( τ trig -1) and the proof follows similarly for the other case. Since we also have ̂ r i ( τ trig -1) = α τ trig -τ -2 B ( αX i ( τ )) = α τ trig -τ -2 r i ( τ +1) , (27) can be rewritten as

By (22) of Lemma H.2, we also have the following two inequalities:

Then by (29) and (30), we have

where the last inequality follows from (28). Since k ≤ K ( α ) , we have α t -τ trig +1 ≥ α 2 k -1 ≥ 1 8 , which yields

Since τ trig -τ ≥ t -τ trig +2 , we have F ( τ trig -1 -τ ) ≥ F ( t -τ trig +1) . This further yields

Note that if τ trig -τ > t -τ trig + 1 , then since t -τ = ( t -τ trig ) + ( τ trig -τ ) , we must have 2( τ trig -τ ) > t -τ +1 , and thus

Plugging this into (32) gives r j ( t ) -r i ( t ) > 0 , which contradicts that arm i is the best arm at t . Hence, we must have τ trig -τ ≤ t -τ trig +1 . By our previous argument, this then gives t -τ ≤ 4 k -3 . ■ Lemma H.4. Suppose we apply AR2 to the non-stationary MAB problem with α ∈ [ ¯ α, 1) and k ≤ K ( α ) , for any arm i ∈ [ k ] and round t , we have

where ̂ r sup ( t ) = ̂ r i sup ( t ) ( t ) and i sup ( t ) is defined in (1) .

Proof of Lemma H.4: Without loss of generality, suppose that t is even, and as a result I t = i sup ( t ) . The proof then proceeds in two steps. In step one, we establish a recursive inequality for the estimated reward of the superior arm. In step two, we show that the estimated reward of the superior arm is in fact close to the expected reward of the best arm.

Step 1: Establish a recursive inequality for ̂ r sup ( t ) . We first assume that i sup ( t ) = I t -2 and hence ̂ r sup ( t ) = α B ( αR i sup ( t ) ( t -2)) = αr i sup ( t ) ( t -1) . We also have

which then gives

by definition of the good event G ( t ) (set t 0 = t and t 1 = t +1 ). Since I t = i sup ( t ) , we additionally have

Combining (34) and (35) gives us the following inequality, which is the desired result.

On the other hand, if i sup ( t ) = I t -1 , we have r i sup ( t ) ( t ) = B ( αR i sup ( t ) ( t -1)) = ̂ r sup ( t ) , and combining this with (35) also gives

Overall, we show that ̂ r sup ( t +1) ≥ α ̂ r sup ( t ) -2 c 0 ασ , which completes the proof of the first step.

Step 2: Show that ̂ r sup ( t ) and r ⋆ ( t ) are close. For simplicity of notation, let j be the best arm at round t , so r ⋆ ( t ) = r j ( t ) . Let τ = τ j ( t ) . Recall that by Lemma H.3, we have that t -τ ≤ 4 k -3 = O ( k ) . First note that

We also note that

Combining (37) and (38) then gives r ⋆ ( t ) -̂ r sup ( t ) ≤ O ( c 0 σαk ) . The proof works in a similar fashion for the case when t is odd. ■

H.2 Step 3: Distributed regret analysis.

Recall from Definition 6.1 that the distributed regret of arm i at round t is defined as

Before proceeding with the proof of Lemma 6.4, we first establish the following two claims. In Claim H.5 and Claim H.6, we respectively bound the nominator and denominator of D i ( t ) , conditioning on the good event G ( t ) and the value of ∆ r i ( t ) .

Claim H.5. Suppose we apply AR2 to the non-stationary MAB problem with α ∈ [ ¯ α, 1) and k ≤ K ( α ) . Given G ( t ) , for any arm i ∈ [ k ] , let η ≜ ∆ r i ( t ) > 0 for time τ i ( t ) ≤ t < τ next i ( t ) , we have that the nominator of (39) satisfies: α 2( t -τ i ( t )) ∆ r i ( τ i ( t )) ≤ O ( η + c 0 σα √ k ) .

Proof of Claim H.5: For simplicity of notation, let τ = τ i ( t ) and τ next = τ next i ( t ) . Let τ trig be the first round at which arm i gets triggered after it gets pulled at round τ . We divide our proof into two cases.

Case 1: τ trig -τ ≤ t -τ trig . If τ trig -τ ≤ t -τ trig ≤ O ( k ) , we must have t -τ = O ( k ) . Then by (22) of Lemma H.2, we have

Case 2: τ trig -τ > t -τ trig . Consider time t such that τ ≤ t < τ next . Then, we have

(40)

Let t ′ = τ trig -1 denote the round right before the round arm i gets triggered. Then since at round t ′ the arm has not been triggered, we have

This then gives

By (22) of Lemma H.2 and Equation (42), we also have

We also have

since t -t ′ = t -τ trig +1 ≤ 2( k -1) + 1 = 2 k -1 . Now since k ≤ K ( α ) = 1 2 (log( 1 8 ) / log α +1) , we have α 2 k -1 ≥ 1 8 , and hence

Now by (22) of Lemma H.2, we also have

where the last inequality follows from (42) and t -t ′ ≤ t ′ -τ . By multiplying the last equation by α t -t ′ , and by applying Equation (45), we then have

Since t -τ ≥ 0 , we then have α 2( t -τ ) ∆ r i ( τ ) ≤ O ( η + c 0 ασ ) .

In summary of both cases, we have α 2( t -τ ) ∆ r i ( τ ) ≤ O ( η + c 0 σα √ k ) . ■

Claim H.6. Suppose we apply AR2 to the non-stationary MAB problem with α ∈ [ ¯ α, 1) and k ≤ K ( α ) . Given G ( t ) , for any arm i ∈ [ k ] , let η ≜ ∆ r i ( t ) > 0 for time τ ≤ t < τ next , the denominator of (3) satisfies

Proof of Claim H.6 The first equality follows from the geometric sum formula. Since 1+ α 2 + · · · + α 2(∆ τ -1) ≥ 1 , it suffices to show that it is also lower bounded by Ω( η 2 c 2 0 σ 2 α 2 k 2 ) . Let τ ≤ t < τ next and recall that η = ∆ r i ( t ) . We divide our proof into two cases.

Case 1: α t -τ ∆ r i ( τ ) ≤ η/ 2 . Then by (22) of Lemma H.2, we have

Since ∆ τ ≥ t -τ , we have 1 -α 2∆ τ 1 -α 2 ≥ 1 -α 2( t -τ ) 1 -α 2 ≥ Ω( η 2 c 2 0 σ 2 α 2 ) .

Case 2: α t -τ ∆ r i ( τ ) > η/ 2 . We would show that 1 -α 2∆ τ 1 -α 2 ≥ Ω( η 2 c 2 0 σ 2 α 2 k 2 ) . We have

Now consider τ trig , the round at which arm i gets triggered. By the triggering criteria, we have

Since we must have ∆ τ > τ trig -τ , we have

Having showed Claim H.5 and Claim H.6, we now formally prove Lemma 6.4.

Proof of Lemma 6.4: Fix arm i ∈ [ k ] and round t . For simplicity of notation, let τ = τ i ( t ) , τ next = τ next i ( t ) and ∆ τ = ∆ τ i ( t ) . Let us write E [ D i ( t ) ✶ G ( t ) ] = E [ D i ( t ) ✶ G ( t ) ✶ { ∆ r i ( t )=0 } ] + E [ D i ( t ) ✶ G ( t ) ✶ { ∆ r i ( t ) > 0 } ] , and in the following analysis, we will bound the two terms on the righthand side respectively.

First term (Arm i is optimal at round t ). We first rewrite the first term as

E [ D i ( t ) ✶ G ( t ) ✶ { ∆ r i ( t )=0 } ] = E [ D i ( t ) |G ( t ) ∩ { ∆ r i ( t ) = 0 } ] · P [ G ( t ) ∩ { ∆ r i ( t ) = 0 } ] . Note that if G ( t ) takes place and ∆ r i ( t ) = 0 , we have α t -τ ∆ r i ( τ ) = ∣ ∣ α t -τ ∆ r i ( τ ) -∆ r i ( t ) ∣ ∣ ≤ 2 F ( t -τ ) . (50)

by (24) in Lemma H.2. By Definition 6.1, we have

That is,

Now, let j be the optimal arm at time τ . If G ( t ) takes place and ∆ r i ( t ) = 0 , we have

by (24) in Lemma H.2. Hence we have

̸

Recall from the discussion in Appendix D and Remark G.1 that the steady state distribution of r i ( t ) can be reasonably approximated by a uniform distribution, and (17) states that the PDF of the steady state distribution f ( x ) satisfies ρ -≤ f ( x ) ≤ ρ + for all x ∈ [ -, ] , where ρ -, ρ + are constants. Now let r ′ 1 , . . . , r ′ k be k i.i.d. uniform ( -1 / 2 ρ + , 1 / 2 ρ + ) random variables. We have

̸

Combining (53) and (54) thus gives

Overall, by (52) and (55), for all t ≥ τ we have

Second term (Arm i is sub-optimal at round t ). Given G ( t ) and the value of η ≜ ∆ r i ( t ) > 0 , we have bounded the nominator and denominator of D i ( t ) in Claim H.5 and Claim H.6 respectively. For any time τ ≤ t < τ next , in Claim H.5 we have showed α 2( t -τ ) ∆ r i ( τ ) ≤ O ( η + c 0 σα √ k ) and in Claim H.6 we have showed 1 + α 2 + · · · + α 2(∆ τ -1) ≥ max { Ω( η 2 c 2 0 σ 2 α 2 k 2 ) , 1 } . These results then give √

Let ˜ C = c 0 ασ √ k , this then yields

This then further leads to

We can now bound the second term by taking expectation of the above expression with respect to η . Recall from (17) that the PDF of the steady state distribution of r i ( t ) satisfies ρ -≤ f ( x ) ≤ ρ + for all x ∈ [ -, ] , where ρ -and ρ + are constants. This then gives

and

which provides an upper bound for the second term.

Finally, summing (56), (57), and (58) gives E [ D i ( t ) ✶ G ( t ) ] ≤ O ( c 2 0 α 2 σ 2 k 2 log( c 0 ασ √ k )) , which concludes the proof of Lemma 6.4. ■

I Missing Proof of Section 8

Proof of Proposition 8.1: Before proceeding, we first make some notational changes that would simplify our proof. Since we pull arm i consecutively for T est rounds for some fixed arm i ∈ [ k ] , in the rest of this proof we omit the dependency of R i ( t ) on i and denote R t ≜ R i ( t ) .

Let us first define the loss function L ( α ) stated in the optimization problem

which is the negative of the log-likelihood function:

where Φ( . ) is the CDF of standard normal distribution Note that the three terms in the loss function respectively represent the log likelihood of the observed reward at the upper boundary, at the lower boundary and in between the boundaries.

Recall that ̂ α is the solution to the optimization problem in (59). We can apply the second-order Taylor's Theorem to obtain the following:

for some ˜ α between true AR parameter α and estimated AR parameter ̂ α . In the rest of this proof, we will provide a bound for | ̂ α -α | via bounding the first and second derivatives of the loss function L ( . ) .

We thus have

where log ′ h ( y ) denote the derivative of function log h ( y ) with respect to y . Let

Note that L ′ ( α ) = 1 T est ∑ T est t =1 u t ( α ) · R t -1 σ . To bound the first derivative of the loss function, our next goal is to bound ∑ T est t =1 u t ( α ) . Let S j = ∑ j t =1 u t ( α ) . Observe that

Let us define

Since Φ( y ) is the CDF of the standard normal distribution, y 1 ∈ (0 , 2 /σ ) and y 2 ∈ -2 /σ, 0) , we have that L 1 is a constant well defined by the above expression. Also observe that

where Φ( y ) and ϕ ( y ) are the CDF and PDF of the standard normal distribution. The third equality above follows from ϵ t being independent from S 1 , . . . , S j -1 . Hence, S j is a martingale with bounded difference.

By Azuma-Hoeffding Inequality (see Lemma J.5), we thus have that for γ > 0 , the following concentration result holds:

Let us denote the high probability event, i.e., | S T est | ≤ L 1 √ 2 γT est log T est , as E 1 . Note that conditioning on E 1 , we would have

We also bound the second derivative using a constant L 2 ≥ 0 as follow:

Here, we define the constant L 2 as

Similar to our arguments above, since Φ( y ) is the CDF of the standard normal distribution, y 1 ∈ (0 , 2 /σ ) and y 2 ∈ -2 /σ, 0) , we have that L 2 ≥ 0 is a constant well defined by the last equality. Let V = E [ R 2 t ] > 0 denote the variance of reward under the steady state distribution. Since R 2 t is bounded, we can apply Hoeffding's Inequality (see Lemma J.4) that states

Let E 2 denote the high probability event that | ∑ T est t =1 R 2 t -1 -T est V | ≤ T est V 2 . Then, under this event, we have that ∑ T est t =1 R 2 t -1 ≥ T est V 2 . Hence, conditioning on the event E 2 , we have

After bounding the first and second derivatives of L ( α ) respectively, we are now ready to show the proximity between our estimated parameter ̂ α and the true parameter α , conditioning on the high-probability events E 1 ∩ E 2 . By optimality of ̂ α , we have that L ( ̂ α ) ≤ L ( α ) . This, together with (61), gives

Using (67), we can bound the left-hand side as

Conditioning on E 1 , we also know from (64) have that

Finally, combining (68), (69) and (70), we have that with probability at least 1 -2 /T γ est -2 exp( -T est V 2 / 2 2 ) , we have

which implies

J Concentration Inequalities

In this section, we state some useful concentration inequalities for subgaussian variables (see [29, 8, 45] for references).

Definition J.1 (Subgaussian variables) . We say that a zero-mean random variable X is σ -subgaussian (or, X ∼ subG ( σ ) ), if for all λ ∈ R , E [ e λX ] ≤ e λ 2 σ 2 2 .

In particular, note that if X ∼ N (0 , σ ) , then X ∼ subG( σ ) . Subgaussian variables satisfy the following concentration inequalities:

Lemma J.2 (Concentration of subgaussian variables) . A zero-mean random variable X is σ -subgaussian if and only if for any s > 0 , P [ | X | ≥ s ] ≤ 2 exp( -s 2 2 σ 2 ) .

Lemma J.3 (Hoeffding's Inequality) . Let X 1 , . . . , X n be independent zero-mean random variables such that X j ∼ subG ( σ j ) , then for any s > 0 , P [∣ ∣ ∣ 1 n ∑ n j =1 X j ∣ ∣ ∣ ≥ s ] ≤ 2 exp( -n 2 s 2 2 ∑ n j =1 σ 2 j ) .

In particular, bounded variables are known to be subgaussian, and we thus have a special case of the Hoeffding's Inequality that applies to bounded variables:

Lemma J.4 (Hoeffding's Inequality for Bounded Variables) . Let X 1 , . . . , X n be independent random variables such that a i ≤ X i ≤ b i . Consider the sum of these random variables S n = ∑ n i =1 X i . We

Another related concentration result is the Azuma-Hoeffding's Inequality stated below, which applies to martingales with bounded differences.

Lemma J.5 (Azuma-Hoeffding's Inequality) . Suppose that { S j : j = 0 , 1 , 2 , . . . } is a martingale and has bounded difference | S k -S k -1 | ≤ b k almost surely. Then, the following concentration bound holds for any N ∈ Z + and s > 0 : P( | S N -S 0 | ≥ s ) ≤ 2 exp ( -s 2 2 ∑ N k =1 b 2 k ) .

Peer reviews:
comments: SUMMARY:
The authors propose a bandit algorithm in the restless setting, when the rewards have an auto-regressive structure.

STRENGTHS:
1. Contributes to the non-stationary case, in contrast to the vast majority of results in the stationary setting.

WEAKNESSES:
1. Knowledge of (single parameter) alpha, and the problem of estimating it simultaneously. It is unclear of the theoretical claims (in addition to the numerical results) hold when these are relaxed (ref. Sec 2)

QUESTIONS:
1. Some discussion of Exp3's limitation is already discussed in the lower bound section. Would be great to see a discussion on on adversarial bandits and how much tighter your results are compared to some of the algorithms there. 
2. Unclear why the dynamic benchmark was chosen instead of the static one (other than it does seem suitable and is harder?). If instead we use the static benchmark, would there be any regret UB/LB gains?

LIMITATIONS:
None identified.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work uses AR(1) model to model the non-stationary multi-armed bandit (MAB) problem. This paper considers a new performance metric, dynamic steady-state regret, and establishes lower bound on regret. Furthermore, This paper proposes the AR2 algorithm and provides a relatively tight regret upper bound.\n\nSTRENGTHS:\nPrevious works on non-stationary multi-armed bandit (MAB) problems have often assumed bounded non-stationarity, as only bounded non-stationarity allows for achieving sublinear dynamic regret. This work, however considers the setting of unbounded non-stationarity. This provides a new perspective on handling non-stationarity and addresses the evaluation of algorithm performance in the context of linear regret. Additionally, this paper introduces the concept of steady state error to quantify the algorithm's performance under these conditions.\n\nWEAKNESSES:\n1. In this paper, the AR(1) model is defined with α∈(0,1), which actually qualifies it as a stationary model. α would need to be at least equal to 1 to be considered non-stationary. For example, in scenarios involving bounded non-stationarity, the expected reward's variation bound is typically represented by \\sum_{t=1}^T||r(t+1)-r(t)||_\\infty, which corresponds to the case of AR(1) with α=1. Therefore, using this stationary AR(1) model ( α∈(0,1) ) to characterize non-stationary multi-armed bandit (MAB) problems seems to be unreasonable.\n\n2. It seems unreasonable to assume that the expected reward has upper and lower bounds of [-R, R] in this setting. If we assume that the expected reward is bounded within [-R, R], even with random arm selection, the per-round dynamic regret should be at most 2R. This implies that any algorithm (e.g., algorithms without restarts) would eventually reach a steady state, which is unrealistic in a non-stationary environment. However, Theorem 5.1 in the paper does not seem to make this point clear: as α approaches 1, the per-round dynamic regret upper bound for random arm selection tends to infinity (which should be at most 2R when the expected reward is bounded by [-R,R]).\n\nQUESTIONS:\nI hope the authors can provide an explanation for the issues I mentioned regarding the stationary AR(1) model, the bounded expected rewards, and Theorem 5.1.\n\nIf the response is reasonable, I will consider increasing the score.\n\nLIMITATIONS:\nAs mentioned in the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of non-stationary bandit learning in bandits where rewards have auto-regressive temporal dependency. More specifically, this paper considers bandits where the evolution of the expected mean reward of each arm undergoes an independent AR-1 process truncated to a bounded interval. All arms share the same AR parameter $\\alpha$. The authors propose an algorithm Alternating and Restarting algorithm for dynamic AR bandits (AR2), which takes the AR parameter $\\alpha$, a stochastic rate of change $\\sigma$, epoch size $\\Delta_{\\mathrm{ep}}$ and a parameter $c_0$ as input. The algorithm “addresses the exploration-exploitation tradeoff by alternating between exploiting the superior arm and exploring the triggered arm within each epoch,” and it “handles the tradeoff between remembering and forgetting” via restarting. To illustrate the efficacy of the algorithm, an upper bound on the regret of AR2 is established, and is compared with a lower bound on the regret. Numerical experiments and a real-world case study on tourism have also been conducted.\n\nSTRENGTHS:\nThe paper is well-organized and the exposition is clear and easy to follow. The use of AR-1 model in characterizing non-stationary dynamics is also well-motivated.\n\nWEAKNESSES:\nThe main weakness of the paper lies in the insufficient justification of the efficacy of the algorithm AR2. Specifically: \n1. The paper suggests that a key mechanism of the algorithm is “an alternation mechanism adept at leveraging temporal dependencies to dynamically balance exploration with exploitation,” without discussing the intuition behind this counter-intuitive argument. \n- One question I have is that it seems that via alternation, the algorithm explores and exploits at the same intensity. Shouldn’t the intensities change according to the AR-1 parameter $\\alpha$ and the stochastic rate of change $\\sigma$? \n2. The regret upper bound established by Theorem 5.2 doesn’t sufficiently justify that AR2 performs well, due to the somewhat restrictive assumptions. \n- The bound does not consider the range $\\alpha \\in [0, \\overline{\\alpha})$. Does AR2 perform well in this regime? In addition, I understand that some other naive algorithm performs well in this regime, but Theorem 5.1 cannot fully justify that because the bound on average regret explodes as the number of arms $k \\rightarrow +\\infty$ (the bound becomes vacuous when $k$ is large enough because rewards are bounded). Also, Theorem 5.1 does not apply to AR2. \n- The assumption of $k \\leq \\mathcal{K}(\\alpha)$ suggests that Theorem 5.2 can only be applied when the number of arms $k$ is small. For example, when $\\alpha = \\overline{\\alpha} = 0.5$, the number of arms $\\mathcal{K}(\\alpha) = 2$: the theorem only applies to bandits with two arms. \n- Moreover, the bound on average regret grows in $k^3$, which can be very large when the number of arms $k$ is moderately large. \n3. I also have concerns on the numerical experiments.\n- One concern is that although the paper suggests that a real-world case study has been completed, experiments were not conducted on the real data, but rather, synthetic data generated by an AR-4 model fitted to the data.\n\nQUESTIONS:\nMy concerns and questions are raised in the Weakness section. In summary: \n1. It would be great if the authors could explain the intuition behind why the alternation mechanism can effectively balance exploration with exploitation, although it seems that the extent to which the algorithm explores is the same as the extent to which the algorithm exploits. \n2. Can the authors provide justifications of how well the algorithm AR2 performs when each of the assumptions $\\alpha \\in [\\overline{\\alpha}, 1)$, and $k \\leq \\mathcal{K}(\\alpha)$ in Theorem 5.2, is violated, and when $k$ is large? \n3. Can the authors explain why experiments are run against synthetic data instead of real data, when real data is provided in the tourism case study?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a non-stationary bandits problem where the reward rate of each arm evolves according to the AR(1) model, i.e., shrinks by a known multiplicative factor and adds an indep noise. They proposed an algo that strikes a balance between (i) exploration v. exploitation and (ii) remembering v. forgetting. As their main results, they showed that this algo achieves nearly optimal regret. Moreover, they presented a real world case study.\n\nSTRENGTHS:\n- the model is novel, neat and fundamental.\n- provided a nearly optimal regret\n- presented a high quality real world case study\n- the manuscript is well-written\n\nWEAKNESSES:\n- Since the main term in the regret is sigma, it is better to specify the function g in the lower bound (Thm 3.1), especially how it depends on sigma. After skimming the appendix, I am still not 100% sure whether the claimed UB and LB really match.\n\nQUESTIONS:\n- To ensure boundedness of the mean reward, you truncated the AR process. Why is this wlog?\n- this problem is reminiscent of the Slivkins Upfal COLT'08 paper - that paper also considered stochastically evolving reward rates, truncated at the boundary, and their results mainly depends on \\sigma. Can you comment on the connections between these two papers? E.g. does the S-U paper implies any preliminary results for your problem?\n- Can this problem be approximately viewed as RL? Specifically, assuming alpha is a constant, then the history older than log # rounds has little impact on the current reward rate. Can we encode the log n - step history as the state? (If so, the transition matrix is also known) Can we immediately obtain a regret bound using existing RL results?\n- the theoretical part considers AR(1) but the case study considers AR(4). Is this difference essential?\n\nother comments: \nline 145: \"2exogenous\" - is this a typo?\n\nLIMITATIONS:\n(typesetting issue) when I click on a reference, the pdf file does not jump to the reference page", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}