SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qinyi Chen
Operations Research Center Massachusetts Institute of Technology qinyic@mit.edu
Negin Golrezaei
Sloan School of Management  
Massachusetts Institute of Technology  
golrezae@mit.edu
Djallel Bouneffouf
djallel.bouneffouf@ibm.com
Traditional multi-armed bandit (MAB) frameworks, predominantly examined under stochastic or adversarial settings, often overlook the temporal dynamics inherent in many real-world applications such as recommendation systems and online advertising. This paper introduces a novel non-stationary MAB framework that captures the temporal structure of these real-world dynamics through an auto-regressive (AR) reward structure. We propose an algorithm that integrates two key mechanisms: (i) an alternation mechanism adept at leveraging temporal dependencies to dynamically balance exploration and exploitation, and (ii) a restarting mechanism designed to discard out-of-date information. Our algorithm achieves a regret upper bound that nearly matches the lower bound, with regret measured against a robust dynamic benchmark. Finally, via a real-world case study on tourism demand prediction, we demonstrate both the efficacy of our algorithm and the broader applicability of our techniques to more complex, rapidly evolving time series.
The multi-armed bandit (MAB) framework [51, 46] is commonly used to study online decision-making under uncertainty. It is primarily examined under either the stochastic setting [51, 6], where arms have fixed unknown reward distributions, or the adversarial setting [7, 5, 31, 16], where an adversary determines the reward distributions that can change arbitrarily over time. However, neither setting accurately represents real-world decision-making problems (see examples in dynamic pricing [30], online advertising [42], and online auctions [33]), where reward distributions change over time following intrinsic temporal structures. These structures often exhibit frequent variations and temporal dependencies, making them challenging to approximate using stationary distributions.
This motivates us to consider a non-stationary bandits setting involving certain temporal structure, which captures the real-world characteristics. Specifically, we are interested in temporal structures with linear amount of changes, which are distinct from the infrequent, limited changes typically handled by change-point detection [25] or seasonality analysis [15]. Such frequent changes are commonly observed in applications such as financial data [14] and click-through rates in online advertising [2], where the data experience lots of volatility within a short time. Traditional bandits algorithms, such as Thompson sampling or UCB, may explore too aggressively in response to these drastic changes, leading to suboptimal performance [39]. That being said, the observed time series in these scenarios typically exhibit certain temporal dependencies between past and present, which can be leveraged to improve decision-making.
To encapsulate the main characteristics of real-world time series, we relax the previously restrictive assumptions on the reward distributions and study a non-stationary MAB problem, where the reward
of each arm evolves over time following an auto-regressive (AR) temporal structure. The AR model is a popular time series model for predicting outcomes in applications such as sales [47], advertising [1] and marketing [18]. It simultaneously captures frequent changes (i.e., linear amount of changes over the time horizon) and temporal dependencies, both of which impact the quality of online decision-making. Although real-world data cannot be perfectly represented by an AR model, it has been shown that the AR parameters accurately capture temporal dependencies and can serve as a useful proxy [47]. The non-stationary bandits framework with an AR reward structure can effectively model a wide range of real-world applications, including:
(1) recommendation systems, where MAB can determine the best product/contents to display to users, while the AR model captures the evolution of user preferences and demand over time [21] (see Section 7 for a related case study on tourism demand prediction);
(2) online advertising, where MAB adjusts budget allocation for ad campaigns, and the AR model can capture the evolution of click-through rates of ads. The amount of changes are usually linear in time, making the AR structure a potential modeling choice [2];
(3) financial portfolio management, where MAB can determine the investment allocation, and the AR model can capture the evolution of returns for each investment/asset [4, 19].
When making online decisions in the presence of frequent changes, traditional methods that are designed for stationary or adversarial settings can be ineffective, due to two main challenges.
1. Balancing exploration and exploitation becomes more challenging, as complete exploration is often unattainable. Continuous exploration is crucial, but there is also a risk of over-exploration. Therefore, finding the right balance becomes essential in navigating this tradeoff effectively.
2. Frequent changes in the environment can significantly diminish the value of past learnings. Failing to promptly leverage the knowledge we have gained can lead to a swift deterioration in our confidence levels regarding the accuracy of our reward estimates.
The goal of this work is to show, via our non-stationary bandits framework with an AR reward structure, that effectively leveraging knowledge of temporal structure can address these challenges. In our approach, we would use our knowledge or estimates of the temporal dependency, measured by the AR parameter. Since non-stationary bandits with temporal structures exhibiting such real-world characteristics have been scarcely explored in the literature, we consider the AR structure as a suitable starting point for showcasing key ideas and techniques. We anticipate that this work will inspire future research on bandits with alternative temporal structures with similar features.
We present an algorithm for the non-stationary AR bandits, called AR2, which stands for "Alternating and Restarting" algorithm for non-stationary "AR" bandits. AR2 features two mechanisms: (i) an alternation mechanism that handles the first challenge above by enforcing exploitation at least every other round, and switch to exploration only when we discover an arm with potential, instead of simply adopting the principle of "optimism in the fact of uncertainty" which can lead to over-exploration. (ii) a restarting mechanism that handles the second challenge by discarding unnecessary information when our confidence level deteriorates significantly, which balances a second tradeoff known as "remembering"-and-"forgetting"; this tradeoff is crucial even for time series with limited changes [10]. Overall, our work addresses an important gap in the literature by studying non-stationary bandits with temporal structure that exhibit frequent changes and temporal dependency. While the AR structure does not capture all temporal structures in the real world, we believe that our high-level messages and techniques will guide future research on bandits with other temporal structures.
In evaluating the performance of our algorithm, we employ the concept of per-round steady state dynamic regret (Definition 2.1). This metric serves as a robust dynamic benchmark that competes with the best arm in each round. It surpasses the static benchmark used in stochastic and adversarial MAB, making it a considerably stronger measure of performance. We provide an upper bound on the regret of AR2 (Theorem 5.2), for which the analysis is rather intricate (Section 6), and show that it almost matches the regret lower bound (Theorem 3.1). Our lower bound result also characterizes the challenges embedded in AR temporal processes, as it shows the per-round dynamic regret does not go to zero as  $T$  increases, implying that any algorithm needs to keep exploring over time.
Finally, we conduct a real-world case study on tourism demand prediction [36] in Section 7, confirming the superiority of AR2 compared to benchmark algorithms. There, we also show that the techniques and high-level ideas of our algorithm can be readily extended to handle more complicated, rapidly changing temporal structure (e.g., general AR- $p$  processes) while still achieving good performance. Our case study is complemented by synthetic experiments in Appendix A which show the strength of AR2 against a number of benchmarks designed for stationary and non-stationary settings.
Our work, which focuses on MAB with an AR temporal structure, contributes to the non-stationary MAB literature that draws attention in recent years. Most related works on non-stationary bandits focused on either the rested bandits [26] or restless bandits [53]. In rested bandits, only the expected reward of the arm that is being pulled will change, while in restless bandits, the expected reward of each arm changes at each round according to a known yet arbitrary stochastic process, regardless of whether the arm is pulled. Recently, there has also been a third stream of literature that studies non-stationary bandits where the changes in reward distributions depend on both which arm is pulled and the time period that passes by since its last pull (see, e.g., [32, 13]). The non-stationary bandit problem that we study belongs to the restless setting, since the reward distributions change at each round based on AR processes, independent of our actions. Restless bandits, however, is known to be difficult and intractable [43], and many work thus focus on studying its approximation [28] or relaxation [9].
One line of closely related work on restless bandits studies non-stationary MAB problems with limited amount of changes. These works either assume piecewise stationary rewards (see [25, 37, 12]), or impose a variation budget on the total amount of changes (see [10]). They differ from our setting in that they rely on the amount of changes in the environment to be sublinear in  $T$ , while in AR processes, changes are more rapid and linear in  $T$ . Another line of relevant research on restless bandits assumes different types of temporal structures for the reward distributions. See, e.g., [38, 41, 50, 40] for finite-state Markovian processes, [27] for stationary  $\phi$ -mixing processes and [48] for Brownian processes, which is a special case of an AR model but does not experience the exponential decay in the correlation between the past and the future. Recently, [39] proposed a predictive sampling algorithm that can also be applied to AR-1 bandits; their main focus, however, is to show the algorithm's superiority over the traditional Thompson sampling algorithm in non-stationary environments.
Similar to the second line of works above, our work also aims to exploit the temporal structure of the reward distributions to devise well-performing learning algorithms. However, the AR process is much more unpredictable than the temporal structures studied previously, due to its infinite state space and the fast decay in the values of the past observations. We are thus prompted to take a different approach that is designed specifically for adapting to the rapid changes attributed to the AR process.
Expected rewards. Consider a non-stationary MAB problem with  $k$  arms over  $T$  rounds. The state (expected reward) of arm  $i \in [k]$  at any round  $t \in [T]$  is denoted by  $r_i(t)$ , where  $r_i(t) \in [-\mathfrak{R}, +\mathfrak{R}]$ . Conditioned on the state  $r_i(t)$ , the realized reward of arm  $i$  at round  $t$ , denoted by  $R_i(t)$ , is given by  $R_i(t) = r_i(t) + \epsilon_i(t)$ , where  $\epsilon_i(t) \sim N(0, \sigma)$  is independent across arms and rounds and we call  $\sigma \in (0, 1)$  the stochastic rate of change. More generally, if each arm  $i \in [k]$  has a different, unknown stochastic rate of change  $\sigma_i$ , as long as we have an upper bound  $\sigma \geq \max_i \sigma_i$ , all of our theoretical results naturally carry over with the same dependency on  $\sigma$ .
Evolution of expected rewards. The expected reward of each arm  $i \in [k]$ ,  $r_i(t)$ , evolves over time. In our model, we assume that  $r_i(t)$  undergoes an independent AR-1 process:
$$
r _ {i} (t + 1) = \alpha \left(r _ {i} (t) + \epsilon_ {i} (t)\right) = \alpha R _ {i} (t),
$$
or equivalently,  $R_{i}(t + 1) = \alpha R_{i}(t) + \epsilon_{i}(t + 1)$ , where  $\alpha \in (0,1)$  is the parameter of the AR-1 model and can be used as a proxy to measure temporal correlation over time. As  $\alpha$  increases, the temporal correlation increases. As commonly seen in the bandit literature (e.g., [35]), we make the rewards bounded by truncating  $r_{i}(t + 1)$  when its absolute value exceeds some finite boundary value  $\Re > 0$ . That is,  $r_{i}(t + 1) = \mathcal{B}\big(\alpha (r_{i}(t) + \epsilon_{i}(t))\big)$ , where  $\mathcal{B}(y) = \min \{\max \{y, -\mathfrak{R}\}, \mathfrak{R}\}$ .
Here, we assume that the AR parameter  $\alpha$  is known to the decision maker, a common assumption in literature [10, 27, 48], and often justifiable in real-world scenarios where sufficient historical data enables accurate fitting of AR models (see Section 7). This assumption is made to simplify the analysis. However, in Section 8, we would show that this assumption can be relaxed, and the AR parameter can be effectively estimated via maximum likelihood estimation.
We also assume here that all arms share the same AR parameter  $\alpha$ . As our numerical studies in Appendix A and Section 8 suggest, this assumption can be relaxed too. There, we generalize our model by considering heterogeneous AR parameters  $\alpha_{i}$  for each arm  $i \in [k]$ , and our algorithm maintains its performance. Nonetheless, it is worth noting that the homogeneity of AR parameter can be justified in some applications. One example can be seen in our case study in Section 7, where a travel agency offers vacation packages to travelers originating from a particular origin and heading
to a specific destination. In this scenario, the demand is modeled as a general AR-4 process, with the AR parameters being consistent across all arms. This is because the temporal correlation here is primarily determined by exogenous factors related to the travel route, such as popular attractions and the availability of flights between the two locations, hence impacting all arms in the same way.
Feedback structure and goal. Our goal is to design an algorithm that obtains high reward, against a strong dynamic benchmark that in every round  $t$ , pulls an arm that has the highest expected reward. This benchmark is much more demanding than the time-invariant benchmark used in traditional MAB literature. Obtaining good performance against the dynamic benchmark is particularly challenging as in any round  $t \in [T]$ , only the reward of the pulled arms can be observed. Not being able to observe the reward of unpulled arms in dynamic and time-varying environments introduces missing values in the AR process associated with arms. This, in turn, deteriorates the prediction quality of the future expected rewards and the performance of any algorithm that relies on such predictions (see [3] for a work that studies the impact of missing values on the prediction quality in AR processes).
We now formally define how we measure the performance of any non-anticipating algorithm.
Definition 2.1 (Dynamic steady state regret). Let  $\mathcal{A}$  be a non-anticipating algorithm that, at each round  $t\in [T]$ , pulls arm  $I_{t}$  based on the history  $\{I_1,R_1,\dots ,I_{t - 1},R_{t - 1}\}$ , where  $R_{t^{\prime}} = R_{I_{t^{\prime}}}(\bar{t}^{\prime})$  is the observed reward at round  $t^\prime < t$ . The dynamic regret of  $\mathcal{A}$  at round  $t$  is defined as
$$
\operatorname {R E G} _ {\mathcal {A}} (t) = r ^ {\star} (t) - r _ {I _ {t}} (t),
$$
where  $r^{\star}(t) = \max_{i\in [k]}\{r_i(t)\}$  is the maximum expected reward at round  $t$ . Furthermore, the per-round steady state regret of algorithm  $\mathcal{A}$  is defined as
$$
\overline {{\operatorname {R E G}}} _ {\mathcal {A}} = \lim  _ {T \rightarrow \infty} \sup  _ {t = 1} \mathbb {E} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} \operatorname {R E G} _ {\mathcal {A}} (t) \right].
$$
We remark that in Definition 2.1, our per-round regret is defined asymptotically mainly because we would like to focus on evaluating the steady state performance of our algorithm (i.e., under the steady state distribution). All of our theoretical analyses for regret lower bound (Section 3) and upper bounds (Section 5) are also performed under the steady state distribution.
In this work, we would like to design an algorithm with a small per-round steady state regret, where the per-round regret is a function of the stochastic rate of change  $(\sigma)$  and the temporal correlation  $(\alpha)$ .
3 Regret Lower Bound
We now provide a lower bound on the best achievable dynamic per-round steady state regret.
Theorem 3.1 (Regret Lower Bound). Consider a non-stationary MAB problem with  $k$  arms, where the expected reward of each arm evolves as an independent AR-1 process with parameter  $\alpha$ , stochastic rate of change  $\sigma$  and truncating boundaries  $[-\mathfrak{R}, \mathfrak{R}]$ . Then the per-round steady state regret of any algorithm  $\mathcal{A}$  is at least  $\Omega(g(k, \alpha, \sigma)\alpha\sigma)$ , where  $g(k, \alpha, \sigma)$  is the probability that two best arms are within  $\alpha\sigma$  distance of each other at any given round, under the steady state distribution. See the expression of this probability in Equation (16) in Appendix F.
The proof of Theorem 3.1, provided in Appendix F, builds on the following idea: even if algorithm  $\mathcal{A}$  has access to all past information at round  $t$ , due to stochastic noise  $\epsilon_i(t)$ ,  $\mathcal{A}$  will pull a sub-optimal arm with constant probability at round  $t + 1$ . We show that the probability of the two best arms' expected rewards being within  $\alpha \sigma$  of each other can be expressed as  $g(k,\alpha ,\sigma)$  (see (16)). If the two best arms are  $\alpha \sigma$  close,  $\mathcal{A}$  will incur  $\Omega (\alpha \sigma)$  regret with constant probability at round  $t + 1$ .
In the extreme case when  $\alpha$  is close to one, the steady state distribution of  $r_i(t)$  can be approximated with a uniform distribution within the boundaries and two probability masses at the boundaries (see Appendix D). Using the uniform approximation, one can show that  $g(k,\alpha ,\sigma) = \Omega (k\alpha \sigma)$  (see discussion in Appendix F). This then yields a regret lower bound of order  $\Omega (k\alpha^2\sigma^2)$ .
Theorem 3.1 implies that under our setup, the best achievable per-round regret with respect to a strong dynamic benchmark does not converge to zero, which differs from stationary or adversarial
<sup>1</sup>If we assume that the initial state  $r_i(0)$  is drawn from the steady state distribution, defined in Appendix D, we can simply define our per-round regret as  $\overline{\mathrm{REG}}_{\mathcal{A}} = \mathbb{E}\left[\frac{1}{T}\sum_{t = 1}^{T}\mathrm{REG}_{\mathcal{A}}(t)\right]$ , and all of our results remain valid. This also matches the definition of dynamic regret in works such as [10, 11, 39].
See, also, Appendix E.1 for an illustration of how  $g(k,\alpha ,\sigma)$  evolves with respect to  $\sigma$  under different  $\alpha$ .
Algorithm 1 Alternating and Restarting algorithm for non-stationary AR bandits (AR2)
Input: AR Parameter  $\alpha$ , stochastic rate of change  $\sigma$ , epoch size  $\Delta_{\mathrm{ep}}$ , parameter  $c_0$ .
1. Set the epoch index  $s = 1$ , parameter  $c_{1} = 24c_{0}$ .
2. Repeat while  $s \leq \left\lceil T / \Delta_{\mathrm{ep}} \right\rceil$ :
(a) Initialization: Set  $t_0 = (s - 1)\Delta_{\mathrm{ep}}$ . Set the initial triggered set  $\mathcal{T} = \emptyset$ . Pull each arm  $i \in [k]$  at round  $t_0 + i$  and set  $\tau_i = t_0 + i$ . Set estimates  $\widehat{r}_i(t_0 + k + 1) = \alpha^{k - i} \cdot \mathcal{B}(\alpha R_i(t_0 + i))$ .
(b) Repeat for  $t = t_0 + (k + 1),\ldots ,\min \{t_0 + \Delta_{\mathrm{ep}},T\}$
- Update the identity of the superior arm and its estimated reward
$$
i _ {\sup } (t) = \left\{ \begin{array}{l l} I _ {t - 1} & \text {i f} \widehat {r} _ {I _ {t - 1}} (t) \geq \widehat {r} _ {I _ {t - 2}} (t) \\ I _ {t - 2} & \text {i f} \widehat {r} _ {I _ {t - 1}} (t) <   \widehat {r} _ {I _ {t - 2}} (t) \end{array} \right. \quad \text {a n d} \quad \widehat {r} _ {\sup } (t) = \widehat {r} _ {i _ {\sup } (t)} (t). \tag {1}
$$
- Trigger arms with potential: For  $i \notin \mathcal{T} \cup \{i_{\mathrm{sup}}(t)\}$ , trigger arm  $i$  if
$$
\widehat {r} _ {\sup } (t) - \widehat {r} _ {i} (t) \leq c _ {1} \sigma \sqrt {\left(\alpha^ {2} - \alpha^ {2 \left(t - \tau_ {i} + 1\right)}\right) / \left(1 - \alpha^ {2}\right)}. \tag {2}
$$
If triggered, add  $i$  to  $\mathcal{T}$  and set  $\tau_{i}^{\mathrm{trig}} = t$ .
Alternate between exploration and exploitation:
- If  $t$  is odd and  $\mathcal{T} \neq \emptyset$ , pull a triggered arm with the earliest triggering time:  $I_{t} = \arg \min_{j \in \mathcal{T}} \tau_{j}^{\mathrm{trig}}$ .
- Otherwise, pull the superior arm  $I_{t} = i_{\mathrm{sup}}(t)$ .
- Receive a reward  $R_{I_t}(t)$ , and set  $\tau_{I_t} = t$ .
- Maintain Estimates: Set  $\widehat{r}_{I_t}(t + 1) = \mathcal{B}(\alpha R_{I_t}(t))$  and set  $\widehat{r}_i(t + 1) = \alpha \widehat{r}_i(t)$  for  $i\neq I_t$
(c) Set  $s = s + 1$ .
bandits with a time-invariant benchmark where algorithms like UCB for stationary bandits and Exp3 for adversarial bandits can achieve zero per-round regret as  $T$  approaches infinity. In our setting, each arm undergoes linear amount of changes, making it challenging for any algorithm to adapt to the changing environment in time. Our result aligns with the lower bound from [10], which states that when the total variation of expected rewards is  $O(T)$ , the regret also grows linearly.
4 Algorithm AR2
In this section, we present our algorithm, called AR2 (Alternating and Restarting algorithm for nonstationary "AR" bandits). Algorithm 1 outlines the workings of AR2, which operates in epochs of a fixed length. Within each epoch, AR2 maintains and updates estimates of the expected rewards for each arm  $i \in [k]$ . It alternates between exploitation and exploration steps based on these estimates. During exploitation, AR2 plays a "superior arm" expected to yield high rewards, while in exploration, it selects a "triggered arm" that hasn't been pulled recently but has high potential. At the end of an epoch, the algorithm restarts. As alluded earlier, AR2 effectively balances two inherent tradeoffs in non-stationary MAB problems with AR reward structure. Firstly, it addresses the exploration-exploitation tradeoff by alternating between exploiting the superior arm and exploring the triggered arm within each epoch. Secondly, it handles the tradeoff between "remembering"-and-"forgetting", a commonly considered tradeoff in non-stationary environments [10, 11], via restarting.
Maintaining estimates of expected reward of arms. For any round  $t$  and arm  $i \in [k]$ , let  $\tau_{i}(t)$  be the last round before  $t$  (including  $t$ ) at which arm  $i$  is pulled, and let  $\tau_{i}^{\mathrm{next}}(t)$  be the next round after  $t$  (excluding  $t$ ) at which arm  $i$  is pulled. Let  $\Delta \tau_{i}(t) = \tau_{i}^{\mathrm{next}}(t) - \tau_{i}(t)$  be the gap between two consecutive pulls of arm  $i$  around  $t$ . Define  $\widehat{r}_i(t)$  as the estimate of the reward of arm  $i$  at  $t$  based on the most recent observed reward of arm  $i$  (i.e.,  $R_{i}(\tau_{i}(t))$ ). Via recursive updates (see Step (b)), AR2 maintains the following estimate of expected reward for each arm:
$$
\widehat {r} _ {i} (t) = \alpha^ {t - \tau_ {i} (t) - 1} \widehat {r} _ {i} (\tau_ {i} (t) + 1) = \alpha^ {t - \tau_ {i} (t) - 1} \mathcal {B} \bigl (\alpha R _ {i} (\tau_ {i} (t)) \bigr).
$$