SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Superior arms. The superior arm at round  $t$ , denoted by  $i_{\mathrm{sup}}(t)$ , is one of the two most recently pulled arms, i.e.,  $I_{t-1}$  or  $I_{t-2}$ , that has the higher estimated reward (see (1)). We further define  $\widehat{r}_{\mathrm{sup}}(t) = \widehat{r}_{i_{\mathrm{sup}}(t)}(t)$  as the estimated reward of the superior arm at round  $t$ . We remark that here, for simplicity of analysis, our definition of the superior arm only considers the two most recently pulled arms. We can in fact set the superior arm to be the one with the highest estimated reward among the  $m$  most recently pulled arms for any constant  $m \geq 2$ . A similar theoretical analysis will then yield the same theoretical guarantee, as shown in Theorem 5.2. We further verify the robustness of AR2 to the choice of  $m$  in our numerical studies, where we consider all arms as potential candidates for the superior arm, and AR2 maintains its competitive performance (see Appendix A).
Triggered arms. In order to adapt to changes, AR2 identifies and keeps track of a set of arms with potential that are not pulled recently. We refer to these arms as triggered arms and denote their associated set by  $\mathcal{T}$ . In a round  $t$ , an arm  $i \neq i_{\mathrm{sup}}(t)$  gets triggered if the triggering criteria in (2) is satisfied. We call the earliest such time  $t > \tau_i(t)$  as the triggering time of arm  $i$  and denote it as  $\tau_i^{\mathrm{trig}}(t)$ . If arm  $i$  is triggered, it is added to the triggered set  $\mathcal{T}$ . When arm  $i$  gets pulled or is chosen as the superior arm, it is removed from  $\mathcal{T}$ . The right-hand side of the triggering criteria is a confidence bound constructed based on Hoeffding's Inequality (see details in Section 6 and Appendix H).
Note that at the exploration step, AR2 would only pull a triggered arm if the triggered set  $\mathcal{T}$  is non-empty. This inherently adjusts the rate of exploration in our alternation mechanism. In a slowly-changing environment (e.g., with small  $\sigma$ ), the triggered set may not always include arms needing exploration, thus allowing focused exploitation of the superior arm; in a fast-changing environment, the rate of exploration can be as high as the rate of exploitation.
5 Regret of Algorithm AR2
Our algorithm AR2 works for AR-1 model with any choice of parameter  $\alpha \in (0,1)$ . However, it turns out that the problem is less challenging when the future is weakly correlated with the past (i.e., when  $\alpha$  is small). Theorem 5.1 states that any algorithm—including both a naive approach that continuously pulls the same arm throughout the horizon and our algorithm AR2—can achieve near-optimal performance when  $\alpha$  is too small. This is because with small  $\alpha$ , (i) for any arm  $i \in [k]$ , the steady state distribution of its expected reward  $r_i(t)$  concentrates around zero with high probability, and (ii) our observations of past rewards quickly deteriorate in their value of providing insights on the evolution of expected rewards in the future.
Theorem 5.1. Any non-anticipating algorithm  $\mathcal{A}$  incurs per-round steady state regret of at most  $O\Big(\min (\sqrt{\log(1 / \alpha\sigma)} +\log k\cdot \frac{\alpha\sigma}{\sqrt{1 - \alpha^2}},2\Re)\Big)$ .
In Figure 1, we compare the order of the regret upper bound in Theorem 5.1 with the order of the regret lower bound in Theorem 3.1. Observe that  $\bar{\alpha} \triangleq 0.5$  serves as a rough threshold value such that when  $\alpha \in (0, \bar{\alpha})$ , the orders of lower and upper bounds almost match each other, which suggests limited room for improvement. (That being said, our numerical studies in Appendix A in fact show that AR2 still outperforms other benchmarks even when  $\alpha$  is small.) On the other hand, as  $\alpha \to 1$ , the gap between lower and upper bounds starts to expand, implying that the problem becomes more difficult and simplistic approaches such as the naive algorithm would no longer produce satisfying performance.
In the rest of the discussion, we thereby focus on the more challenging regime (i.e.,  $\alpha \in [\bar{\alpha}, 1)$ ), and establish a regret upper bound for AR2 when  $\alpha \in [\bar{\alpha}, 1)$ .
Figure 1: Comparison of the orders of the regret upper bound in Theorem 5.1 and the lower bound in Theorem 3.1, obtained with  $k = 5$ ,  $\sigma = 0.2$ ,  $C = 0.4$ . (Similar plots can be obtained for different values of  $k$  and  $\sigma$ .)
Theorem 5.2. Let  $\mathcal{K}(\alpha) \triangleq \left\lfloor \frac{1}{2} (\log \left(\frac{1}{8}\right) / \log \alpha + 1)\right\rfloor$ , and let  $\mathcal{A}$  be the AR2 algorithm that gets restarted in each epoch of length  $\Delta_{ep} = \lceil k\alpha^{-3}\sigma^{-3}\rceil$ . If  $\alpha \in [\bar{\alpha},1)$  and  $k \leq \mathcal{K}(\alpha)$ , the per-round steady state regret of  $\mathcal{A}$  satisfies  $\overline{\mathrm{REG}}_{\mathcal{A}} \leq O(c_0^2\alpha^2\sigma^2 k^3\log (c_0\alpha\sigma\sqrt{k}))$ , where  $c_0 = \sqrt{4\log(1 / \alpha\sigma) + 4\log\Delta_{ep} + 2\log(4k)}$ .
Recall that if  $\alpha$  is close to one, the regret lower bound can be characterized as  $\Omega (k\alpha^2\sigma^2)$ . This shows that our algorithm is optimal in terms of AR parameter  $\alpha$  and stochastic rate of change  $\sigma$  (up to logarithmic factors). In Appendix E.3, we further highlight the significance of Theorem 5.2 by illustrating the evolution of upper and lower bounds, as well as the per-round regret attained by AR2, at different values of  $\alpha$  and  $\sigma$ . We show that Theorem 5.2 well characterizes the performance of AR2 under different settings.
We remark that in terms of the dependency on  $k$ , Theorem 5.2 requires  $k \leq \mathcal{K}(\alpha)$  mainly for the rigor of theoretical analysis, and this bound loosens as  $\alpha$  approaches one (for example, when  $\alpha = 0.95$ ,
See, also, Figure 5 in Appendix E.2 for a comparison of upper/ lower bounds with respect to  $\sigma$
$\mathcal{K}(\alpha) = 20.$  ). Our numerical studies in Appendix A also reveal that (i) AR2 maintains its competitive performance even when the number of arms  $k$  exceeds  $\mathcal{K}(\alpha)$ ; and (ii) its per-round regret grows modestly with  $k$ . These suggest that the assumption/dependency on  $k$  for our upper bound are artifacts of our analysis, rather than an intrinsic property of our algorithm.
6 Proof of Theorem 5.2
The proof of Theorem 5.2 proceeds in four steps. Step 1 introduces distributed regret, a novel notion that distributes instantaneous regret over subsequent rounds, enabling us to rephrase per-round steady state regret. Step 2 defines a high-probability good event crucial for analyzing distributed regret. In Step 3, we leverage the good event to establish an upper bound for each round's distributed regret. Finally, Step 4 aggregates these regrets across rounds and epochs, establishing the per-round steady state regret of AR2 against the dynamic benchmark.
Step 1: Distributed regrets. Let  $\Delta r_i(t) = r^\star(t) - r_i(t)$  be the instantaneous regret from pulling arm  $i$  at round  $t$ , where  $r^\star(t) = \max_{i \in [k]} r_i(t)$  is the maximum expected reward at round  $t$ . Consider the expected per-round regret:  $\overline{\mathrm{REG}}_A = \mathbb{E}\left[\frac{1}{T} \sum_{t=1}^{T} \mathrm{REG}_A(t)\right] = \frac{1}{T} \sum_{t=1}^{T} \mathbb{E}\left[\Delta r_{I_t}(t)\right]$ , where  $\Delta r_{I_t}(t) = r^\star(t) - r_{I_t}(t)$  is the instantaneous regret incurred at round  $t$ . In the following, we show that we can distribute this instantaneous regret over subsequent rounds.
Note that for any  $i \in [k]$ , let  $\ell_i^{(j)}$  denote the round at which arm  $i$  gets pulled for the  $j$ th time within an epoch. Then during the period between two consecutive pulls  $\ell_i^{(j)} \leq t < \ell_i^{(j+1)}$ , the only regret incurred from pulling arm  $i$  is the instantaneous regret  $\Delta r_i(\ell_i^{(j)})$ . By previous definition, we also have  $\tau_i(t) = \ell_i^{(j)}$ ,  $\tau_i^{\mathrm{next}}(t) = \ell_i^{(j+1)}$  and  $\Delta \tau_i(t) = \ell_i^{(j+1)} - \ell_i^{(j)}$  for all  $\ell_i^{(j)} \leq t < \ell_i^{(j+1)}$ . Observe that we can decompose  $\Delta r_i(\ell_i^{(j)})$  as follows, which then leads to the notion of distributed regret:
$$
\Delta r _ {i} (\ell_ {i} ^ {(j)}) = \sum_ {t = \ell_ {i} ^ {(j)}} ^ {\ell_ {i} ^ {(j + 1)} - 1} \Big (\frac {\Delta r _ {i} (\ell_ {i} ^ {(j)}) \alpha^ {2 (t - \ell_ {i} ^ {(j)})}}{1 + \alpha^ {2} + \cdots + \alpha^ {2 (\ell_ {i} ^ {(j + 1)} - 1 - \ell_ {i} ^ {(j)})}} \Big) = \sum_ {t = \ell_ {i} ^ {(j)}} ^ {\ell_ {i} ^ {(j + 1)} - 1} \Big (\frac {\Delta r _ {i} (\tau_ {i} (t)) \alpha^ {2 (t - \tau_ {i} (t))}}{1 + \alpha^ {2} + \cdots + \alpha^ {2 (\Delta \tau_ {i} (t) - 1)}} \Big).
$$
Definition 6.1 (Distributed regret). The distributed regret of arm  $i$  at round  $t$  is defined as
$$
D _ {i} (t) = \left(\frac {\Delta r _ {i} \left(\tau_ {i} (t)\right)}{1 + \alpha^ {2} + \cdots + \alpha^ {2 \left(\Delta \tau_ {i} (t) - 1\right)}}\right) \alpha^ {2 \left(t - \tau_ {i} (t)\right)} \tag {3}
$$
Note that  $D_{i}(t) = 0$  if arm  $i$  is the best arm at round  $\tau_{i}(t)$ . Now, let  $T_{i}$  be the total number of rounds we pull arm  $i$  in the horizon, we can rewrite the expected per-round regret as
$$
\overline {{\operatorname {R E G}}} _ {\mathcal {A}} = \frac {1}{T} \sum_ {i = 1} ^ {k} \sum_ {j = 1} ^ {T _ {i}} \mathbb {E} \left[ \Delta r _ {i} \left(\ell_ {i} ^ {(j)}\right) \right] = \frac {1}{T} \sum_ {i = 1} ^ {k} \sum_ {i = 1} ^ {T} \mathbb {E} \left[ D _ {i} (t) \right] = \frac {1}{T} \sum_ {i = 1} ^ {k} \sum_ {s = 1} ^ {S} \sum_ {t = (s - 1) \Delta_ {\mathrm {e p}} + 1} ^ {s \Delta_ {\mathrm {e p}}} \mathbb {E} [ D _ {i} (t) ], \tag {4}
$$
where the second equality follows from the Definition 6.1 and the third equality follows from that AR2 proceeds in epochs of length  $\Delta_{\mathrm{ep}}$ , and  $S = T / \Delta_{\mathrm{ep}}$  is the total number of epochs.
Step 2: Good event and its implications. Before proceeding, we first define a good event  $\mathcal{G}(t)$  at round  $t$ , which would help simplify our analysis. In principle, we say that a good event  $\mathcal{G}(t)$  happens at round  $t$  if the noises within the epoch including  $t$  are not too large in magnitude. Recall from Section 2 that  $r_i(t)$  follows an AR-1 process with truncating boundaries:  $r_i(t + 1) = \mathcal{B}(\alpha (r_i(t) + \epsilon_i(t)))$ , where  $\mathcal{B}(y) = \min \{\max \{y, -\mathcal{R}\}, \mathcal{R}\}$ . Hence, we need to first define a new noise term that shows the influence of the truncating boundary.
$$
\tilde {\epsilon} _ {i} (t) = \left\{ \begin{array}{l l} \epsilon_ {i} (t) & \text {i f} \alpha \left(r _ {i} (t) + \epsilon_ {i} (t)\right) \in [ - \mathfrak {R}, \mathfrak {R} ] \\ \frac {1}{\alpha} \left[ \mathcal {B} \left(\alpha \left(r _ {i} (t) + \epsilon_ {i} (t)\right)\right) - \alpha r _ {i} (t) \right] & \text {o t h e r w i s e} \end{array} \right. \tag {5}
$$
4Here, the bound  $\kappa (\alpha)$  results from our loose upper bound for the number of triggered arms in the proof of Lemma H.3, where we used the fact that there are at most  $k - 1 <   \kappa (\alpha)$  triggered arms at any round to limit how quickly the values of the past observations diminish. If, at any given round, the number of triggered arm is  $O(1)$ , the upper bound  $\kappa (\alpha)$  would no longer be required, and the regret upper bound can be further reduced to  $O(c_0^2\alpha^2\sigma^2 k\log (c_0\alpha \sigma))$  , which matches the lower bound up to logarithmic factors.
In particular, the new noise  $\tilde{\epsilon}_i(t)$  satisfies the following recursive relationship:  $r_i(t + 1) = \alpha r_i(t) + \alpha \tilde{\epsilon}_i(t)$ . We now formally define the good event  $\mathcal{G}(t)$ , which states that for any sub-interval  $[t_0, t_1]$  within the epoch including  $t$ , the weighted sum of  $\tilde{\epsilon}_i(t)$  satisfies a concentration inequality.
Definition 6.2 (Good event at round  $t$ ). We say that the good event  $\mathcal{G}(t)$  occurs at round  $t$  if for every  $i \in [k]$ , and for every sub-interval  $[t_0, t_1]$ , where  $t - \Delta_{ep} \leq t_0 < t_1 \leq t + \Delta_{ep}$ , the weighted sum of the noises  $\tilde{\epsilon}_i(t)$  satisfies  $\left| \sum_{t=t_0}^{t_1-1} \alpha^{t_1-t} \tilde{\epsilon}_i(t) \right| < c_0 \sigma \sqrt{(\alpha^2 - \alpha^{2(t_1-t_0+1)}) / (1 - \alpha^2)}$ , where  $c_0 = \sqrt{4\log(1/\alpha\sigma) + 4\log\Delta_{ep} + 2\log(4k)}$ .
By building a connection between  $\tilde{\epsilon}_i(t)$  and  $\epsilon_{i}(t)$  and Hoeffding's inequality, we show Lemma 6.3, which confirms that the good event  $\mathcal{G}(t)$  happens with high probability.
Lemma 6.3. For any  $t \in [T]$ , we have  $\mathbb{P}[\mathcal{G}^c(t)] \leq (\alpha \sigma)^2$ .
Step 3: Distributed regret analysis. For a given round  $t$  in the  $s$ -th epoch, we can bound its expected distributed regret by first decomposing it into two terms:  $\mathbb{E}[D_i(t)] = \mathbb{E}[D_i(t)\mathbb{1}_{\mathcal{G}^c (t)}] + \mathbb{E}[D_i(t)\mathbb{1}_{\mathcal{G}(t)}]$ , where  $\mathbb{1}_{\mathcal{E}}$  is the indicator function of event  $\mathcal{E}$ . We can bound the first term by applying Lemma 6.3
$$
\mathbb {E} \left[ D _ {i} (t) \mathbb {1} _ {\mathcal {G} ^ {c} (t)} \right] \leq 2 \mathfrak {R} \cdot \mathbb {P} \left[ \mathcal {G} ^ {c} (t) \right] \leq 2 \mathfrak {R} \cdot (\alpha \sigma) ^ {2} = O \left(\sigma^ {2} \alpha^ {2}\right). \tag {6}
$$
To bound the second term, we rely on the implications of the good event  $\mathcal{G}(t)$  (presented in Appendix H.1) to show the following.
Lemma 6.4. Suppose that we apply AR2 to the non-stationary MAB problem with  $\alpha \in [\bar{\alpha},1)$  and  $k \leq \mathcal{K}(\alpha)$ , for every arm  $i \in [k]$  and some round  $\tau_i(t) \leq t < \tau_i^{\mathrm{next}}(t)$ , where  $\tau_i(t)$  and  $\tau_i^{\mathrm{next}}(t)$  are two consecutive rounds at which  $i$  gets pulled, we have  $\mathbb{E}[D_i(t)\mathbb{1}_{\mathcal{G}(t)}] \leq O(c_0^2\alpha^2\sigma^2 k^2\log (c_0\sigma\alpha\sqrt{k}))$ .
The proof of Lemma 6.4 is deferred to Appendix H.2. There, we provide a bound for the nominator and denominator of  $D_{i}(t)$  respectively, conditioning on the good event  $\mathcal{G}(t)$  and the value of  $\Delta r_{i}(t)$ . In the proof, we critically use the implications of the good event (see Appendix H.1). Now, combining (6) and Lemma 6.4, for any round  $t$  within the sth epoch such that  $\tau_{i}(t) \leq t < \tau_{i}^{\mathrm{next}}(t)$ , we have
$$
\mathbb {E} [ D _ {i} (t) ] = \mathbb {E} [ D _ {i} (t) \mathbb {1} _ {\mathcal {G} ^ {c} (t)} ] + \mathbb {E} [ D _ {i} (t) \mathbb {1} _ {\mathcal {G} (t)} ] \leq O \left(c _ {0} ^ {2} \sigma^ {2} \alpha^ {2} k ^ {2} \log \left(c _ {0} \sigma \alpha \sqrt {k}\right)\right). \tag {7}
$$
Step 4: Summing the distributed regrets. Given Equations (4), (7) and  $\Delta_{\mathrm{ep}} = \lceil k\alpha^{-3}\sigma^{-3}\rceil$  summing the expected distributed regrets first within an epoch, and then along the horizon yields
$$
\overline {{\mathrm {R E G}}} _ {\mathcal {A}} = \frac {1}{T} \sum_ {i = 1} ^ {k} \sum_ {s = 1} ^ {S} \sum_ {t = (s - 1) \Delta_ {\mathrm {e p}} + 1} ^ {s \Delta_ {\mathrm {e p}}} \mathbb {E} [ D _ {i} (t) ] \leq \frac {1}{T} \sum_ {i = 1} ^ {k} S \cdot [ O (\tilde {C} ^ {2} \log (\tilde {C})) \Delta_ {\mathrm {e p}} + 2 \mathfrak {K} ] = O (k \tilde {C} ^ {2} \log (\tilde {C})),
$$
where the additional  $2\mathfrak{R}$  term comes from the initialization round and the last round we pull arm  $i$  within an epoch. This concludes the proof if we plug in  $\tilde{C} = c_0\alpha \sigma \sqrt{k}$ .
7 A Real-World Case Study on Tourism Demand Prediction
In this section, we numerically demonstrate the efficacy of AR2 via a real-world case study based on a international Tourism Demand dataset for Australia [36]. In this case study, we act as a travel agency that needs to determine which vacation package to offer, where the demand for each vacation package is highly dependent on the tourism demand during each quarter. Our case study is further complemented by a number of synthetic experiments in Appendix A, where we compare AR2 against various benchmarks designed for both stationary and non-stationary settings and again show the superior performance of AR2 in adapting to the rapidly changing environment.
Dataset and setup. The international tourism demand dataset [36], obtained from the Australian Bureau of Statistics, records the number of individual tourist arrivals to Australia from Hong Kong during each quarter between the years 1975-1989. The authors of [36] have fitted an AR model to the logarithms of quarterly tourist arrivals, which results in an AR-4 model with a trend component:  $r_i(t) = -0.01 + 0.32R_i(t - 2) + 0.6R_i(t - 4)$ . Note that here, the number of tourist arrivals is strongly correlated with the number of arrivals four quarters before, which is likely due to seasonal patterns. The time series analysis in [36] also supports our assumption in Section 2 that the AR
<sup>5</sup>We keep the significant lags in forecasting tourist arrivals, which are the second and fourth lags.
parameters are known. In this dataset, as well as many others from the real world, decision-makers are likely to have access to historical data, which enables them to fit AR processes with good precision.
Given the AR model, we consider a recommendation setting where the travel agencies have  $k = 5$  vacation packages (arms) to offer to the tourists from Hong Kong, and wish to dynamically feature the package with the highest demand (reward) at each round. For each arm, we simulate its reward sequence by randomly drawing the initial reward from a uniform distribution in [0, 1] and simulate  $r_i(t)$  for a total of  $T = 200$  rounds, based on the AR-4 model above. We take the stochastic rate of change to be  $\sigma = 0.1$  for all arms.
Extension of AR2 to AR- $p$  processes. We extend our algorithm AR2 to handle MAB problems with rewards modeled using general AR- $p$  processes with trends; see Algorithm 2 in Appendix B.2. At a high level, the extension of AR2 uses similar techniques as described in Section 4. The key difference is that, for each arm  $i \in [k]$ , it maintains not only an estimate  $\widehat{r}_i(t)$  of the arm's reward at round  $t$ , but also an estimate  $\widehat{E}_i(t)$  that captures the associated estimation error. These estimates are dynamically updated based on the structure of the AR- $p$  process with trends. The extended algorithm, akin to Algorithm 1, (i) selects the arm with the highest estimated reward  $\widehat{r}_i(t)$  and (ii) triggers arms with potential, where the confidence bound in the triggering criteria now depends on the estimate of the error  $\widehat{E}_i(t)$ . For a comprehensive discussion of our extension, please refer to Appendix B.2.
Results. We compare the performance of AR2 against the two most competitive benchmarks ( $\epsilon$ -greedy and a modified UCB algorithm) that we identified in synthetic experiments; see Appendix A for comparisons with the other benchmarks. The comparison is shown in Table 1. It can be seen that AR2 stands out in terms of both the regret<sup>6</sup> and the number of times it pulls the optimal arm. In this case study, while the  $\epsilon$ -greedy algorithm frequently selects the optimal arm, it still accumulates high regret due to its purely random exploration, which can lead to the selection of arms with low rewards. Similarly, the modified UCB algorithm, which leverages the knowledge of the temporal structure in its confidence bound (see description in Appendix C), doesn't perform well. Due to the rapidly changing nature of the AR- $p$  process, modified UCB over-explores AR2, on the other hand, prove to be effective even amidst the rapid changes introduced by more complex time series.
Table 1: Performance comparison.
Algorithm	normalized regret	# optimal arms
AR2	0.26 (0.14)	142.04 (15.71)
UCB-mod	0.60 (0.27)	106.62 (7.12)
ε-greedy	0.38 (0.16)	133.83 (12.47)
8 Extension on Learning the AR Parameter
In the previous sections, we have assumed full knowledge of the AR parameter, which as discussed in Sections 2 and 7, when we have access to past data. This section proposes an algorithm extension for when the AR parameter needs estimation. We introduce a maximum likelihood method and numerically evaluate the performance of AR2, which show that AR2 still remains competitive despite noise in the estimated AR parameter.
A Maximum Likelihood Estimator. If the decision maker does not have prior knowledge of the AR parameter  $\alpha$ , one can learn the AR parameter via Maximum Likelihood Estimation (MLE). In the first  $T_{\mathrm{est}}$  rounds of the time horizon, where  $T_{\mathrm{est}} = O(T^{\beta})$  for some  $\beta \in (0,1)$ , we pull one arm  $i \in [k]$  consecutively for a fixed arm  $i$  and observe rewards  $R_{i}(1), \ldots, R_{i}(T_{\mathrm{est}})$ . We then define the maximum likelihood estimator  $\hat{\alpha} = \arg \min_{\alpha \in (0,1)} \mathcal{L}(\alpha)$ , where  $\mathcal{L}(\alpha)$  is the negative of the log-likelihood function defined in (60) of Appendix I. Observe that here we cannot directly apply linear regression for estimating  $\alpha$  because the existence of truncating boundaries would lead to a biased estimator. MLE, on the other hand, remains robust even when the expected reward of our arm hits the boundary. Since the number of rounds used for estimation  $T_{\mathrm{est}}$  scales sublinearly with  $T$ , the regret incurred within the first  $T_{\mathrm{est}}$  rounds would not impact our per-round regret upper bound. If we have heterogeneous AR parameters  $\alpha_{i}$  for arms  $i \in [k]$  (as in Appendix A), one can simply pull each arm  $i \in [k]$  consecutively for  $T_{\mathrm{est}}$  rounds and perform MLE for each  $\alpha_{i}$ .
The following proposition quantifies the amount of noise in the estimated AR parameter.
Proposition 8.1. Let  $\widehat{\alpha} = \arg \min_{\alpha \in (0,1)}\mathcal{L}(\alpha)$  be the estimated AR parameter. Then, for  $\gamma >0$  with probability at least  $1 - 2 / T_{est}^{\gamma} - 2\exp (-T_{est}V^{2} / 2\Re^{2})$ , there exists constants  $L_{1},L_{2}$  such that
Here we evaluate each algorithm using the normalized regret:  $\sum_{t=1}^{T} \mathrm{REG}_{\mathcal{A}}(t) / \sum_{t=1}^{T} r^{\star}(t)$ , which normalizes the total regret with the optimal reward in hindsight.
$|\widehat{\alpha} - \alpha| \leq (4\sigma \Re L_1) / (VL_2) \cdot \sqrt{2\gamma \log T_{est} / T_{est}}$ , where  $V$  is the variance of the observed reward in the steady state distribution, which is independent of our algorithm or the time horizon  $T$ .
We plot the normalized regret incurred by AR2,  $\epsilon$ -greedy and mod-UCB using the estimated AR parameters, obtained through MLE with  $T_{\mathrm{est}} = \{25, 50, 100\}$ . We observe that whenever the AR parameters  $\alpha_{i}$  are small or large, the performance of AR2 outcompetes the other two benchmarks. In particular, AR2 appears to be robust to the noises in the estimated AR parameter, with performance close to what we would otherwise obtain with the accurate AR parameters (see Table 2 in Appendix A).
Figure 2: Box plots of normalized regret incurred by each algorithm at  $T = 10,000$  (AR2 highlighted in gray). The input AR parameter  $\widehat{\alpha}_i$  is the maximum-likelihood estimator of the true AR parameter. We vary the magnitudes of AR parameters  $\mathbb{E}[\alpha_i] \in \{0.4,0.9\}$  and the number of rounds used for estimation  $T_{\mathrm{est}} \in \{25,50,100\}$ . The parameters of our instances are the same as in Appendix A, and we take  $k = 6$ .
9 Conclusion and Future Directions