SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Arun Verma Zhongxiang Dai Yao Shu Bryan Kian Hsiang Low
Department of Computer Science, National University of Singapore, Republic of Singapore {arun, daizhongxiang, shuyao, lowkh}@comp.nus.edu.sg
We study a novel variant of the parameterized bandits problem in which the learner can observe additional auxiliary feedback that is correlated with the observed reward. The auxiliary feedback is readily available in many real-life applications, e.g., an online platform that wants to recommend the best-rated services to its users can observe the user's rating of service (rewards) and collect additional information like service delivery time (auxiliary feedback). In this paper, we first develop a method that exploits auxiliary feedback to build a reward estimator with tight confidence bounds, leading to a smaller regret. We then characterize the regret reduction in terms of the correlation coefficient between reward and its auxiliary feedback. Experimental results in different settings also verify the performance gain achieved by our proposed method.
Parameterized bandits (Slivkins, 2019; Lattimore and Szepesvári, 2020) have many real-life applications in online recommendation, advertising, web search, and e-commerce. In this bandit problem, a learner selects an action and receives a reward for the selected action. Due to the large (or infinite) number of actions, the mean reward of each action is assumed to be parameterized by an unknown function, e.g., linear (Li et al., 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011; Agrawal and Goyal, 2013), GLM (Filippi et al., 2010; Li et al., 2017; Jun et al., 2017), and non-linear (Valko et al., 2013; Chowdhury and Gopalan, 2017). The learner aims to learn the best action as quickly as possible. However, it depends on the tightness of confidence bounds of function that correlate action with the reward. The learner exploits any available information like side information (i.e., information available to the learner before selecting an action, e.g., contexts) (Li et al., 2010; Agrawal and Goyal, 2013; Li et al., 2017) and side observations (i.e., information about other actions, e.g., graph feedback) (Alon et al., 2015; Wu et al., 2015) to make confidence bounds as tight as possible. This paper considers another type of additional information (correlated with the reward) that a learner can observe with reward for the selected action, which we call auxiliary feedback.
The auxiliary feedback is readily available in many real-life applications. For example, consider an online food delivery platform that wishes to recommend the best restaurants (actions) to its users. After receiving food, the platform observes user ratings (rewards) for the order and can collect additional information like food delivery time (auxiliary feedback). Since the restaurant's rating also depends on overall food delivery time, one can expect it to be correlated with the user rating. The platform can estimate or know the average delivery time for a given order from historical data. Similar scenarios arise in recommending the best cab to users (auxiliary information can be the cab's distance from the rider or driver's response to ride request), e-commerce platforms choosing top sellers to buyers (auxiliary information can be seller's response time for order confirmation and delivery), queuing network (Lavenberg and Welch, 1981; Lavenberg et al., 1982), jobs scheduler (Verma and Hanawal, 2021), and many more. Therefore, the following question naturally arises: How to exploit correlated auxiliary feedback to improve the performance of a bandit algorithm?
One possible method is to use auxiliary feedback in the form of control variates (Lavenberg et al., 1982; Nelson, 1990) for the observed reward. A control variate represents any random variable (auxiliary feedback) with a known mean that is correlated with the random variable of interest (reward). Several works (Kreutzer et al., 2017; Sutton and Barto, 2018; Vlassis et al., 2021; Verma and Hanawal, 2021) have used control variates to estimate the mean reward estimator with smaller variance, leading to tight confidence bounds and hence better performance. The closest work to our setting is Verma and Hanawal (2021). However, it only focuses on the non-parameterized setting and assumes a finite number of actions. We thus consider a more general bandit setting with a large (or even infinite) number of actions and allow an unknown function to parameterize auxiliary feedback.
Motivated by control variate theory (Nelson, 1990), we first introduce hybrid reward, which combines the reward and its auxiliary feedback in such a way that hybrid reward is an unbiased reward estimator with smaller variance than the observed reward. However, the optimal combination of reward and its auxiliary feedback requires knowledge of the covariance matrix among auxiliary feedback and covariance between reward and its auxiliary feedback, which may not be available in practice. Since the reward and its auxiliary feedback are functions of the selected action, existing control variate results can not be useful to our sequential setting. Naturally, we face the question of how to combine reward and its auxiliary feedback efficiently using available information. To answer this, we extend control variate theory results to the problems where known functions can parameterize control variates (in Section 3) and then extend to setting where unknown functions parameterize control variates (in Section 4). These contributions are themselves of independent interest in control variate theory.
Equipped with these results, we show that the variance of hybrid rewards is smaller than observed rewards (Theorem 1 and Theorem 3). We then propose a method that uses hybrid rewards instead of observed rewards for estimating reward function, resulting in tight confidence bounds and, hence, lower regret. We introduce the notion of Auxiliary Feedback Compatible (AFC) bandit algorithm. An AFC bandit algorithm can use hybrid rewards instead of only observed rewards. We prove that the expected instantaneous regret of any AFC bandit algorithm using hybrid rewards is smaller by a factor of  $O((1 - \rho^2)^{\frac{1}{2}})$  compared to the same AFC bandit algorithm using only observed rewards, where  $\rho$  is the correlation coefficient of the reward and its auxiliary feedback (Theorem 2 and Theorem 4). Our experimental results in different settings also verify our theoretical results (in Section 5).
Several prior works use additional information to improve the performance of bandit algorithms. In the following, we discuss how auxiliary feedback differs from side information and side observation.
Side Information: Several works use context as side information to select the best action to play. This line of work is popularly known as contextual bandits (Li et al., 2010; Chu et al., 2011; Agrawal and Goyal, 2013; Li et al., 2017). Here, the mean reward of each arm is a function of context and is often parameterized, e.g., linear (Li et al., 2010; Chu et al., 2011; Agrawal and Goyal, 2013), GLM (Li et al., 2017), and non-linear (Valko et al., 2013). These contexts are assumed to be observed before an action is taken. However, we consider a problem where additional information is correlated with rewards that can only be observed after selecting the action.
Side Observations: Several works consider the different side observations settings in the literature, e.g., stochastic (Caron et al., 2012), adversarial (Mannor and Shamir, 2011; Kocak et al., 2014), graph feedback (Alon et al., 2015; Wu et al., 2015; Alon et al., 2017), and cascading feedback (Verma et al., 2019, 2020a,b). Side observations represent the additional information available about actions that the learner does not select. Auxiliary feedback is different from side information as it is available only for selected action and provides more information about the reward of that action.
Auxiliary Feedback: We use auxiliary feedback as control variates, which are used extensively for variance reduction in Monte-Carlo simulation of complex systems (Lavenberg and Welch, 1981; Lavenberg et al., 1982; James, 1985; Nelson, 1989, 1990; Botev and Ridder, 2017; Chen and Ghahramani, 2016). Recent works (Kreutzer et al., 2017; Vlassis et al., 2021; Verma and Hanawal, 2021) and (Sutton and Barto, 2018, Chapter 7.4) have exploited the availability of these control variates to build estimators with smaller variance and develop algorithms that have better performance guarantees. The closest work to our setting is Verma and Hanawal (2021). However, they only consider a non-parameterized setting with a finite number of actions. We thus consider a more general bandit setting with large (infinite) actions and allow a function to parameterize auxiliary feedback.
We consider a novel parameterized bandits problem in which the learner can observe auxiliary feedback correlated with the observed reward. In this problem, a learner has been given an action set, denoted by  $\mathcal{X} \subset \mathbb{R}^d$  where  $d \geq 1$ . At the beginning of round  $t$ , the learner selects an action  $x_t$  from action set  $\mathcal{X}$ . Then, the environment generates a stochastic reward  $y_t \doteq f(x_t) + \varepsilon_t$  for the selected action  $x_t$ , where  $f: \mathbb{R}^d \to \mathbb{R}$  is an unknown reward function and  $\varepsilon_t$  is a zero-mean Gaussian noise with variance  $\sigma^2$ . Apart from the stochastic reward  $y_t$ , the environment generates  $q$  of auxiliary feedback. The  $i$ -th auxiliary feedback is denoted by  $w_{t,i} \doteq g_i(x_t) + \varepsilon_{t,i}^w$ , where  $g_i: \mathbb{R}^d \to \mathbb{R}$  and  $\varepsilon_{t,i}^w$  is a zero-mean Gaussian noise with variance  $\sigma_{w,i}^2$ . The multiple correlation coefficient of reward and its auxiliary feedback is denoted by  $\rho$  and assumed to be the same across all actions.
The optimal action (denoted by  $x^{\star}$ ) has the maximum function value, i.e.,  $x^{\star} \in \operatorname{argmax}_{x \in \mathcal{X}} f(x)$ . After selecting an action  $x_{t}$ , the learner incurs a penalty (or instantaneous regret)  $r_{t}$ , where  $r_{t} = f(x^{\star}) - f(x_{t})$ . Since the optimal action is unknown, we sequentially estimate the reward function using available information on rewards and associated auxiliary feedback for the selected actions and then use it for choosing the action in the following round. Our goal is to learn a sequential policy that selects actions such that the total penalty (or regret) incurred by the learner is as minimum as possible. After  $T$  rounds, the regret of a sequential policy  $\pi$  that selects action  $x_{t}$  in the round  $t$  is given by
$$
\Re_ {T} (\pi) \doteq \sum_ {t = 1} ^ {T} r _ {t} = \sum_ {t = 1} ^ {T} \left(f \left(x ^ {\star}\right) - f \left(x _ {t}\right)\right). \tag {1}
$$
A policy  $\pi$  is a good policy if it has sub-linear regret, i.e.,  $\lim_{T\to \infty}\Re_T(\pi) / T = 0$ . This implies that the policy  $\pi$  will eventually learn to recommend the best action.
3 Known auxiliary feedback functions
We first focus on a simple case where all auxiliary feedback functions are assumed to be known. This assumption is not very strict in many applications as the learner can construct auxiliary feedback such that its mean value is known beforehand (see Kreutzer et al. (2017), Vlassis et al. (2021), and Chapter 12.9 of Sutton and Barto (2018) for such examples). When auxiliary feedback functions are unknown, we can estimate them using historical data or additional samples of auxiliary feedback. However, it will have some penalty in the performance (more details are in Section 4 and Section 5).
The first challenge we face is how to exploit auxiliary feedback to get a better reward function estimator. To resolve this, we extend control variate theory (Nelson, 1990) to the problems where a function can parameterize control variates. This new contribution is itself of independent interest.
3.1 Control variate
Let  $\mu$  be the unknown variable that needs to be estimated and  $y$  be its unbiased estimator, i.e.,  $\mathbb{E}[y] = \mu$ . Any random variable  $w$  with a known mean value  $(\omega)$  can be treated as a control variate for  $y$  if it is correlated with  $y$ . The control variate method (Nelson, 1990) exploits errors in estimates of known random variables to reduce the estimator's variance for the unknown random variable. This method works as follows. For any choice of a coefficient  $\beta$ , define a new random variable as  $z \doteq y - \beta(w - \omega)$ . Note that  $z$  is also an unbiased estimator of  $\mu$  (i.e.,  $\mathbb{E}[z] = \mu$ ) as
$$
\mathbb {E} [ z ] = \mathbb {E} [ y ] - \beta \mathbb {E} [ (w - \omega) ] = \mu - \beta (\mathbb {E} [ w ] - \omega) = \mu - \beta (\omega - \omega) = \mu .
$$
Using properties of variance and covariance, the variance of  $z$  is given by
$$
\mathbb {V} (z) = \mathbb {V} (y) + \beta^ {2} \mathbb {V} (w) - 2 \beta \operatorname {C o v} (y, w).
$$
The variance of  $z$  is minimized by setting  $\beta$  to  $\beta^{\star} = \operatorname{Cov}(y, w) / \mathbb{V}(w)$  and the minimum value is  $(1 - \rho^{2})\mathbb{V}(y)$ , where  $\rho = \operatorname{Cov}(y, w) / \sqrt{\mathbb{V}(w)\mathbb{V}(y)}$  is the correlation coefficient of  $y$  and  $w$ . We exploit this variance reduction to design a reward function estimator with tight confidence bounds.
3.2 Auxiliary feedback as control variates
Since the auxiliary feedback functions are known, we define a new variable using the reward sample and its auxiliary feedback. We refer to this variable as 'hybrid reward.' The hybrid reward definition
is motivated by the control variate method, except the control variate is parameterized by function in our setting. As  $w_{s,i}$  is the  $i^{\text{th}}$  auxiliary feedback observed with reward  $y_{s}$ , the hybrid reward for reward  $(y_{s})$  with its  $q$  auxiliary feedback  $\{w_{s,i}\}_{i=1}^{q}$  is defined by
$$
z _ {s, q} \dot {=} y _ {s} - \sum_ {i = 1} ^ {q} \beta_ {i} \left(w _ {s, i} - g _ {i} \left(x _ {s}\right)\right) = y _ {s} - \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}\right) \boldsymbol {\beta}, \tag {2}
$$
where  $\pmb{w}_s = (w_{s,1},\dots ,w_{s,q})$ $\pmb {g}_s = (g_1(x_s),\ldots ,g_q(x_s))$  , and  $\beta = (\beta_{1},\dots,\beta_{q})^{\top}$  . Let  $\Sigma_{\pmb{w}\pmb{w}}\in$ $\mathbb{R}^{q\times q}$  be the covariance matrix among auxiliary feedback and  $\sigma_{\pmb{y}\pmb{w}}\in \mathbb{R}^{q\times 1}$  be the vector of covariance between the reward and each of its auxiliary feedback. Then, the variance of  $z_{s,q}$  is minimized by setting the coefficient vector  $\beta$  to  $\beta^{\star} = \Sigma_{\pmb{w}\pmb{w}}^{-1}\sigma_{\pmb{y}\pmb{w}}$  , and the minimum value is  $(1 - \rho^2)\sigma^2$  , where  $\rho^2 = \sigma_{\pmb{y}\pmb{w}}^\top \Sigma_{\pmb{w}\pmb{w}}^{-1}\sigma_{\pmb{y}\pmb{w}} / \sigma^2$  is the multiple correlation coefficient of reward and its auxiliary feedback.
However,  $\Sigma_{ww}$  and  $\sigma_{yw}$  can be unknown in practice and need to be estimated to get the best estimate for  $\beta^{\star}$  to achieve maximum variance reduction. In our following result, we drive the best linear unbiased estimator of  $\beta$  (i.e.,  $\hat{\beta}_t$ ) using  $t$  observations of rewards and their auxiliary feedback.
Lemma 1. Let  $t > q + 2 \in \mathbb{N}$  and  $f_{t}$  be the estimate of function  $f$  which uses all information available at the end of round  $t$ , i.e.,  $\{x_s, y_s, \pmb{w}_s\}_{s=1}^t$ . Then, the best linear unbiased estimator of  $\beta^{\star}$  is
$$
\hat {\boldsymbol {\beta}} _ {t} \doteq (\boldsymbol {W} _ {t} ^ {\top} \boldsymbol {W} _ {t}) ^ {- 1} \boldsymbol {W} _ {t} ^ {\top} \boldsymbol {Y} _ {t},
$$
where  $\mathbf{W}_t$  is a  $t \times q$  matrix whose  $s^{th}$  row is  $(\mathbf{w}_s - \mathbf{g}_s)$  and  $\mathbf{Y}_t = (y_1 - f_t(x_1), \ldots, y_t - f_t(x_t))$ .
The proof follows after doing some manipulations in Eq. (2) and then using results from linear regression theory. The detailed proof of Lemma 1 and all other missing proofs are given in the supplementary material. After having a new observation of reward and its auxiliary feedback, the best linear unbiased estimator of  $\beta^{\star}$  is re-estimated. If  $\Sigma_{\boldsymbol{w}\boldsymbol{w}}$  or  $\sigma_{\boldsymbol{y}\boldsymbol{w}}$  are known, we can directly use them to estimate  $\beta^{\star}$  by replacing  $\boldsymbol{W}_t^\top \boldsymbol{W}_t$  with  $\Sigma_{\boldsymbol{w}\boldsymbol{w}}$  and  $\boldsymbol{W}_t^\top \boldsymbol{Y}_t$  with  $\sigma_{\boldsymbol{y}\boldsymbol{w}}$  in Lemma 1. The following result describes the properties of the hybrid reward when  $\beta^{\star}$  is replaced by  $\hat{\beta}_t$  in Eq. (2).
Theorem 1. Let  $t > q + 2 \in \mathbb{N}$ . If  $\hat{\beta}_t$  as defined in Lemma 1 is used to compute hybrid reward  $z_{s,q}$  for any  $s \leq t \in \mathbb{N}$ , then  $\mathbb{E}[z_{s,q}] = f(x_s)$  and  $\mathbb{V}(z_{s,q}) = \left(1 + \frac{q}{t - q - 2}\right)(1 - \rho^2)\sigma^2$ .
The key takeaways from Theorem 1 are as follows. First, the hybrid reward using  $\hat{\beta}_t$  is an unbiased estimator of reward function and hence, we can still use it to estimate the reward function  $f$ . Second, there is a less reduction in variance (i.e., by a factor  $(t - 2) / (t - q - 2)$  of maximum possible variance reduction) when  $\hat{\beta}_t$  is used for constructing hybrid reward in Eq. (2).
The variance reduction (given in Theorem 1) depends on the auxiliary feedback via two terms:  $\frac{t - 2}{t - q - 2}$  and  $\rho^2$  (defined in Line 142). Setting  $q = 1$  gives the minimum value for  $\frac{t - 2}{t - q - 2}$ , but  $\rho^2$  for  $q = 1$  will also be small as it only considers one auxiliary feedback, and hence maximum variance reduction will not be achieved. As we increase the number of auxiliary feedbacks,  $\rho^2$  increases, leading to more variance reduction. However, the term  $\frac{t - 2}{t - q - 2}$  increases at the same time, which can negate the variance reduction achieved by smaller  $\rho^2$ . Hence, keeping the number of auxiliary feedback used for hybrid reward small is important. A simple method for selecting a subset of auxiliary feedback (Lavenberg et al., 1982) works as follows: select the auxiliary feedback whose sample correlation coefficient with reward is the largest. Then, select the next auxiliary feedback whose sample partial correlation coefficient with reward was the largest given the first auxiliary feedback selected. Keep repeating the process until there is a variance reduction using additional auxiliary feedback.
3.3 Linear bandits with known auxiliary feedback functions
To highlight the main ideas, we restrict to the linear bandit setting in which the reward and auxiliary feedback functions are linear. In this setting, a learner selects an action  $x_{t}$  and observes a reward  $y_{t} = x_{t}^{\top}\theta^{\star} + \varepsilon_{t}$ , where  $\theta^{\star} \in \mathbb{R}^{d}$  ( $d \geq 1$ ) is unknown and  $\varepsilon_{t}$  is the zero-mean Gaussian noise with known variance  $\sigma^2$ . The learner also observes  $q$  auxiliary feedback, where  $i$ -th auxiliary feedback is denoted by  $w_{t,i} = x_{t}^{\top}\theta_{w,i}^{\star} + \varepsilon_{t,i}^{w}$ . Here,  $\theta_{w,i}^{\star} \in \mathbb{R}^{d}$  is known and  $\varepsilon_{t,i}^{w}$  is the zero-mean Gaussian noise with unknown variance  $\sigma_{w,i}^{2}$ . Our goal is to learn a policy that minimizes regret as defined in Eq. (1). We later extend our method for non-linear reward and auxiliary feedback functions in Section 4.3.
Let  $I$  be the  $d \times d$  identity matrix,  $V_{t} \doteq \sum_{s=1}^{t-1} x_{s} x_{s}^{\top}$ , and  $\overline{V}_{t} \doteq V_{t} + \lambda I$ , where  $\lambda > 0$  is the regularization parameter that ensures matrix  $\overline{V}_{t}$  is a positive definite matrix. The notation  $\|x\|_{A}$  denotes the weighted  $l_{2}$ -norm of vector  $x \in \mathbb{R}^{d}$  with respect to a positive definite matrix  $A \in \mathbb{R}^{d \times d}$ .
As shown in Theorem 1, the hybrid rewards is an unbiased reward estimator with a smaller variance than observed rewards. Thus, hybrid rewards lead to tighter confidence bounds for parameter  $\theta^{\star}$  than observed rewards. We propose a simple but effective method to exploit correlated auxiliary feedback, i.e., using hybrid rewards to estimate reward function instead of observed rewards.
Using this method, we adapt the well-known linear bandit algorithm OFUL (Abbasi-Yadkori et al., 2011) to our setting and named this algorithm OFUL-AF. This algorithm works as follows. It takes  $\lambda > 0$  as input and then initializes  $\overline{V}_1 = \lambda I$  and sets  $\hat{\theta}_1^z = 0_{\mathbb{R}^d}$  as initial estimate of parameter  $\theta^\star$ . The superscript  $z'$  in  $\hat{\theta}_1^z$  implies that hybrid rewards are used for estimating  $\theta^\star$ . At the beginning of round  $t$ , the algorithm selects an action  $x_t$  that maximizes the upper confidence bound of the action's reward, which is a sum of the estimated reward for the action  $(x^\top \hat{\theta}_t^z)$  and a confidence bonus  $\alpha_t^\sigma \| x \|_{\overline{V}_t^{-1}}$ . In the confidence bonus, the first term  $(\alpha_t^\sigma)$  is a slowly increasing function in  $t$  whose value is given in Theorem 2, and the second term  $(\| x \|_{\overline{V}_t^{-1}})$  decreases to zero as  $t$  increases.
OFUL-AF Algorithm for Linear Bandits with Auxiliary Feedback
1: Input:  $\lambda > 0$   
2: Initialization:  $\overline{V}_1 = \lambda I$  and  $\hat{\theta}_1^z = 0_{\mathbb{R}^d}$   
3: for  $t = 1, 2, \ldots$  do  
4: Select action  $x_t = \operatorname{argmax}_{x \in \mathcal{X}} \left( x^\top \hat{\theta}_t^z + \sigma \alpha_t \|x\|_{\overline{V}_t^{-1}} \right)$   
5: Observe reward  $y_t$  and its auxiliary feedback  $\{w_{t,i}\}_{i=1}^q$   
6: If  $t > q + 2$ , compute upper bound of hybrid reward's sample variance  $(\bar{\nu}_{z,t})$  or else  $\bar{\nu}_{z,t} = \sigma^2$   
7: If  $t \leq q + 2$  or  $\bar{\nu}_{z,t} \geq \sigma^2$ , set  $\hat{\beta}_t = 0$  or else compute  $\hat{\beta}_t$  using Lemma 1  
8:  $\forall s \leq t \in \mathbb{N}$ : compute  $z_{s,q}$  using  $\hat{\beta}_t$  in Eq. (2)  
9: Set  $\overline{V}_{t+1} = \overline{V}_t + x_tx_t^\top$ ,  $\hat{\theta}_{t+1}^z = \overline{V}_{t+1}^{-1} \sum_{s=1}^t x_s z_{s,q}$   
10: end for
After selecting an action  $x_{t}$ , the algorithm observes the reward  $y_{t}$  with its associated auxiliary feedback  $\{w_{t,i}\}_{i = 1}^{q}$ . It computes the upper bound on sample variance of hybrid reward (denoted by  $\bar{\nu}_{z,t}^1$ ) if  $t > q + 2$  or else it is set to  $\sigma^2$  and then checks two conditions. The first condition (i.e.,  $t \leq q + 2$ ) guarantees the sample variance is well-defined. Whereas the second condition (i.e.,  $\bar{\nu}_{z,t} > \sigma^2$ ) ensures the algorithm at least be as good as OFUL because  $\bar{\nu}_{z,t}$  can be larger than  $\sigma^2$  due to the overestimation in initial rounds. If both conditions  $t \leq q + 2$  and  $\bar{\nu}_{z,t} \geq \sigma^2$  fail, the value of  $\hat{\beta}_t$  is re-computed as defined in Lemma 1. The updated  $\hat{\beta}_t$  is then used to update all hybrid rewards, i.e.,  $z_{s,q}$ ,  $\forall s \leq t \in \mathbb{N}$ . Finally, the values of  $\overline{V}_{t + 1}$  and  $\hat{\theta}_{t + 1}$  are updated as  $\overline{V}_{t + 1} = \overline{V}_t + x_tx_t^\top$  and  $\hat{\theta}_{t + 1}^z = \overline{V}_{t + 1}^{-1}\sum_{s = 1}^t x_s z_{s,q}$ , which are then used to select the action in the following round. When  $\hat{\beta}_t = 0$  for all hybrid rewards, hybrid rewards are the same as the observed rewards, and hence OFUL-AF is the same as OFUL.
The regret analysis of any bandit algorithm hinges on bounding the instantaneous regret for each action. The following result gives an upper bound on the instantaneous regret of OFUL-AF.
Theorem 2. With a probability of at least  $1 - 2\delta$ , the instantaneous regret of OFUL-AF in round  $t$  is