SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
r _ {t} (\text {O F U L - A F}) \leq 2 \left(\alpha_ {t} ^ {\sigma} + \lambda^ {1 / 2} S\right) \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}},
$$
where  $\alpha_{t}^{\sigma} = \sqrt{\min\left(\sigma^{2},\bar{\nu}_{z,t - 1}\right)}$ $\alpha_{t}$ $\| \theta^{\star}\|_{2}\leq S,$  and  $\alpha_{t} = \sqrt{d\log\left(\frac{1 + tL^{2} / \lambda}{\delta}\right)}$  . For  $t > q + 2$  and  $\bar{\nu}_{z,t} <   \sigma^2$ $\mathbb{E}\left[r_t(\mathrm{OFUL - AF})\right]\leq \widetilde{O}\left(\left(\frac{(t - 3)(1 - \rho^2)}{t - q - 3}\right)^{\frac{1}{2}}r_t(\mathrm{OFUL})\right)$  . Here,  $\widetilde{O}$  hides constant terms.
<sup>1</sup>Let  $\hat{\nu}_{z,t}$  be the sample variance estimate of hybrid rewards (details in Appendix A.2). Then,  $\bar{\nu}_{z,t} = \frac{(t - 2)\hat{\nu}_{z,t}}{\chi_{1 - \delta,t}^2}$ , where  $\chi_{1 - \delta ,t}^2$  denotes  $100(1 - \delta)^{\mathrm{th}}$  percentile value of the chi-squared distribution with  $t - 2$  degrees of freedom.
The proof follows by bounding the estimation error of the parameter  $\theta^{\star}$  when the estimation method uses auxiliary feedback. This result shows that auxiliary feedback leads to a better instantaneous regret upper bound and a better regret (as defined in Eq. (1)) than the vanilla OFUL algorithm. Since the improvement in instantaneous regret increase with  $t$ , having a single constant to compare with regret of OFUL may lead to weaker regret upper bound than the sum of all instantaneous regret.
4 Estimated auxiliary feedback functions
Auxiliary feedback functions may be unknown in many real-life problems. However, the learner can construct an unbiased estimator for the auxiliary feedback function using historical data or acquiring more samples of auxiliary feedback. But these estimated functions offer a lower variance reduction than known auxiliary functions. To study the effect of using the estimated auxiliary feedback functions on the performance of bandit algorithms, we borrow some techniques from approximate control variate theory (Gorodetsky et al., 2020; Pham and Gorodetsky, 2022) as we discussed next.
4.1 Approximate control variates
Let  $y$  be an unbiased estimator of an unknown variable  $\mu$  and a random variable  $w$  with a known estimated mean  $(\omega_e)$  be a control variate of  $y$ . As long as the known estimated mean is an unbiased estimator of  $w$ , one can use it to reduce the variance of  $y$  as follows. For any choice of a coefficient  $\beta_e$ , define a new random variable as  $z_e \doteq y - \beta_e \bar{w}$ , where  $\bar{w} = w - \omega_e$ . Since  $\omega_e$  is an unbiased estimator of  $w$ , it is straightforward to show that  $z_e$  is also an unbiased estimator of  $y$ .
By using properties of variance and covariance, the variance of  $z_{e}$  is given by
$$
\operatorname {V a r} \left(z _ {e}\right) = \operatorname {V a r} (y) + \beta_ {e} ^ {2} \operatorname {C o v} (\bar {w}, \bar {w}) - 2 \beta_ {e} \operatorname {C o v} (y, \bar {w}).
$$
The variance of  $z_{e}$  is minimized by setting  $\beta_{e}$  to  $\beta_{e}^{\star} = \mathrm{Cov}(\bar{w},\bar{w})^{-1}\mathrm{Cov}(y,\bar{w})$  and the minimum value of  $\operatorname{Var}(z_e)$  is  $(1 - \rho_e^2)\operatorname {Var}(y)$ , where  $\rho_{e} = \operatorname {Cov}(y,\bar{w})\left(\operatorname {Cov}(\bar{w},\bar{w})^{-1} / \operatorname {Var}(y)\right)\operatorname {Cov}(y,\bar{w})$ .
4.2 Auxiliary feedback with unknown functions as approximate control variates
We now introduce a new definition of hybrid reward that uses estimated auxiliary feedback functions. Let  $w_{s,i}$  be the  $i^{\text{th}}$  auxiliary feedback observed with reward  $y_{s}$  and  $g_{e,i}$  be the unbiased estimator of function  $g_{i}$ . Then, the hybrid reward with  $q$  estimated auxiliary feedback functions is defined by
$$
z _ {e, s, q} = y _ {s} - \sum_ {i = 1} ^ {q} \beta_ {e, i} \left(w _ {s, i} - g _ {e, i} \left(x _ {t}\right)\right) = y _ {s} - \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {e, s}\right) \boldsymbol {\beta} _ {e}. \tag {3}
$$
where  $\pmb{w}_{s} = (w_{s,1},\dots ,w_{s,q})$ $g_{e,s} = (g_{e,1}(x_s),\ldots ,g_{e,q}(x_s))$  , and  $\beta_{e} = (\beta_{e,1},\dots ,\beta_{e,q})^{\top}$  . Let  $\Sigma_{\bar{\pmb{w}}\bar{\pmb{w}}} \in \mathbb{R}^{q \times q}$  denote the covariance matrix among centered auxiliary feedback (i.e.,  $\bar{\pmb{w}}_s = \pmb {w}_s - \pmb {g}_{e,s})$  and  $\sigma_{\pmb{y}\bar{\pmb{w}}} \in \mathbb{R}^{q \times 1}$  denote the vector of covariance between reward and its centered auxiliary feedback. Then, the variance of  $z_{e,s,q}$  is minimized by setting the  $\beta_{e}$  to  $\beta_{e}^{\star} = \Sigma_{\bar{\pmb{w}}\bar{\pmb{w}}}^{-1}\sigma_{\pmb{y}\bar{\pmb{w}}}$  , and the minimum value of  $\mathrm{Var}(z_{e,s,q})$  is  $(1 - \rho_e^2)\sigma^2$  , where  $\rho_e^2 = \sigma_{\pmb{y}\bar{\pmb{w}}}^\top \Sigma_{\bar{\pmb{w}}\bar{\pmb{w}}}^{-1}\sigma_{\pmb{y}\bar{\pmb{w}}} / \sigma^2$
The definition of hybrid reward given in Eq. (3) is very flexible and allows different estimators to estimate auxiliary feedback functions. The only difference among these estimators is how they partition the available samples of auxiliary feedback to estimate auxiliary function  $g_{i}$ . As no optimal partitioning strategy is known, we adopt the Independent Samples (IS) and Multi-Fidelity (MF) sampling strategy for our setting where finite samples of auxiliary feedback are available. Both strategies are proven to be asymptotically optimal (Gorodetsky et al., 2020), implying the variance reduction is asymptotically the same as if auxiliary feedback functions are known.
IS and MF sampling strategy: Let  $s$  and  $s_i \supset s$  be the sample sets used for estimating functions  $f$  and  $g_i$ , respectively. Then, for the IS sampling strategy,  $(s_i \setminus s) \cap (s_j \setminus s) = \emptyset$  for  $i \neq j$ , i.e., the extra samples used for estimating the function  $g_i$  are unique. Whereas, for the MF sampling strategy,  $s_i = s \cup_{j=1}^{i} s_j'$  and  $s_i' \cap s_j' = \emptyset$  for  $i \neq j$ , i.e., the estimation of function  $g_i$  uses the samples that were used for estimating function  $g_{i-1}$  with some additional samples. Refer to Fig. 1 for the visual representation of both sampling strategies. After adopting Theorem 3 and Theorem 4 from
Figure 1: Left two figures: Visualization of IS and MF sampling strategies. Each column represents samples used for estimating function (written at the top), and the same color is used to show shared samples among auxiliary function estimation. Right two figures: Interaction between AFC bandit algorithm and environment. AFC bandit algorithm that only uses observed rewards (second from right), and AFC bandit algorithm that also uses auxiliary feedback as hybrid rewards (rightmost).
Gorodetsky et al. (2020) to our setting, we can further simplify  $\Sigma_{\bar{w}\bar{w}}$  and  $\sigma_{y\bar{w}}$  when IS and MF sampling strategies (denoted by  $e$ ) are used for estimating auxiliary feedback functions as follows:
$$
\Sigma_ {\tilde {w} \bar {w}} = \left(\Sigma_ {\boldsymbol {w w}} \circ \boldsymbol {F} _ {e}\right) / t \text {a n d} \sigma_ {\boldsymbol {y} \tilde {w}} = \left(\operatorname {d i a g} \left(\boldsymbol {F} _ {e}\right) \circ \sigma_ {\boldsymbol {y w}}\right) / t,
$$
where  $t$  denotes the number of reward observations with its auxiliary feedback,  $\mathrm{diag}(A)$  represents a vector whose elements are the diagonal of the matrix  $A$ , and  $\circ$  denotes an element-wise product. The  $ij$ -th element of matrix  $\mathbf{F}_e \in \mathbb{R}^{q \times q}$  is
$$
f _ {e, i j} = \left\{ \begin{array}{l l} ((r _ {i} - 1) (r _ {j} - 1)) / (r _ {i} r _ {j}) & \mathrm {i f} i \neq j \mathrm {a n d} e = \mathrm {I S} \\ (\min (r _ {i}, r _ {j}) - 1) / \min (r _ {i}, r _ {j}) & \mathrm {i f} i \neq j \mathrm {a n d} e = \mathrm {M F} \\ (r _ {i} - 1) / r _ {i} & \mathrm {o t h e r w i s e ,} \end{array} \right.
$$
where  $r_i \in \mathbb{R}^+$  is the ratio between the total number of samples used for estimating function  $g_i$  by sampling strategy  $e$  and the total number of samples used for estimating  $f$ .
Since  $\Sigma_{\bar{w}\bar{w}}$  and  $\sigma_{y\bar{w}}$  may be unknown, they must be estimated to get the best estimate for  $\beta^{\star}$ . Our following result gives the best linear unbiased estimator of  $\beta_{e}$  (i.e.,  $\hat{\beta}_{e,t}$ ) that uses  $t$  observations of rewards and their auxiliary feedback with estimated auxiliary feedback functions.
Lemma 2. Let  $t > q + 2 \in \mathbb{N}$ ,  $e$  is the sampling strategy, and  $f_{t}$  be the estimate of function  $f$  at the end of round  $t$  which uses  $\{x_s, y_s, w_s\}_{s=1}^t$ . Then, the best linear unbiased estimator of  $\beta_e^\star$  is
$$
\hat {\boldsymbol {\beta}} _ {e, t} = \left(\boldsymbol {W} _ {t} ^ {\top} \boldsymbol {W} _ {t} \circ \boldsymbol {F} _ {e}\right) ^ {- 1} \left(\operatorname {d i a g} \left(\boldsymbol {F} _ {e}\right) \circ \boldsymbol {W} _ {t} ^ {\top} \boldsymbol {Y} _ {t}\right),
$$
where  $\mathbf{W}_t$  is a  $t \times q$  matrix whose  $s^{th}$  row is  $\mathbf{w}_s - \mathbf{g}_{e,s}$  and  $\mathbf{Y}_t = (y_1 - f_t(x_1),\dots,y_t - f_t(x_t))$ .
After adopting matrix manipulation tricks from Pham and Gorodetsky (2022) to our setting, the proof follows similar steps as the proof of Lemma 1. We now characterize the properties of the hybrid reward that uses either IS or MF sampling strategy for estimating auxiliary feedback functions.
Theorem 3. Let  $t > q + 2 \in \mathbb{N}$  and  $e$  is the sampling strategy. If  $\hat{\beta}_{e,t}$  as defined in Lemma 2 is used to compute hybrid reward  $z_{e,s,q}$  for any  $s \leq t \in \mathbb{N}$ , then  $\mathbb{E}[z_{e,s,q}] = f(x_s)$  and  $\mathbb{V}(z_{e,s,q}) = \left(1 + \frac{a(e)q}{t - q - 2}\right)(1 - \rho_e^2)\sigma^2$ , where  $a(IS) = 1$ ,  $a(MF) = \frac{r - 1}{r}$  if  $r_i = r$ ,  $\forall i \in \{1,2,\dots,q\}$  when using MF sampling strategy for estimating auxiliary feedback functions.
The key takeaways from Theorem 3 are as follows. First, the hybrid reward with estimated auxiliary feedback is still an unbiased estimator, so one can use it to estimate the reward function  $f$ . Second, there is a potential loss in variance reduction as it has an extra multiplicative factor  $a(e)$  and  $\rho_e^2 \leq \rho^2$ .
Remark 1. As samples for estimating auxiliary functions increase compared to the reward function, the variance reduction from IS and MF sampling strategy converges to the reduction achieved using known auxiliary functions. As  $\forall i: r_i \to \infty$ , then  $F_e \to 1_{q \times q}$ . It is now straightforward to see that  $\Sigma_{\bar{w}\bar{w}}$  will become  $\Sigma_{ww}$ ,  $\sigma_{y\bar{w}}$  will become  $\sigma_{yw}$ , and hence  $\rho_e^2 = \rho^2$  as  $\forall i: r_i \to \infty$ .
Remark 2. The IS and MF sampling strategies are shown to be asymptotically optimal (Gorodetsky et al., 2020), i.e., the variance reduction achieved by both strategies is asymptotically the same as if
auxiliary feedback functions are known. However, both sampling strategies are useful in different applications, e.g., the IS sampling strategy suits the problems in which different auxiliary feedback can be independently sampled. In contrast, the MF sampling suits the problems where auxiliary feedback can not be sampled independently.
4.3 Parameterized bandits with estimated auxiliary feedback functions
We now consider the parameterized bandit setting described in Section 2, where the reward and auxiliary feedback function can be non-linear. To exploit the available auxiliary feedback in linear bandits, we propose a method in Section 3.3 that uses hybrid reward in place of rewards to get tight upper confidence bound for the estimator of an unknown reward function and hence smaller regret as compared to the vanilla OFUL due to the smaller variance of the hybrid rewards. We generalize this observation and introduce the notion of Auxiliary Feedback Compatible (AFC) bandit algorithm.
Definition 1 (AFC Bandit Algorithm). Any bandit algorithm  $\mathfrak{A}$  is Auxiliary Feedback Compatible if: (i)  $\mathfrak{A}$  can use correlated reward samples to construct upper confidence bound for reward function and (ii) with probability  $1 - \delta$ , its estimated reward function  $f_{t}^{\mathfrak{A}}$  has the following property:
$$
\left| f _ {t} ^ {\mathfrak {A}} (x) - f (x) \right| \leq \sigma h (x, \mathcal {O} _ {t}) + l (x, \mathcal {O} _ {t}),
$$
where  $x \in \mathcal{X}$ ,  $\sigma^2$  is the variance of Gaussian noise in observed reward, and  $\mathcal{O}_t$  denotes the observations of actions and their rewards with the parameters of  $\mathfrak{A}$  at the beginning of round  $t$ .
As the estimated coefficient vector uses all past samples, the resultant hybrid rewards become correlated due to using this estimated coefficient vector. Bandit algorithms like UCB1 (Auer et al., 2002) and kl-UCB (Cappé et al., 2013) are not AFC as they need independent reward samples to construct upper confidence bounds. In contrast, bandit algorithms like OFUL (Abbasi-Yadkori et al., 2011), Lin-UCB (Chu et al., 2011), UCB-GLM (Li et al., 2017), IGP-UCB, and GP-TS (Chowdhury and Gopalan, 2017) are AFC as they all use techniques proposed in Abbasi-Yadkori et al. (2011) for building the upper confidence bound, which does not need reward samples to be independent.
As AFC bandit algorithms use the noise variance of observed reward for constructing the confidence upper bound, they can also exploit available auxiliary feedback by replacing reward with its respective hybrid reward as shown in Fig. 1 (rightmost figure). We next give an upper bound on the instantaneous regret for any AFC bandit algorithm that uses hybrid rewards instead of observed rewards.
Theorem 4. Let  $\mathfrak{A}$  be an AFC bandit algorithm with  $|f_t^{\mathfrak{A}}(x) - f(x)| \leq \sigma h(x, \mathcal{O}_t) + l(x, \mathcal{O}_t)$  and  $\bar{\nu}_{e,z,t}$  be the upper bound on sample variance of hybrid reward, whose value is set to  $\sigma^2$  for  $t \leq q + 2$ . Then, with a probability of at least  $1 - 2\delta$ , the instantaneous regret of  $\mathfrak{A}$  after using hybrid rewards (named  $\mathfrak{A}$ -AF) for reward function estimation in round  $t$  is
$$
r _ {t} (\mathfrak {A} \text {- A F}) \leq 2 \min  (\sigma , (\bar {\nu} _ {e, z, t}) ^ {\frac {1}{2}}) h (x, \mathcal {O} _ {t}) + l (x, \mathcal {O} _ {t}),
$$
where  $e = \{IS, MF, KF\}$ , and  $KF$  denotes the case where auxiliary functions are known. For  $t > q + 2$
$$
a n d \bar {\nu} _ {e, z, t} <   \sigma^ {2}, \mathbb {E} \left[ r _ {t} (\mathfrak {A} - \mathrm {A F}) \right] \leq \widetilde {O} \left(\left(\left(\frac {t - (1 - a (e)) q - 3}{t - q - 3}\right) \left(1 - \rho_ {e} ^ {2}\right)\right) ^ {\frac {1}{2}} r _ {t} (\mathfrak {A})\right), w h e r e a (K F) = 1.
$$
After using Theorem 1 and Theorem 3 to replace the variance of hybrid reward, the proof follows similar steps as the proof of Theorem 2. We have given more details about the values of  $h(x,\mathcal{O}_t)$  and  $l(x,\mathcal{O}_t)$  for different AFC bandit algorithms in Table 1.
Table 1: Values of  $h(x,\mathcal{O}_t)$  and  $l(x,\mathcal{O}_t)$  for different AFC bandit algorithms
AFC bandit algorithm	h(x, O_t)	l(x, O_t)
OFUL (Abbasi-Yadkori et al., 2011)	√d log(1+tL2/λ/δ) ||x||Vt-1	λ1/2S ||x||Vt-1
Lin-UCB (OFUL for contextual setting)	√d log(1+tL2/λ/δ) ||x||Vt-1	λ1/2S ||x||Vt-1
GLM-UCB (Li et al., 2017)	√d/2 log(1+2t/d)+log(1/δ) ||x||Vt-1/κ	0
IGP-UCB (Chowdhury and Gopalan, 2017)	√2(γt-1+1+log(1/δ))σt-1(x)	Bσt-1(x)
To validate our theoretical results, we empirically demonstrate the performance gain due to auxiliary feedback in different settings of parameterized bandits. We repeat all our experiments 50 times and show the regret as defined in Eq. (1) with a  $95\%$  confidence interval (vertical line on each curve shows the confidence interval). Due to space constraints, the details of used problem instances are given in Appendix A.5 of the supplementary material.
Comparing regret with benchmark bandit algorithms: We considered three bandit settings for our experiments: linear bandits, linear contextual bandits, and non-linear contextual bandits (details are given in Appendix A.5). The formal setting of a contextual bandits with auxiliary feedback is given in Appendix A.4. We used the following existing bandit algorithms for these settings: OFUL (Abbasi-Yadkori et al., 2011) for linear bandits, Lin-UCB (Chu et al., 2011) for linear contextual bandits, and Lin-UCB with the polynomial kernel (which we named NLin-UCB) for non-linear contextual bandits. We compare the performance of these benchmark bandit algorithms with four different variants of our algorithms. The first variant assumes the auxiliary feedback functions are known (highlighted by adding '-AF' to the benchmark algorithms). When auxiliary feedback functions are unknown, we use IS and MF sampling strategy while maintaining  $r = 2$  (i.e., getting one extra sample of auxiliary feedback in each round). The IS and MF sampling strategies are the same when only one auxiliary feedback exists. Since we only use one auxiliary feedback in our experiments, we highlight this variant by adding '-IS/MF' to the benchmark algorithms. When IS and MF sampling strategies are used, one needs to update the auxiliary feedback functions in each round to get better estimators. However, it leads to the re-computation of all variables that are needed for updating the hybrid rewards, which is not needed when auxiliary feedback functions are fixed. Therefore, we consider two more computationally efficient variants for the unknown auxiliary functions setting. One variant assumes the knowledge of biased auxiliary feedback, i.e.,  $g_{i}(x) + \varepsilon_{g}$  is available instead of  $g_{i}(x)$  (highlighted by adding '-BE' to the benchmark algorithms). Another variant assumes that some initial samples of auxiliary feedback are available, which are used to get the auxiliary feedback function estimator. We highlight this variant by adding '-EH' to the benchmark algorithms. All variants with given parameters perform better than benchmark bandit algorithms (see Fig. 2a, Fig. 2b, and Fig. 2c). We observe the expected performance among these variants as the variant with a known auxiliary feedback function outperforms all other variants. At the same time, IS/MF sampling strategy-based variant outperforms the other two heuristic variants for the setting of unknown auxiliary feedback function.
Regret vs. different biased estimator: To know the effect of bias in auxiliary feedback (i.e.,  $\varepsilon_{g}$ ) in the mean value of auxiliary feedback, we run an experiment with the same linear contextual bandits experiment setup mentioned above. To see the variation in regret, we set  $\varepsilon_{g} = \{1, 0.2, 0.1, 0.07, 0.05\}$ . As shown in Fig. 2d, the regret increases with an increase in bias and even starts performing poorly than Lin-UCB. This experiment demonstrates that as long as the bias in auxiliary feedback is within a limit, there will be an advantage to using this computationally efficient variant.
Regret vs. number of historical samples of auxiliary feedback : Increasing the number of historical samples of auxiliary feedback for estimating the auxiliary feedback function reduces the error in its estimation, leading to better performance. To observe this, we use estimators using different numbers of auxiliary feedback samples, i.e.,  $n_h = \{5, 7, 10, 15, 20\}$  in linear contextual bandits setting. As expected, the regret decreases with an increase in auxiliary feedback samples, but using an estimator with a few samples even performs poorly than Lin-UCB, as shown in Fig. 2e.
Regret vs. correlation coefficient: As theoretical results imply that the regret decreases when the correlation between reward and auxiliary feedback increases. To validate this, we used problem instances with different correlation coefficients in linear contextual bandits setting. As expected, we observe that the regret decreases as the correlation coefficient increases, as shown in Fig. 2f.
This paper studies a novel parameterized bandit problem in which a learner observes auxiliary feedback correlated with the observed reward. We first introduce the notion of 'hybrid reward,' which combines the reward and its auxiliary feedback. To get the maximum benefit from hybrid reward, we treat auxiliary feedback as a control variate and then extend control variate theory to a setting
(a) Linear Bandit
(b) Linear Contextual Bandits
(c) Non-linear Contextual Bandits
(d) Biased Estimator
(e) Estimator from historical data
(f) Influence of correlation
Figure 2: Top row: Comparing regret of different variants with their benchmark bandit algorithms in different settings. Bottom row: Regret vs. different biases in Lin-UCB-BE (left figure), regret vs. number of historical samples of auxiliary feedback in Lin-UCB-EH (middle figure), and regret of Lin-UCB-AF vs. varying correlation coefficients of reward and its auxiliary feedback (right figure).
where a function can parameterize control variates. Equipped with these results, we show that the variance of hybrid rewards is smaller than observed rewards. We then use these hybrid rewards to estimate the reward function, leading to tight confidence bounds and hence smaller regret. We have proved that the expected instantaneous regret of any AFC bandit algorithm after using hybrid rewards is improved by a factor of  $O((1 - \rho^2)^{\frac{1}{2}})$ , where  $\rho$  is the correlation coefficient of the reward and its auxiliary feedback. Our experiments also validate our theoretical results. An interesting future direction is to extend these results to bandit settings with heteroscedastic and non-Gaussian noise.
DesCartes: this research is supported by the National Research Foundation, Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.
Aleksandris Slivkins. Introduction to Multi-Armed Bandits. Foundations and Trends® in Machine Learning, 2019.
Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, 2020.
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A Contextual-Bandit Approach to Personalized News Article Recommendation. In Proc. WWW, pages 661-670, 2010.
Wei Chu, Lihong Li, Lev Reyzin, and Robert E Schapire. Contextual Bandits with Linear Payoff Functions. In Proc. AISTATS, pages 208-214, 2011.
Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved Algorithms for Linear Stochastic Bandits. In Proc. NeurIPS, pages 2312-2320, 2011.
Shipra Agrawal and Navin Goyal. Thompson Sampling for Contextual Bandits with Linear Payoffs. In Proc. ICML, pages 127-135, 2013.
Barry L Nelson. Batch size effects on the efficiency of control variates in simulation. European Journal of Operational Research, pages 184-196, 1989.
Zdravko Botev and Ad Ridder. Variance Reduction. Wiley statsRef: Statistics reference online, pages 1-6, 2017.
Yutian Chen and Zoubin Ghahramani. Scalable Discrete Sampling as a Multi-Armed Bandit Problem. In Proc. ICML, pages 2492-2501, 2016.
Alex A Gorodetsky, Gianluca Geraci, Michael S Eldred, and John D Jakeman. A Generalized Approximate Control Variate Framework for Multifidelity Uncertainty Quantification. Journal of Computational Physics, page 109257, 2020.
Trung Pham and Alex A Gorodetsky. Ensemble Approximate Control Variate Estimators: Applications to MultiFidelity Importance Sampling. Journal on Uncertainty Quantification, pages 1250-1292, 2022.
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time Analysis of the Multiarmed Bandit Problem. Machine Learning, pages 235-256, 2002.
Olivier Cappé, Aurélien Garivier, Odalric-Ambrym Maillard, Rémi Munos, and Gilles Stoltz. Kullback-Leibler Upper Confidence Bounds for Optimal Sequential Allocation. The Annals of Statistics, pages 1516-1541, 2013.
F Hayashi. Econometrics. Princeton University Press, 2000.
Sara A Van De Geer. Estimation. Encyclopedia of Statistics in Behavioral Science, pages 549-553, 2005.
Bruce Schmeiser. Batch Size Effects in the Analysis of Simulation Output. Operations Research, pages 556-568, 1982.