SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A.1 Missing proofs related to auxiliary feedback
Results from linear regression
We first state results for linear regression that we will use in the subsequent proofs. Consider the following regression problem with  $t$  samples and  $q$  features:
$$
z _ {s} = \boldsymbol {x} _ {s} ^ {\top} \boldsymbol {\theta} + \varepsilon_ {s}, \quad i \in \{1, 2, \dots , t \}
$$
where  $z_{s} \in \mathbb{R}$  is the  $s^{\mathrm{th}}$  target variable,  $\pmb{x}_s = (x_{s1},\dots,x_{sq}) \in \mathbb{R}^q$  is the  $s^{\mathrm{th}}$  feature vector,  $\pmb{\theta} \in \mathbb{R}^q$  is the unknown regression parameters, and  $\varepsilon_s$  is a normally distributed noise with mean 0 and constant variance  $\sigma^2$ . The values of noise  $\varepsilon_s$  form an IID sequence and are independent of  $\pmb{x}_s$ . Let
$$
\boldsymbol {Z} _ {t} = \left( \begin{array}{c} z _ {1} \\ \vdots \\ z _ {t} \end{array} \right), \qquad \boldsymbol {X} _ {t} = \left( \begin{array}{c c c} x _ {1 1} & \ldots & x _ {1 q} \\ \vdots & \ldots & \vdots \\ X _ {t 1} & \ldots & x _ {t q} \end{array} \right), \text {a n d} \quad \varepsilon_ {t} = \left( \begin{array}{c} \varepsilon_ {1} \\ \vdots \\ \varepsilon_ {t} \end{array} \right).
$$
Then, the best linear unbiased estimator of  $\pmb{\theta}$  is  $\hat{\pmb{\theta}}_t = (X_t^\top X_t)^{-1}X_t^\top Z_t$ , which has the following finite sample properties.
Fact 1. The following are the finite sample properties of the least square estimator  $\hat{\pmb{\theta}}_t$ :
1.  $\mathbb{E}\left[\hat{\pmb{\theta}}_t|\pmb {X}_t\right] = \pmb {\theta},$
(unbiased estimator)
2.  $\operatorname{Var}(\hat{\pmb{\theta}}_t|\pmb{X}_t) = \sigma^2 (\pmb{X}_t^\top \pmb{X}_t)^{-1}$ , and
(expression for the variance)
3.  $\operatorname{Var}(\hat{\pmb{\theta}}_{ti}|\mathbf{X}_t) = \sigma^2 (\mathbf{X}_t^\top \mathbf{X}_t)_{ii}^{-1}$ ,
element-wise variance)
where  $(\pmb{X}_t^\top \pmb{X})_{ii}^{-1}$  is the  $ii$ -element of the matrix  $(\pmb{X}_t^\top \pmb{X})^{-1}$ .
In the above result, the first two properties are from Proposition 1.1 of Hayashi (2000), whereas the third property is from Van De Geer (2005). The following result gives the finite sample properties of the estimator of noise variance  $\sigma^2$ .
Fact 2. (Hayashi, 2000, Proposition 1.2) Let  $\hat{\sigma}_t^2 = \frac{1}{t - q}\sum_{s = 1}^{t}(z_s - \pmb{x}_s^\top \hat{\pmb{\theta}}_t)^2$  be estimator of  $\sigma^2$  and  $t > q$  (so that  $\hat{\sigma}_t^2$  is well defined). Then,  $\hat{\sigma}_t^2$  is an unbiased estimator of  $\sigma^2$ , i.e.,  $\mathbb{E}\left[\hat{\sigma}_t^2 |X_t\right] = \sigma^2$ .
Using the Schur complement, we have the following results about the inverse of the block matrix.
Fact 3. Let  $G = \left( \begin{array}{cc}t & B\\ C & D \end{array} \right)$  be a block matrix, where  $t\in \mathbb{R}\setminus \{0\}$ ,  $B,C,D$  are respectively  $1\times q$ ,  $q\times 1$ , and  $q\times q$  matrices of real numbers. Then,  $G_{11}^{-1} = t^{-1} + t^{-1}B(tD - CB)^{-1}C$ .
Control variates theory
Let  $y$  be the random variable of interest with unknown mean  $\mu$ . There are  $q$  control variates correlated with  $y$ , where  $i^{\mathrm{th}}$  control variate has mean  $\omega_{i}$  and its  $s^{\mathrm{th}}$  observation is denoted by  $w_{s,i}$ . For any  $s \in \{1, \dots, t\}$ , we define a variable  $z_{s}$  using  $s^{\mathrm{th}}$  observation of  $y_{s}$  and its control variates as follows:
$$
z _ {s} = y _ {s} - \left(\boldsymbol {w} _ {s} - \boldsymbol {\omega}\right) \boldsymbol {\beta},
$$
where  $\pmb{w}_s = (w_{s,1},\dots ,w_{s,q})$  and  $\omega = (\omega_{1},\ldots ,\omega_{q})$ . The above equation can be re-written as:
$$
y _ {s} = z _ {s} + \left(\boldsymbol {w} _ {s} - \boldsymbol {\omega}\right) \boldsymbol {\beta}.
$$
Under the assumption of  $z_{s}$  being a unbiased estimator of  $\mu$ , we can write  $y_{s}$  as follows:
$$
y _ {s} = \mu + (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) \beta + \varepsilon_ {z, s}.
$$
where  $\varepsilon_{z,1},\ldots ,\varepsilon_{z,t}$  are IID and have zero mean Gaussian noise with variance  $(1 - \rho^2)\sigma^2$  . Let
$$
\overline {{\boldsymbol {Y}}} _ {t} = \left( \begin{array}{c} y _ {1} \\ \vdots \\ y _ {t} \end{array} \right),   \overline {{\boldsymbol {W}}} _ {t} = \left( \begin{array}{c c} 1 & \boldsymbol {w} _ {1} - \boldsymbol {\omega} \\ \vdots & \vdots \\ 1 & \boldsymbol {w} _ {t} - \boldsymbol {\omega} \end{array} \right),   \gamma = \left( \begin{array}{c} \mu \\ \boldsymbol {\beta} \end{array} \right),   \text {a n d}   \varepsilon_ {z, t} = \left( \begin{array}{c} \varepsilon_ {z, 1} \\ \vdots \\ \varepsilon_ {z, t} \end{array} \right).
$$
The best linear unbiased estimator of  $\gamma$  is  $\hat{\gamma} = (\overline{W}_t^\top \overline{W}_t)^{-1}\overline{W}_t^\top \overline{Y}_t$ . To get  $\hat{\mu}_{z,t}$  and  $\hat{\beta}^{\star}$ , we expand  $\hat{\gamma}$  as follows:
$$
\begin{array}{l} \hat {\gamma} = \left(\left( \begin{array}{c c} 1 & \boldsymbol {w} _ {1} - \boldsymbol {\omega} \\ \vdots & \vdots \\ 1 & \boldsymbol {w} _ {t} - \boldsymbol {\omega} \end{array} \right) ^ {\top} \left( \begin{array}{c c} 1 & \boldsymbol {w} _ {1} - \boldsymbol {\omega} \\ \vdots & \vdots \\ 1 & \boldsymbol {w} _ {t} - \boldsymbol {\omega} \end{array} \right)\right) ^ {- 1} \left( \begin{array}{c c} 1 & \boldsymbol {w} _ {1} - \boldsymbol {\omega} \\ \vdots & \vdots \\ 1 & \boldsymbol {w} _ {t} - \boldsymbol {\omega} \end{array} \right) ^ {\top} \left( \begin{array}{c} y _ {1} \\ \vdots \\ y _ {t} \end{array} \right) \\ = \left(\left( \begin{array}{c c c} 1 & \dots & 1 \\ \boldsymbol {w} _ {1} - \boldsymbol {\omega} & \dots & \boldsymbol {w} _ {t} - \boldsymbol {\omega} \end{array} \right) \left( \begin{array}{c c} 1 & \boldsymbol {w} _ {1} - \boldsymbol {\omega} \\ \vdots & \vdots \\ 1 & \boldsymbol {w} _ {t} - \boldsymbol {\omega} \end{array} \right)\right) ^ {- 1} \left( \begin{array}{c c c} 1 & \dots & 1 \\ \boldsymbol {w} _ {1} - \boldsymbol {\omega} & \dots & \boldsymbol {w} _ {t} - \boldsymbol {\omega} \end{array} \right) \left( \begin{array}{c} y _ {1} \\ \vdots \\ y _ {t} \end{array} \right) \\ = \left( \begin{array}{c c} t & \sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) \\ \sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) & \sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) ^ {\top} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) \end{array} \right) ^ {- 1} \left( \begin{array}{c} \sum_ {s = 1} ^ {t} y _ {s} \\ \sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) y _ {s} \end{array} \right) \\ \end{array}
$$
After taking first matrix from RHS to LHS and using  $\hat{\gamma} = \left( \begin{array}{c}\hat{\mu}_{z,t}\\ \hat{\beta}_t \end{array} \right)$ , we have
$$
\left( \begin{array}{c c} t & \sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) \\ \sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) & \sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) ^ {\top} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) \end{array} \right) \binom {\hat {\mu} _ {z, t}} {\hat {\beta} _ {t}} = \binom {\sum_ {s = 1} ^ {t} y _ {s}} {\sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) y _ {s}}. \tag {4}
$$
From above, we get the following equation:
$$
\begin{array}{l} t \hat {\mu} _ {z, t} + \left(\sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {\omega})\right) \hat {\boldsymbol {\beta}} _ {t} = \sum_ {s = 1} ^ {t} y _ {s} \\ \Rightarrow \hat {\mu} _ {z, t} = \frac {1}{t} \sum_ {s = 1} ^ {t} y _ {s} - \left(\frac {1}{t} \sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {\omega})\right) \hat {\beta} _ {t}. \\ \end{array}
$$
Using  $\hat{\mu}_{y,t} = \frac{1}{t}\sum_{s=1}^{t} y_s$  and  $\hat{\boldsymbol{\omega}}_t = \frac{1}{t}\sum_{s=1}^{t} \boldsymbol{w}_s$ , we get
$$
\hat {\mu} _ {z, t} = \hat {\mu} _ {y, t} - (\hat {\omega} _ {t} - \omega) \hat {\beta} _ {t}. \tag {5}
$$
Similarly, we have another equation as follows:
$$
\begin{array}{l} \hat {\mu} _ {z, t} \sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) + \left(\sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) ^ {\top} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}))\right) \hat {\beta} _ {t} = \sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) y _ {s} \\ \Rightarrow \hat {\boldsymbol {\beta}} _ {t} = \left(\sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) ^ {\top} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}))\right) ^ {- 1} \left(\sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) (y _ {s} - \hat {\mu} _ {z, t})\right). \\ \end{array}
$$
Using  $\mathbf{W}_t = \begin{pmatrix} \mathbf{w}_1 - \boldsymbol{\omega} \\ \vdots \\ \mathbf{w}_t - \boldsymbol{\omega} \end{pmatrix}$  and  $\mathbf{Y}_t = \begin{pmatrix} y_1 - \hat{\mu}_{z,t} \\ \vdots \\ y_t - \hat{\mu}_{z,t}. \end{pmatrix}$ , we have
$$
\Rightarrow \hat {\boldsymbol {\beta}} _ {t} = \left(\boldsymbol {W} _ {t} ^ {\top} \boldsymbol {W} _ {t}\right) ^ {- 1} \boldsymbol {W} _ {t} ^ {\top} \boldsymbol {Y} _ {t}. \tag {6}
$$
In the following, we first state the fundamental results from the control variates theory.
Fact 4. (Nelson, 1990, Theorem 1) Let  $O_{s} = (Y_{s}, W_{s,1}, \ldots, W_{s,q})^{\top}$  follow a  $(q + 1)$ -variate normal distribution with mean vector  $(\mu, \omega)$  and  $\{O_{1}, \ldots, O_{t}\}$  be a IID sequence. Assume  $\hat{\mu}_{z,t} = \sum_{s=1}^{t} z_{s}$ , where  $z_{s} = y_{s} - (\boldsymbol{w}_{s} - \boldsymbol{\omega})\hat{\beta}_{t}$  and  $\hat{\beta}_{t}$  used here is given by Eq. (6), then
$$
\mathbb {E} \left[ \hat {\mu} _ {z, t} \right] = \mu \text {a n d}
$$
$$
\mathbb {V} \left(\hat {\mu} _ {z, t}\right) = \left(1 + \frac {q}{t - q - 2}\right) \left(1 - \rho^ {2}\right) \mathbb {V} \left(\hat {\mu} _ {y, t}\right),
$$
where  $\sigma_{Y\mathbf{W}}\Sigma_{\mathbf{W}\mathbf{W}}^{-1}\sigma_{Y\mathbf{W}}^{\top} / \sigma^{2}$  is the square of the multiple correlation coefficient,  $\sigma^2 = \mathbb{V}(Y)$ , and  $\sigma_{Y\mathbf{W}} = (\mathrm{Cov}(Y,W_1),\ldots ,\mathrm{Cov}(Y,W_q))$  (we have dropped the subscript  $s$  as observations are IID).
Auxiliary feedback as control variates
Lemma 1. Let  $t > q + 2 \in \mathbb{N}$  and  $f_{t}$  be the estimate of function  $f$  which uses all information available at the end of round  $t$ , i.e.,  $\{x_s, y_s, w_s\}_{s=1}^t$ . Then, the best linear unbiased estimator of  $\beta^{\star}$  is
Proof. Recall Eq. (2) for  $s^{\mathrm{th}}$  hybrid reward with known auxiliary functions, i.e.,  $z_{s,q} = y_s - (\pmb{w}_s - \pmb{g}_s)\beta$ , which can be re-written as  $y_s = z_{s,q} + (\pmb{w}_s - \pmb{g}_s)\beta$ . By definition,  $z_{s,q} = f(x_s) + \varepsilon_s^z$  for optimal  $\beta$ , where  $\varepsilon_{z,s}$  is zero-mean Gaussian noise with variance  $(1 - \rho^2)\sigma^2$ . Then,  $y_s = f(x_s) + (\pmb{w}_s - \pmb{g}_s)\beta + \varepsilon_{z,s}$ . Let  $\varphi$  be an unknown function that maps every  $x$  to a space where  $f(x) = \varphi(x)^\top f$  holds. Then we can re-write the above equation as follows:
$$
y _ {s} = f ^ {\top} \varphi (x _ {s}) + (\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}) \boldsymbol {\beta} + \varepsilon_ {z, s}.
$$
Adapting Eq. (4) to our setting, we have
$$
\left( \begin{array}{c c} \sum_ {s = 1} ^ {t} \varphi (x _ {s}) ^ {\top} \varphi (x _ {s}) & \sum_ {s = 1} ^ {t} \varphi (x _ {s}) ^ {\top} (\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}) \\ \sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}) ^ {\top} \varphi (x _ {s}) & \sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}) ^ {\top} (\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}) \end{array} \right) \left( \begin{array}{c} f _ {t} \\ \hat {\boldsymbol {\beta}} _ {t} \end{array} \right) = \left( \begin{array}{c} \sum_ {s = 1} ^ {t} \varphi (x _ {s}) y _ {s} \\ \sum_ {s = 1} ^ {t} (\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}) y _ {s} \end{array} \right).
$$
Let  $f_{t}$  is the estimated  $f$  using available information, i.e.,  $\{x_{s}, y_{s}, \boldsymbol{w}_{s}\}_{s=1}^{t}$ . To get best linear unbiased estimator for  $\beta^{\star}$ , we use the following equation from above matrix,
$$
\begin{array}{l} \left(\sum_ {s = 1} ^ {t} \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}\right) ^ {\top} \varphi \left(x _ {s}\right)\right) f _ {t} + \left(\sum_ {s = 1} ^ {t} \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}\right) ^ {\top} \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}\right)\right) \hat {\boldsymbol {\beta}} _ {t} = \sum_ {s = 1} ^ {t} \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}\right) y _ {s} \\ \Rightarrow \left(\sum_ {s = 1} ^ {t} \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}\right) ^ {\top} \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}\right)\right) \hat {\beta} _ {t} = \sum_ {s = 1} ^ {t} \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}\right) y _ {s} - \sum_ {s = 1} ^ {t} \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}\right) ^ {\top} \left(\varphi \left(x _ {s}\right) ^ {\top} f _ {t}\right) \\ \Rightarrow \hat {\boldsymbol {\beta}} _ {t} = \left(\sum_ {s = 1} ^ {t} \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}\right) ^ {\top} \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}\right)\right) ^ {- 1} \sum_ {s = 1} ^ {t} \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}\right) \left(y _ {s} - \varphi \left(x _ {s}\right) ^ {\top} f _ {t}\right) \\ \Rightarrow \hat {\beta} _ {t} = \left(\sum_ {s = 1} ^ {t} \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}\right) ^ {\top} \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}\right)\right) ^ {- 1} \sum_ {s = 1} ^ {t} \left(\boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}\right) \left(y _ {s} - f _ {t} \left(x _ {s}\right)\right) \\ \end{array}
$$
Using definition  $f_{t}(x_{s}) = \varphi (x_{s})^{\top}f_{t}$ $\pmb {W}_t = \left( \begin{array}{c}\pmb {w}_1 - \pmb {g}_s\\ \vdots \\ \pmb {w}_t - \pmb {g}_s \end{array} \right),$  and  $\mathbf{Y}_t = \left( \begin{array}{c}y_1 - f_t(x_1)\\ \vdots \\ y_t - f_t(x_t). \end{array} \right)$  we get
$$
\Rightarrow \hat {\boldsymbol {\beta}} _ {t} = \left(\boldsymbol {W} _ {t} ^ {\top} \boldsymbol {W} _ {t}\right) ^ {- 1} \boldsymbol {W} _ {t} ^ {\top} \boldsymbol {Y} _ {t}.
$$
Since the reward and its auxiliary feedback observations are functions of the selected action, we can not directly use the control variate theory due to parameterized mean values of the reward and its auxiliary feedback. To overcome this challenge, we centered the observations by its function value and defined new centered variables as follows:
$$
y _ {s} ^ {c} = y _ {s} - f (x _ {s}), \quad \boldsymbol {w} _ {s} ^ {c} = \boldsymbol {w} _ {s} - \boldsymbol {g} _ {s}, \text {a n d} z _ {s, q} ^ {c} = z _ {s, q} - f (x _ {s}).
$$
In our setting, these centered variables  $(y_s^c, w_s^c$ , and  $z_s^c$ ), follow zero mean Gaussian distributions with variance  $\sigma^2$ ,  $\sigma_w^2 = (\sigma_{w,1}^2, \ldots, \sigma_{w,q}^2)$ , and  $(1 - \rho^2)\sigma^2$ , respectively.
Proof. The sequence  $(y_s^c, \pmb{w}_s^c)_{s=1}^t$  is an IID sequence and follows a Gaussian distribution with mean 0. We now define  $z_{s,q}^c = y_s^c - \pmb{w}_s^c\pmb{\beta} = y_s^c - (\pmb{w}_s - \pmb{g}_s)\pmb{\beta}$ , which can be re-written as  $y_{s,q}^c = z_s^c + (\pmb{w}_s - \pmb{g}_s)\pmb{\beta}$ . Let  $f_t$  be the estimated  $f$  using available information, i.e.,  $\{x_s, y_s, \pmb{w}_s\}_{s=1}^t$
and hence we can write estimated  $z_{s,q}$  as  $\hat{z}_{s,q} = f_t(x_s)$  and hence  $\hat{z}_{s,q}^c = f_t(x_s) - f(x_s)$ . Now, adapting Eq. (6) to our setting and replacing estimated mean in  $Y_t$  by  $\hat{z}_{s,q}^c$ ,  $s^{\text{th}}$  value of  $Y_t$  is  $y_s^c - \hat{z}_{s,q}^c = y_s - f(x_s) - (f_t(x_s) - f(x_s)) = y_s - f_t(x_s)$ . With these manipulations, we get the following best linear unbiased estimator for  $\beta^\star$ :
$$
\hat {\boldsymbol {\beta}} _ {t} = \left(\boldsymbol {W} _ {t} ^ {\top} \boldsymbol {W} _ {t}\right) ^ {- 1} \boldsymbol {W} _ {t} ^ {\top} \boldsymbol {Y} _ {t},
$$
which is the same as defined in Lemma 1.
By adapting Fact 4 for a single sample (i.e.,  $z_{s,q}$ ) while using  $\hat{\beta}_t$  to define hybrid reward, we have
$$
\mathbb {E} \left[ z _ {s, q} ^ {c} \right] = 0 \text {a n d}
$$
$$
\mathbb {V} \left(z _ {s, q} ^ {c}\right) = \left(1 + \frac {q}{t - q - 2}\right) (1 - \rho^ {2}) \mathbb {V} \left(y _ {s} ^ {c}\right),
$$
By extending the definition of  $\mathbb{E}\left[z_{s,q}^{c}\right]$  we have,
$$
\mathbb {E} \left[ z _ {s, q} - f \left(x _ {s}\right) \right] = 0 \Longrightarrow \mathbb {E} \left[ z _ {s, q} \right] - f \left(x _ {s}\right) = 0 \Longrightarrow \mathbb {E} \left[ z _ {s, q} \right] = f \left(x _ {s}\right)
$$
This proofs the hybrid reward is an unbiased estimator of reward.
Since variance is invariant to constant change, we have
$$
\begin{array}{l} \mathbb {V} \left(z _ {s, q}\right) = \mathbb {V} \left(z _ {s, q} - f (x _ {s})\right) \\ = \mathbb {V} \left(z _ {s, q} ^ {c}\right) \\ = \left(1 + \frac {q}{t - q - 2}\right) \left(1 - \rho^ {2}\right) \mathbb {V} \left(y _ {s} ^ {c}\right) \\ = \left(1 + \frac {q}{t - q - 2}\right) (1 - \rho^ {2}) \mathbb {V} \left(y _ {s} - f \left(x _ {s}\right)\right) \\ = \left(1 + \frac {q}{t - q - 2}\right) (1 - \rho^ {2}) \mathbb {V} \left(y _ {s}\right). \\ \end{array}
$$
Since  $\mathbb{V}(y_s) = \sigma^2$ , we have  $\mathbb{V}(z_{s,q}) = \left(1 + \frac{q}{t - q - 2}\right)(1 - \rho^2)\sigma^2$ .
Lemma 2. Let  $t > q + 2 \in \mathbb{N}$ ,  $e$  is the sampling strategy, and  $f_{t}$  be the estimate of function  $f$  at the end of round  $t$  which uses  $\{x_{s}, y_{s}, \boldsymbol{w}_{s}\}_{s=1}^{t}$ . Then, the best linear unbiased estimator of  $\beta_{e}^{\star}$  is
Proof. Recall the  $s^{\mathrm{th}}$  hybrid reward defined in Eq. (3) using sampling strategy  $e$  as  $z_{e,s,q}^{e} = y_{s} - (\pmb{w}_{s} - \pmb{g}_{e,s})\pmb{\beta}_{e}$ , which can be re-written as  $y_{s} = z_{e,s,q}^{e} + (\pmb{w}_{s} - \pmb{g}_{e,s})\pmb{\beta}_{e}$ . Following similar steps as of Lemma 1, we can re-write the above equation as  $y_{s} = f^{\top}\varphi(x_{s}) + (\pmb{w}_{s} - \pmb{g}_{e,s})\pmb{\beta}_{e} + \varepsilon_{z,s}$ .
Using  $\pmb{W}_{e,t} = \begin{pmatrix} \pmb{w}_1 - \pmb{g}_{e,s} \\ \vdots \\ \pmb{w}_t - \pmb{g}_{e,s} \end{pmatrix}$ , and  $\pmb{Y}_t = \begin{pmatrix} y_1 - f_t(x_1) \\ \vdots \\ y_t - f_t(x_t) \end{pmatrix}$ , we get  $\hat{\beta}_{e,t} = (\pmb{W}_{e,t}^\top \pmb{W}_{e,t})^{-1} \pmb{W}_{e,t}^\top \pmb{Y}_t$ .
From Appendix D and E of Gorodetsky et al. (2020), we have  $\mathbf{W}_{e,t}^{\top}\mathbf{W}_{e,t} = \mathbf{W}_{t}^{\top}\mathbf{W}_{t} \circ \mathbf{F}_{e}$  and  $\mathbf{W}_{e,t}^{\top}\mathbf{Y}_{t} = \mathrm{diag}(\mathbf{F}_{e}) \circ \mathbf{W}_{t}^{\top}\mathbf{Y}_{t}$ . Using these two equality, we have
$$
\hat {\boldsymbol {\beta}} _ {e, t} = \left(\boldsymbol {W} _ {t} ^ {\top} \boldsymbol {W} _ {t} \circ \boldsymbol {F} _ {e}\right) ^ {- 1} (\operatorname {d i a g} \left(\boldsymbol {F} _ {e}\right) \circ \boldsymbol {W} _ {t} ^ {\top} \boldsymbol {Y} _ {t}).
$$
Proof. The proof follows the similar steps as Theorem 1 except we adapt the part (b.) of Theorem 4 from Pham and Gorodetsky (2022) instead of using Fact 4 to show the variance reduction when sampling strategy (IS or MF) is used for estimating auxiliary feedback functions.  $\square$
A.2 Unbiased estimate of variance
Consider the following regression problem with target variable  $y_{s}$ , which is defined as follows:
$$
y _ {s} = \mu + (\boldsymbol {w} _ {s} - \boldsymbol {\omega}) \boldsymbol {\beta} + \varepsilon_ {z, s}.
$$
Now, using Fact 1, we have  $\mathbb{V}(\hat{\mu}_{z,t}) = \sigma^2 (\overline{\boldsymbol{W}}_t^\top \overline{\boldsymbol{W}}_t)^{-1}$ , where  $(\boldsymbol {Y}^{\top}\boldsymbol {Y})_{11}^{-1}$  is the upper left most element of matrix  $(\boldsymbol {Y}^{\top}\boldsymbol {Y})^{-1}$  (Schmeiser, 1982). Then after  $t$  observations, the unbiased variance estimator of  $\mathbb{V}(\hat{\mu}_{z,t})$  is given by
$$
\hat {\nu} _ {z, t} = \hat {\sigma} _ {z, t} ^ {2} (\overline {{\boldsymbol {W}}} _ {t} ^ {\top} \overline {{\boldsymbol {W}}} _ {t}) _ {1 1} ^ {- 1},
$$
where  $\hat{\sigma}_{z,t}^2 = \frac{1}{t - q - 1}\sum_{s = 1}^{t}(y_s - \hat{\mu}_{z,t})^2$  (Nelson, 1990), which is also an unbiased variance estimator of  $\sigma^2$  (from Fact 2). Further,  $\hat{\nu}_{z,t}$  is also an unbiased estimator of  $\mathbb{V}(\hat{\mu}_{z,t})$ , i.e.,  $\mathbb{E}[\hat{\nu}_{z,t}] = \mathbb{V}(\hat{\mu}_{z,t})$  (Nelson, 1990, Theorem 1). We can adapt this approach to our setting. However when noise variance  $(\sigma)$  is unknown, computing  $(\overline{W}_t^\top \overline{W}_t)_{11}^{-1}$  may not be possible to general function  $f$  as  $\varphi$  function may not be known. Though the setting in which  $(\overline{W}_t^\top \overline{W}_t)_{11}^{-1}$  can be computed, we have to use the upper bound of variance to construct the confidence bound for reward function  $f$  as random sample variance estimate can be small and leads to invalid confidence bounds. Given  $t$  observations, the upper bound of the sample variance is given by  $\bar{\nu}_{z,t} = \frac{(t - 2)\hat{\nu}_{z,t}}{\chi_{1 - \delta,t}^2}$ , where  $\chi_{1 - \delta ,t}^2$  denotes  $100(1 - \delta)^{\mathrm{th}}$  percentile value of the chi-squared distribution with  $t - 2$  degrees of freedom.
A.3 Missing proofs related to regret analysis
where  $\alpha_{t}^{\sigma} = \sqrt{\min\left(\sigma^{2},\bar{\nu}_{z,t - 1}\right)}\alpha_{t},\| \theta^{\star}\|_{2}\leq S,$  and  $\alpha_{t} = \sqrt{d\log\left(\frac{1 + tL^{2} / \lambda}{\delta}\right)}$  For  $t > q + 2$  and  $\bar{\nu}_{z,t} < \sigma^2$ $\mathbb{E}\left[r_t(\mathrm{OFUL - AF})\right]\leq \widetilde{O}\left(\left(\frac{(t - 3)(1 - \rho^2)}{t - q - 3}\right)^{\frac{1}{2}}r_t(\mathrm{OFUL})\right)$ . Here,  $\widetilde{O}$  hides constant terms.
Proof. When only observed rewards are used for estimating underlying unknown parameters in the linear bandit setting, i.e.,  $\hat{\theta}_t = \overline{V}_t^{-1}\sum_{s=1}^t x_s y_s$ , then with probability  $1 - \delta$ , the confidence bound (Abbasi-Yadkori et al., 2011, Theorem 1) is
$$
\left\| \hat {\theta} _ {t} - \theta^ {\star} \right\| _ {\bar {V} _ {t}} \leq \sigma \sqrt {d \log \left(\frac {1 + t L ^ {2} / \lambda}{\delta}\right)} + \lambda^ {1 / 2} S, \tag {7}
$$
where  $\sigma^2$  is the variance of observed rewards given action (since the noise variance is  $\sigma^2$ ). To ensure the performance of OFUL-AF is as good as OFUL, we only used hybrid reward samples for estimation when the upper bound on the variance of hybrid rewards is smaller than the variance of rewards, i.e.,  $\bar{\nu}_{z,t-1} < \sigma^2$ . At the beginning of round  $t$ , the variance upper bound of hybrid rewards is computed using  $t-1$  observations and given by  $\bar{\nu}_{z,t-1} = \frac{(t-2)\hat{\nu}_{z,t-1}}{\chi_{1-\delta,t}^2}$ , where  $\hat{\nu}_{z,t-1}$  is an unbiased sample variance estimate of hybrid rewards using  $t-1$  observations and  $\chi_{1-\delta,t}^2$  (implying the variance upper bound holds with at least probability of  $1-\delta$ ) denotes  $100(1-\delta)^{\mathrm{th}}$  percentile value of the chi-squared distribution with  $t-2$  degrees of freedom. When  $\bar{\nu}_{z,t} < \sigma^2$ , we replace rewards  $\{y_s\}_{s=1}^t$  with its respective hybrid rewards, i.e.,  $\{z_{s,q}\}_{s=1}^t$  to estimate underlying parameter and use it for next round. After using hybrid rewards to estimate the unknown parameter, we replace
$\sigma^2$  in Eq. (7) with the variance upper bound of hybrid rewards. Then we get the following upper bound which holds with a probability of  $1 - 2\delta$ .
$$
\begin{array}{l} \left\| \hat {\theta} _ {t} - \theta^ {\star} \right\| _ {\bar {V} _ {t}} \leq \sqrt {\min (\sigma^ {2} , \bar {\nu} _ {z , t - 1})} \sqrt {d \log \left(\frac {1 + t L ^ {2} / \lambda}{\delta}\right)} + \lambda^ {1 / 2} S \\ = \sqrt {\min  \left(\sigma^ {2} , \bar {\nu} _ {z , t - 1}\right)} \alpha_ {t} + \lambda^ {1 / 2} S \\ \Rightarrow \left\| \hat {\theta} _ {t} - \theta^ {\star} \right\| _ {\bar {V} _ {t}} = \alpha_ {t} ^ {\sigma} + \lambda^ {1 / 2} S, \\ \end{array}
$$